Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for isolated entities #358

Merged
merged 5 commits into from
Apr 26, 2024
Merged

Add support for isolated entities #358

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8640385
add support for isolated entities
sebastianburckhardt Feb 28, 2024
1583f34
address PR feedback
sebastianburckhardt Mar 18, 2024
6536c8f
Merge branch 'main' into pr/support-isolated-entities
davidmrdavid Apr 24, 2024
588a1a5
increase DTFx.Core version to 2.16.1 in DTFx.Netherite
davidmrdavid Apr 24, 2024
a6a210e
remove extra line
davidmrdavid Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion samples/Hello_Netherite_with_DotNetCore/HelloDF.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Azure.DurableTask.Netherite.AzureFunctions" Version="1.3.2" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.12.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.13.0" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to first release the current DF Extension release prior to getting this out, and then update this dependency to that upcoming release. We had to remove the latest DF Extension release due to a regression in OOProc serialization, so I'd like to avoid upgrading Netherite to depending on that faulty extension.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, maybe that means we should not include this PR in the 1.5.0 release?

Since we probably want this out sooner than later to make progress on the "partition corruption" theme.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we probably want the fix in Azure/azure-functions-durable-extension#2748 to be included so I think it makes sense to wait for the next DF Extension release.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, maybe that means we should not include this PR in the 1.5.0 release?

Yeah I'd say let's hold off, but prioritize this release shortly after.

<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.3" />
</ItemGroup>
<ItemGroup>
Expand Down
4 changes: 2 additions & 2 deletions src/DurableTask.Netherite.AzureFunctions/DurableTask.Netherite.AzureFunctions.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.DurableTask.Core" Version="2.15.1" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.12.0" />
<PackageReference Include="Microsoft.Azure.DurableTask.Core" Version="2.16.1" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.13.1" />
</ItemGroup>

<ItemGroup Condition=" '$(TargetFramework)' != 'netcoreapp2.2' ">
Expand Down
22 changes: 19 additions & 3 deletions src/DurableTask.Netherite.AzureFunctions/NetheriteProviderFactory.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ public class NetheriteProviderFactory : IDurabilityProviderFactory
readonly IServiceProvider serviceProvider;
readonly DurableTask.Netherite.ConnectionResolver connectionResolver;

readonly bool usesNewPassthroughMiddlewareForEntities;
readonly bool inConsumption;

// the following are boolean options that can be specified in host.json,
Expand Down Expand Up @@ -96,6 +97,14 @@ bool ReadBooleanSetting(string name) => this.options.StorageProvider.TryGetValue

this.TraceToConsole = ReadBooleanSetting(nameof(this.TraceToConsole));
this.TraceToBlob = ReadBooleanSetting(nameof(this.TraceToBlob));

WorkerRuntimeType runtimeType = platformInfo.GetWorkerRuntimeType();
if (runtimeType == WorkerRuntimeType.DotNetIsolated ||
runtimeType == WorkerRuntimeType.Java ||
runtimeType == WorkerRuntimeType.Custom)
{
this.usesNewPassthroughMiddlewareForEntities = true;
}
}

NetheriteOrchestrationServiceSettings GetNetheriteOrchestrationServiceSettings(string taskHubNameOverride = null, string connectionName = null)
Expand All @@ -109,12 +118,19 @@ NetheriteOrchestrationServiceSettings GetNetheriteOrchestrationServiceSettings(s
// different defaults for key configuration values.
int maxConcurrentOrchestratorsDefault = this.inConsumption ? 5 : 10 * Environment.ProcessorCount;
int maxConcurrentActivitiesDefault = this.inConsumption ? 20 : 25 * Environment.ProcessorCount;
int maxConcurrentEntitiesDefault = this.inConsumption ? 20 : 25 * Environment.ProcessorCount;
int maxEntityOperationBatchSizeDefault = this.inConsumption ? 50 : 5000;

// The following defaults are only applied if the customer did not explicitely set them on `host.json`
this.options.MaxConcurrentOrchestratorFunctions = this.options.MaxConcurrentOrchestratorFunctions ?? maxConcurrentOrchestratorsDefault;
this.options.MaxConcurrentActivityFunctions = this.options.MaxConcurrentActivityFunctions ?? maxConcurrentActivitiesDefault;
this.options.MaxEntityOperationBatchSize = this.options.MaxEntityOperationBatchSize ?? maxEntityOperationBatchSizeDefault;
this.options.MaxConcurrentOrchestratorFunctions ??= maxConcurrentOrchestratorsDefault;
this.options.MaxConcurrentActivityFunctions ??= maxConcurrentActivitiesDefault;
this.options.MaxConcurrentEntityFunctions ??= maxConcurrentEntitiesDefault;
this.options.MaxEntityOperationBatchSize ??= maxEntityOperationBatchSizeDefault;

if (this.usesNewPassthroughMiddlewareForEntities)
{
netheriteSettings.UseSeparateQueueForEntityWorkItems = true;
}

// copy all applicable fields from both the options and the storageProvider options
JsonConvert.PopulateObject(JsonConvert.SerializeObject(this.options), netheriteSettings);
Expand Down
4 changes: 2 additions & 2 deletions src/DurableTask.Netherite/DurableTask.Netherite.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@
<PackageReference Include="Microsoft.Azure.EventHubs.Processor" Version="4.3.2" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="11.2.3" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.16.0" />
<PackageReference Include="Microsoft.FASTER.Core" Version="2.0.23" />
<PackageReference Include="Microsoft.Azure.DurableTask.Core" Version="2.15.1" />
<PackageReference Include="Microsoft.FASTER.Core" Version="2.0.23" />
<PackageReference Include="Microsoft.Azure.DurableTask.Core" Version="2.16.1" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.1.1" PrivateAssets="All" />
<PackageReference Include="System.Threading.Channels" Version="4.7.1" />
Expand Down
12 changes: 8 additions & 4 deletions src/DurableTask.Netherite/OrchestrationService/InstanceQuery.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,12 @@ public class InstanceQuery
[DataMember]
internal bool PrefetchHistory { get; set; }

/// <summary>
/// Whether to exclude entities from the results.
/// </summary>
[DataMember]
internal bool ExcludeEntities { get; set; }

Copy link
Contributor

@jaliyaudagedara jaliyaudagedara Apr 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Extra line break

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, just removed it: a6a210e


/// <summary>
/// Construct an instance query with the given parameters.
Expand All @@ -77,9 +83,6 @@ public InstanceQuery(

internal bool HasRuntimeStatus => this.RuntimeStatus != null && this.RuntimeStatus.Length > 0;

internal bool IsSet => this.HasRuntimeStatus || !string.IsNullOrWhiteSpace(this.InstanceIdPrefix)
|| !(this.CreatedTimeFrom is null) || !(this.CreatedTimeTo is null);

internal bool Matches(OrchestrationState targetState)
{
if (targetState == null)
Expand All @@ -88,7 +91,8 @@ internal bool Matches(OrchestrationState targetState)
return (!this.HasRuntimeStatus || this.RuntimeStatus.Contains(targetState.OrchestrationStatus))
&& (string.IsNullOrWhiteSpace(this.InstanceIdPrefix) || targetState.OrchestrationInstance.InstanceId.StartsWith(this.InstanceIdPrefix))
&& (!this.CreatedTimeFrom.HasValue || targetState.CreatedTime >= this.CreatedTimeFrom.Value)
&& (!this.CreatedTimeTo.HasValue || targetState.CreatedTime <= this.CreatedTimeTo.Value);
&& (!this.CreatedTimeTo.HasValue || targetState.CreatedTime <= this.CreatedTimeTo.Value)
&& (!this.ExcludeEntities || !DurableTask.Core.Common.Entities.IsEntityInstance(targetState.OrchestrationInstance.InstanceId));
}
}
}
Loading