Hello everyone.
I'm currently tasked with optimizing some old badly functioning SSIS packages, and rewriting them entirely in Console Applications for more control over the logic.
I've run into an odd issue, which is that the .NET Framework 4.8 outperforms the .NET 7 OData implementation, with almost four times faster batch inserts.
While it's not a big issue in itself, it doesn't feel right to be stuck in .NET Framework 4.8 when there is so many improvements in .NET 7.
My question is, have I missed anything in the implementation of the OData client? Or is OrganizationService just better performance than any custom implementation OData?
Code examples. I have 5000 Entities in a list, running on 10 threads, and the batch sizes is 200, with
ThreadPool.SetMinThreads(100, 100);
System.Net.ServicePointManager.DefaultConnectionLimit = 65000;
System.Net.ServicePointManager.Expect100Continue = false;
System.Net.ServicePointManager.UseNagleAlgorithm = false;
OrganizationService implementation:
Parallel.ForEach(entityList.ChunkBy(200), parallelOptions,chunk =>
{
var localOrganizationService = ConnectionService.GetOrganizationService();
var executeMultiple = new ExecuteMultipleRequest
{
Settings = new ExecuteMultipleSettings
{
ContinueOnError = true,
ReturnResponses = true
},
Requests = new OrganizationRequestCollection()
};
foreach (var coveragePolicy in chunk)
{
executeMultiple.Requests.Add(new DeleteRequest{Target = entity.ToEntityReference()});
if(executeMultiple.Requests.Count > 100)
{
localOrganizationService.Execute(executeMultiple);
executeMultiple.Requests.Clear();
}
}
if(executeMultiple.Requests.Count > 0)
{
localOrganizationService.Execute(executeMultiple);
}
});
and the OData implementation:
await Parallel.ForEachAsync(list.Chunk(200), parallelOptions, async (chunk, _) =>
{
var batch = new ODataBatch(ConnectionService.GetODataClient());
foreach (var entity in chunk)
{
batch += async oDataClient => await oDataClient.For<Entity>().Set(entity).InsertEntryAsync(false, _);
}
await batch.ExecuteAsync(_);
});
Performance difference with 5000 Entities inserted, Threadsize 10, Batch size 200:
OrganizationService: 37.2 seconds, 134 inserts per second -- 167 deletes per second
OData: 262.6 seconds, 19 inserts per second -- 130 Deletes per second
The implementation is pretty much identical in terms of setup and execution, but performance is simply slower in OData.
The funny thing is that OData performs quite well (but still not as good) as the OrganizationService in regards to Deletion, but the inserts are much more slower.
Is there any radical changes to the OData that I havent implemented yet, or is OrganizationService just better+
Naturally, I'd much rather go for the new and shiny .NET 7, however performance is key, so unless I find out a glaring mistake, I'll have to stay with .NET Framework 4.8.
