Skip to main content

Notifications

Announcements

No record found.

Customer experience | Sales, Customer Insights,...
Suggested answer

SSIS with large amount of data slow and bad Performance via Kingswaysoft

Posted on by Microsoft Employee

We have lots of SSIS packages that updates/creates records in CRM. Some entities like contact , account, customer product (our custom entity to hold all account types) , while we are migrating records facing with slow performans and high CPU in SQL machine (has 20 processor). But generally to other custom entities we do not have any slowness or problem. And to do these operation we use KingswaySoft application. Let's say I have a SSIS package called CASA , if I execute this package with 10.000 data less than 1 minute I can complete my process. But if I want to execute with 250.000 data , package should be completed in half an hour but it is taking 1.5-2 hours almost. We tried the combinations at the below but none of them gave good results for CPU and all of them gave around same time. We do not understand why with less data everything finish so quickly , but cannot handle with larger data. And we realized for every record a query sends to businessprocessflow to check there is a identified business process on it or not and that is using lots of CPU too. But when you enabled business process flow option on entity there is no way to turn back. We use CRM on -premise.

Thread : 64 , Batch : 250 : AutoAdjustBufferSize: True , Default Buffer Max Rows : 1.000.000 , CPU was 90 percent

Thread : 32 , Batch : 250 : AutoAdjustBufferSize: True , Default Buffer Max Rows : 1.000.000 , CPU was around 90 percent

Thread : 20 , Batch : 250 : AutoAdjustBufferSize: True , Default Buffer Max Rows : 1.000.000 , CPU was around 80 percent

Thread : 40 , Batch : 100  : AutoAdjustBufferSize: True , Default Buffer Max Rows : 1.000.000 , CPU was around 80 percent

Thread : 20 , Batch : 250 : AutoAdjustBufferSize:False  Default Buffer Max Rows : 400.000 , DefaultBufferSize : 50.000.000 CPU was around 80 percent

There is a option called Engine thread as default it is set to 10 , we tried to decrease that to 1,2 nothing changed.

  • Suggested answer
    Nishant Rana Profile Picture
    Nishant Rana 55 Microsoft Employee on at
    RE: SSIS with large amount of data slow and bad Performance via Kingswaysoft

    Might be helpful below were the combinations we tried to get the optimum speed

    nishantrana.me/.../

    And as suggested by experts, it could depend on other factors as well.

  • David Jennaway Profile Picture
    David Jennaway 14,063 on at
    RE: SSIS with large amount of data slow and bad Performance via Kingswaysoft

    As a first step, I'd suggest you do more detailed performance analysis to determine which process(es) have the high CPU usage. This will help determine if the issue is at the SSIS side (either dtexec.exe or dtsdebughost.exe depending on how you run the package), or the SQL side (sqlservr.exe). Note that if the issue is at the SSIS side, then it's not easy to determine if the root cause is with KingswaySoft or the SSIS engine, or a combination of the 2

    And can you split the package so that it runs for only 10000 records at a time ? E.g. if you run the package 25 times (either in serial or parallel) each for 10000 records, is this faster than running it once for 250000 records ?

  • Community Member Profile Picture
    Community Member Microsoft Employee on at
    RE: SSIS with large amount of data slow and bad Performance via Kingswaysoft

    Dears,

    Unfortunately this is what a general answer to give. Of course we already check every possible actions and this is why asking to everyone to find a way out. I contacted with KingswaySoft support several times but it was not helpful at all. My question is when I try a package with 10.000 data and 250.000 data why speed and CPU changes that badly and how I can improve it. Please give up to advise so general thing. Our project is on live and just trying trying is not gaining us anything.

  • Suggested answer
    KingswaySoft Profile Picture
    KingswaySoft 545 on at
    RE: SSIS with large amount of data slow and bad Performance via Kingswaysoft

    Hi HasretTaskin,

    The performance may depend on many factors like network latency, the particular instance you are working with, the data volume, plugins or workflows etc. You could try out our different options and see if that helps, but please note that it would depend on your particular environment and the entity you are working with, as this could be different from one case to another. We do not have standard instructions as such, but we certainly do have some recommendations. You can take a look at our FAQ page for more suggestions:

    www.kingswaysoft.com/.../advanced-topics

    You could also enable a proxy like Fiddler (www.telerik.com/fiddler), or use CRM trace logs to find out where the bottleneck is. For working with trace logs, you would need to do that in the CRM server-side.

    Let us know if there is anything else we can help you with and do not hesitate to reach out to our Support team directly.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,214 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans