Skip to main content

Notifications

Announcements

No record found.

Microsoft Dynamics AX (Archived)

Entity Export via DIXF to entity store db. Copy to staging running for 2 days! How to determine bottleneck?

Posted on by Microsoft Employee

Hi All,

I am looking for help in determining why the load is so slow. 

Even though it is a complete refresh, shouldn't it run more quickly?

I have followed documentation to setup a couple target entities (customerAddress, Customer, CustGroup)

CustGroup went accross - even to the Data Store Db , but Customer is queued up behind CustAddress

JobStatus.png

The Entity Store Db is on an Azure instance with connection as ODBC 64 bit, but the problem is that loading

of the staging table has been running for over 24 hours and only about half of the 4 million rows have made it to staging.

At this rate, a complete refresh would take forever.  Here is the Target entity row.

ENT.png

Running SP_whoisActive returns the following.  The query is below that is running..

CPU tempdb_allocations tempdb_current blocking_session_id reads writes physical_reads used_memory status open_tran_count percent_complete host_name
65,969,342 0 0 NULL 123,650,557,912 293,475 35,055 12 runnable 1 NULL AX-DEV

SELECT T1.ENTITY,T1.ENTITYFIELD,T1.MAPPINGTYPE,T1.METHODDATASOURCE,T1.METHODNAME,T1.METHODRETURNPOSITION,T1.POSITION,
            T1.RUNMODIFIEDFIELD,T1.RUNVALIDATEFIELD,T1.XMLFIELD,T1.RECVERSION,T1.PARTITION,T1.RECID,T2.ENTITY,T2.ISAUTOMANUAL,
            T2.TARGETDATASOURCENAME,T2.TARGETFIELD,T2.TARGETTABLE,T2.XMLFIELD,T2.RECVERSION,T2.PARTITION,T2.RECID
FROM DMFTARGETXMLTOENTITYMAP T1
     LEFT OUTER JOIN DMFTARGETXML T2 ON ((T2.PARTITION=@P1)
                                                                    AND (((T1.ENTITY=T2.ENTITY)
                                                                    AND (T1.XMLFIELD=T2.XMLFIELD))
                                                                   AND (T2.TARGETDATASOURCENAME=@P2)))
WHERE ((T1.PARTITION=@P3) AND (T1.ENTITY=@P4)) ORDER BY T1.POSITION

Any things I can check, or any suggestions are welcome.  Thanks!

*This post is locked for comments

  • André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: Entity Export via DIXF to entity store db. Copy to staging running for 2 days! How to determine bottleneck?

    Hi,

    Can you check for long running queries on the SQL server? Does the TaxTrans table have a correct index for the field selection of the query above? If you need customer information to be exported, what entity is calling this TaxTrans query? Is it related to the export or not?

  • Community Member Profile Picture
    Community Member Microsoft Employee on at
    RE: Entity Export via DIXF to entity store db. Copy to staging running for 2 days! How to determine bottleneck?

    Also,  one update,  the server CPU/DISK/Memory do not seem to be being pushed at all..  

  • Community Member Profile Picture
    Community Member Microsoft Employee on at
    RE: Entity Export via DIXF to entity store db. Copy to staging running for 2 days! How to determine bottleneck?

    Hi (again) Andre,  Thanks for the reply.  So, another team in the company had to kill my job - That makes sense as they said thet CIL compiles were not working during the job.

    So,  I first attempted a "skip staging" load of the address table, which put 4M ( essentially empty) records in the target db.

    Then, I created a new schedule with only the Customer Entity.  It ran DMFExtDBSynchronizer successfully, then started DMFStagingWriter.

    It has been running for 40 minutes now, with 0 records in the staging table

    SP_whoisActive has only shown the following queries

           SELECT TOP 1 T1.RECID,T1.TAXGROUP,T1.TAXITEMGROUP

           FROM TAXTRANS T1

          WHERE (((PARTITION=@P1) AND (DATAAREAID=@P2)) AND ((((VOUCHER=@P3) AND (TRANSDATE=@P4)) AND (EXEMPTTAX=@P5)) AND (TAXORIGIN=@P6)))

           SELECT T1.ENTITY,T1.ENTITYFIELD,T1.MAPPINGTYPE,T1.METHODDATASOURCE,T1.METHODNAME,T1.METHODRETURNPOSITION,T1.POSITION,T1.RUNMODIFIEDFIELD,T1.RUNVALIDATEFIELD,T1.XMLFIELD,

           T1.RECVERSION,T1.PARTITION,T1.RECID,T2.ENTITY,T2.ISAUTOMANUAL,T2.TARGETDATASOURCENAME,T2.TARGETFIELD,T2.TARGETTABLE,T2.XMLFIELD,T2.RECVERSION,T2.PARTITION,T2.RECID

          FROM  DMFTARGETXMLTOENTITYMAP T1 LEFT OUTER JOIN DMFTARGETXML T2 ON ((T2.PARTITION=@P1) AND (((T1.ENTITY=T2.ENTITY) AND (T1.XMLFIELD=T2.XMLFIELD)) AND (T2.TARGETDATASOURCENAME=@P2)))

         WHERE ((T1.PARTITION=@P3) AND (T1.ENTITY=@P4)) ORDER BY T1.POSITION

    sp_who2 does not show any blocking...

    Do you happen to know anything else to check?

    Thanks so much

  • André Arnaud de Calavon Profile Picture
    André Arnaud de Cal... 291,280 Super User 2024 Season 2 on at
    RE: Entity Export via DIXF to entity store db. Copy to staging running for 2 days! How to determine bottleneck?

    Hi,

    You can start monitoring for system resources on the AOS and SQL machines. Is there sufficient CPU memory? Have you started the task in batch or on the client? Can you find some other activity on the SQL server e.g.  long running queries or blocking statements?

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

December Spotlight Star - Muhammad Affan

Congratulations to a top community star!

Top 10 leaders for November!

Congratulations to our November super stars!

Tips for Writing Effective Suggested Answers

Best practices for providing successful forum answers ✍️

Leaderboard

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 291,280 Super User 2024 Season 2

#2
Martin Dráb Profile Picture

Martin Dráb 230,214 Most Valuable Professional

#3
nmaenpaa Profile Picture

nmaenpaa 101,156

Leaderboard

Featured topics

Product updates

Dynamics 365 release plans