Entities with huge data not exporting in BYOD

(2) Share

Report

Posted on by Maverick

182

Hi Experts,

I have set up a DW in Azure for making power BI reports using BYOD approach. I have created the required entities and published it into the DW. After that I exported data into DW using DIXF and data populated in all entities except 3 tables. After checking I find that these 3 tables are having huge volume of data. largest table is having approx 8 million records.

I have waited for more that 3 weeks and data is still not inserted. There is no error in the execution log and I can see data there in the staging table. But the table in DW is blank. I have tried this with both UAT and Production environments.

Any helpwould be much appreciated.

Thanks

I have the same question (0)

All responses (13)

Answers (1)

Sort by

Suggested answer

Sergei Minozhenko 23,097 on at

Like (0)

Report
Copy link

Link copied!

Hi Maverick

Have you marked the "Skip staging" option for the entity in data project? If not, it will take a lot of time to perform export through staging tables to BYOD.

BR, Sergei

Was this reply helpful? Yes No
Maverick 182 on at

Like (0)

Report
Copy link

Link copied!

Hi Sergey,

Yes I have marked "Skip staging" option.

Was this reply helpful? Yes No
Sergei Minozhenko 23,097 on at

Like (0)

Report
Copy link

Link copied!

Hi Maverick,

And still staging tables populated in D365?

BR, Sergei

Was this reply helpful? Yes No
Suggested answer

Mea_ 60,286 on at

Like (0)

Report
Copy link

Link copied!

Check Azure SQL Database service tier, P2 or higher is recommended, please refer cloudblogs.microsoft.com/.../ for details

Was this reply helpful? Yes No
Maverick 182 on at

Like (0)

Report
Copy link

Link copied!

Yes, data is populated in staging. Please see the screenshot below.

Was this reply helpful? Yes No
Sergei Minozhenko 23,097 on at

Like (0)

Report
Copy link

Link copied!

Hi Maverick,

So "Skip staging" doesn't work. Is it a custom entity (i can't find it the list)? Do you have PostLoad method overridden there? (at least in community.dynamics.com/.../data-entity-export-slow it was the cause for non-working "Skip staging" )

BR, Sergei

Was this reply helpful? Yes No
Maverick 182 on at

Like (0)

Report
Copy link

Link copied!

Hi Sergey, I tried with both custom and standard entity but both seems not working. No, there is no PostLoad method.

Was this reply helpful? Yes No
Maverick 182 on at

Like (0)

Report
Copy link

Link copied!

Hi ievgen, I checked Azure SQL Database service tier and it is P1. Should we try upgrading?

Was this reply helpful? Yes No
Verified answer

Sergei Minozhenko 23,097 on at

Like (1)

Report
Copy link

Link copied!

Hi Maverick

Just to be sure that everything is in place, you can try to regenerate mapping for the entity in the entity list, republish entity to BYOD, create new data project (switch to "Enhanced mode" if it's in "Standard") and add entity again (also double-check if "Skip staging" marked).

Increasing BYOD tier will help only in case if you have I/O bottleneck on BYOD side, but according to provided information, it's because data is inserted to staging on D365 side before exported.

BR, Sergei

Was this reply helpful? Yes No
Maverick 182 on at

Like (0)

Report
Copy link

Link copied!

Sergey, I followed the steps but still issue not resolved. As I understand, DW in BYOD is using the staging tables to hold data. may be that is why skip staging is not working.

Was this reply helpful? Yes No