web
You’re offline. This is a read only version of the page.
close
Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Finance forum / BYOD Export large amou...
Finance forum

BYOD Export large amount of data

(0) ShareShare
ReportReport
Posted on by Mpaga 25

Hi,
I have some problems with BYOD export.
It:
- always fails for exports with a lot of data (more than 40 million records for the biggest entity)
- sometimes fails for exports with roughly 3 million records
- always works with small records entities
we are talking about transactional and we are using incremental push.

The BYOD Sql Azure Database is a single DB P1 (high performance I think).
The D365FO test enviroment is based on a tier 1 (here it is where we are doing our tests and maybe it's not enough).

Monitoring the Sql in the tier1 machine, we can see the export query in processing but suddenly, after 80 minutes or something like that, it disappear.
Do you have any useful information and advice? Is this problem just because of the tier1 machine?

Thank you very much.

Categories:
Data Integration
Development / Customization / SDK
I have the same question (0)
  • Verified answer
    Sheikh Sohail Profile Picture
    Sheikh Sohail 6,125 on at

    You Can raise ticket with Microsoft, They will increase the SSIS time out for you.

  • Mpaga Profile Picture
    Mpaga 25 on at

    Thank you Sohail for your answer.

    Using the tier 1 machine we intercepted the sql query and it is like this:

    SELECT N'EXP_Test' AS DEFINITIONGROUP,N'EXP_Test-2018-10-23T08:42:45-54561771943D4136A75507D09E8B92F6' AS EXECUTIONID, ...'10/23/2018 08:42:45' AS SYNCSTARTDATETIME,T1.RECID AS RECID, ISNULL(T1.REFRECID, 0) AS REFRECID  
    FROM DLX_BI_GENERALJOURNALACCOUNTENTRYENTITY T1
    WHERE (PARTITION=5637144576)
    ORDER BY T1.REFRECID

    We tried to run this query in Management Studio and it gave us the message that there was not enough disk space.
    Do you think that asking for an increase of the timeout we could have a successful exportation?
    Thank you very much for your help.

  • Suggested answer
    Sheikh Sohail Profile Picture
    Sheikh Sohail 6,125 on at

    Definitely Not, if you are facing this issue On one-box machine then Increase the HDrive and try it again. But I am 100% Sure on Azure(Microsoft Subscription)  HDrive is not an issue especially on 2-Tier Environment.

  • Mpaga Profile Picture
    Mpaga 25 on at

    In this case the one-box machine is a 1-tier environment in LCS.

    So do you think that the disk space is not a problem for a 1-tier environment?

  • Suggested answer
    Sheikh Sohail Profile Picture
    Sheikh Sohail 6,125 on at

    On Azure One Box (Dev-Box MS Subscription) Sql HDrive volume  they keep between 60 GB to 80 GB

    I don't think so 40 Million records will take 61 GB Hardirve Space, however will recommend you to increase hard-drive on your local environment and then check next error. 

  • Mpaga Profile Picture
    Mpaga 25 on at

    I think you're right.

    This weekend we'll have the chance to try these custom entities in a 2-Tier UAT environment.

    If everything works I'll mark the question as answered and solved.

    Thank you in advance.

  • Mpaga Profile Picture
    Mpaga 25 on at

    Hi Sohail,

    we tried the export job in a 2-tier environment and the result for the biggest entity is 

    "The request channel timed out while waiting for a reply after 01:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout."

    Timeout-message.png

    We raised a ticket to Microsoft few hours ago. I will try again and then I'll consider the question as answered and solved.

    Thanks again for your help.

  • Suggested answer
    Sheikh Sohail Profile Picture
    Sheikh Sohail 6,125 on at

    Hi Mpaga,

    Thank you for the update. One more thing while exporting the records make sure check box skip staging set as YES.

  • Mpaga Profile Picture
    Mpaga 25 on at

    You were right. In Dev and UAT environments there are 2 config files to set with a new timeout value. In Prod only Microsoft can do this.

    Thank you

  • Suggested answer
    Sheikh Sohail Profile Picture
    Sheikh Sohail 6,125 on at

    You are welcome.  It was known issue for me, we are running  data warehousing on one of the biggest database in D365FO via BYOD.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

November Spotlight Star - Khushbu Rajvi

Congratulations to a top community star!

Forum Structure Changes Coming on 11/8!

In our never-ending quest to help the Dynamics 365 Community members get answers faster …

Dynamics 365 Community Platform update – Oct 28

Welcome to the next edition of the Community Platform Update. This is a status …

Leaderboard > Finance

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans