web
You’re offline. This is a read only version of the page.
close
Skip to main content

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Microsoft Dynamics 365 | Integration, Dataverse...
Suggested Answer

Best practices for near-real-time data synchronization using RCT in D365 F&O

(0) ShareShare
ReportReport
Posted on by 22

Hi Experts,

We are evaluating replacing scheduled batch exports from Dynamics 365 Finance & Operations with near-real-time synchronization using Row Version Change Tracking (RCT) through Dataverse virtual tables.

Our architecture is:

D365 F&O → Dataverse virtual tables → custom polling adapter → Message Broker → Azure SQL Managed Instance

The adapter performs an initial load, then retrieves incremental changes using delta links at a configurable polling interval, such as 10 seconds. We understand that polling frequency does not guarantee the same end-to-end latency.

The planned scope is approximately 120 source tables, with several containing 20 million+ rows each. These are existing row counts; peak change rates are still being assessed. The goal is to maintain current data for downstream reporting, including inserts, updates, and deletes.

We have validated a limited PoC and would appreciate guidance:

  1. Is this a recommended production pattern, and is it commonly implemented at a comparable scale?
  2. What F&O performance impact, API limits, and throttling should we consider when choosing polling frequency and concurrency?
  3. What are the best practices for large initial loads, delta-token expiry, and recovery after prolonged RCT adapter outages?
     

Any real-world experience or Microsoft guidance would be appreciated.

Thank you in advance!

  • Suggested answer
    Patrick Mouwen Profile Picture
    215 on at

    Hi,
     

    I will try to cover all of your questions:
     

    1. Change tracking on F&O virtual tables is still a preview feature: https://learn.microsoft.com/dynamics365/fin-ops-core/dev-itpro/power-platform/track-changes-fin-ops-virtual-table
     

    2. Every poll is a live query in F&O. Dataverse forwards each request to a custom F&O service, which runs it as a query on the data entity on the AOS and the F&O database. The good thing is that F&O exempts these calls from its own throttling (throttling happens with ODATA and custom end points), but that doesn't make them free. They compete with your users and batch jobs and will get slower when F&O is busy. The throttling you will hit comes from Dataverse, on your adapter user (6,000 requests and 20 minutes of execution time per 5 minutes, per web server). 120 tables every 10 seconds is already about 3,600 calls per 5 minutes before paging.

    3. The delta link is Dataverse's own change tracking, with a token that comes from F&O itself. The initial load goes through the same route: the first call without a token is a full read of the table through F&O, 20M+ rows paged. Tokens are valid for 10 days by default, so after a longer outage you're looking at that full read again. Plan it as an event, not a retry. Change tracking requests also don't support $filter, so you always get the whole table.
     

    What I would need to further help you:
    - What does near-real-time mean for you: seconds, or a few minutes?
    - What consumes the data in Azure SQL MI: reporting, an app, or downstream processes that act on the changes? For the last one you'd want change events rather than polling a copy.
    - Is it one-way, or does anything need to flow back into D365?
     

    For reporting alone, Link to Fabric on the F&O tables is the standard route, with latency in minutes: https://learn.microsoft.com/power-apps/maker/data-platform/fabric-link-plan-latency. 

    For the other scenarios the answer changes quite a bit.
     

    Kind regards,
    Patrick Mouwen

  • AS-21081215-0 Profile Picture
    22 on at

    Hi @Patrick Mouwen ,

    Thanks for the explanatin. Let me share the background and answr your questions.

    The customer currently uses scheduled batch export jobs approximately every 30 minutes to move data from F&O to SQL MI. They face several challenges: F&O jobs get queued, downstream systems are busy processing other requests, and when batch jobs fail, data may not reach downstream applications and requires manual follow-up even sometimes reexport all the tables data (initial load again which is make the customer suffering). Sometimes message lost can also a problem for audit purposes.

     

    That is why we proposed moving to real-time delivery through a message broker. The goal is to prevent message loss, retain messages while downstream systems are unavailable, let applications consume at their own pace, and make business changes available as they happen instead of waiting for the next batch.

    Regarding your questions:

    1. We actually need real-time data. F&O does not provide a straightforward CDC mechanism for our needs. We explored Data Events, but those also have throughput limitations. This is what we find confusing: we need real-time changes, but the available mechanisms have limitations. The community then recommended row version change tracking through Dataverse virtual tables, so we tried that approach with the polling interval set as low as possible to achieve near-real-time delivery.
    2. It is mostly downstream processes that act on the changes, not just reporting. Various applications consume the exported data from SQL MI to support their business processes.
    3. The direction is one-way, from F&O to SQL MI. There is no write-back to D365.
       

    regards,
    Abed

  • Suggested answer
    Patrick Mouwen Profile Picture
    215 on at

    Hi Abed,
     

    Thanks, that 's helpful input. Since downstream processes act on the changes, I'd split this in two, because one mechanism won't serve both well.
     

    1. Keeping SQL MI current (the copy). Couple of things to consider:
    -Row version change tracking gives you the latest state per row, not every change: updates between two polls are merged, and every table has its own token, so a header and its lines can arrive in different polls. 
    -The feed only returns changes up to the oldest transaction that's still open in the F&O database, so a long posting or planning job holds back the feed for every table until it commits. 
    Deletes come from the delete log of the entity's root table, so a delete in a child data source doesn't come through as a delete. 

    For the copy itself, the Link to Fabric route I mentioned still applies if minutes are acceptable. Note that I am not a big fan (yet) of this route due to uncertainty when the data will actually arrive (can be delayed in case of larger data sets) - It's SaaS, so no in your own control.
     

    2. Reacting to changes (the processes). Here you want events rather than polling: business events or data events into Service Bus, for the subset of tables where a process really has to react within seconds. The event tells the consumer what changed; the consumer reads the current record from SQL MI or F&O. Service Bus gives you exactly the retention and own-pace consumption you're after. Data events have throughput limits, so keep them to the tables that need it.


    Two questions decide the further design I'd say: how many of the 120 tables really need seconds rather than minutes, and what peak change volume do you expect on those?

    Patrick

  • AS-21081215-0 Profile Picture
    22 on at

    Hi @Patrick Mouwen 

     

    Thanks for the response. 

     

    1. Could you elaborate a bit more regarding this → “table has its own token, so a header and its lines can arrive in different polls. ” ?
      I mean with current built in data export feature in FO to MS SQL Managed Instance will having the same issue like stated above right?
    2. Yes, I'm also thinking of data events but unfortunately it has performance limit issue. Could we combine business events + RCT? Business events trigger the RCT to get request for some new changes

     

    Thanks.

     

    regards,

    Abed

  • Suggested answer
    Patrick Mouwen Profile Picture
    215 on at

    Hi Abed,

    Sure, an example: a sales order is created at 10:00:05, header and three lines in one transaction. Your poller reads the header table at 10:00:00 (nothing new yet) and the lines table at 10:00:07. The lines arrive in this cycle, the header only in the next one. Anything that reads SQL MI in between sees lines without a header.

    There's no shared snapshot across tables: each table has its own token, read at its own moment. In practice it sorts itself out within a cycle or two, but downstream logic has to tolerate it, e.g. park lines whose header isn't there yet and pick them up in the next cycle.
     

    BYOD: yes, in principle the same. Its incremental export uses SQL change tracking per entity (not row version change tracking), and each entity is exported on its own, so the target can briefly hold lines without their header. 

    Microsoft mentions this consistency challenge in a BYOD article and suggests "Enable triggers in target database" to detect when the sync is done before you transform data downstream: https://learn.microsoft.com/dynamics365/fin-ops-core/dev-itpro/analytics/export-entities-to-your-own-database

    With a 30-minute batch you rarely notice it. The faster you poll, the more often consumers will run into it.


    Business events + RCT: yes, I think that's the most practical middle ground. Use the event as a trigger and RCT as the data. The event only says "something changed in table X"; the consumer then calls the change feed for that table with its stored token and gets everything since the last call in one go.

    What makes it work:
    1. Debounce: one pull per table per few seconds, however many events come in. That also brings your Dataverse request count down a lot compared to polling 120 tables every 10 seconds.
    2. Keep a slow scheduled sweep (e.g. every 5-15 minutes) over all tables as a safety net. A missed or late event then only costs latency, not data.
    3. An empty pull isn't proof that nothing changed: the feed is held back by the oldest open transaction, so the next doorbell or sweep picks it up.
    4. Business events are raised by business processes (e.g. an invoice posted, a workflow step), not by every table change, so check the catalog for your tables. If you use data events as the doorbell instead, put them on header entities only. The documented limits (5,000 events per 5 minutes, up to 50,000 per hour per environment) then count orders, not lines.

    Patrick

  • André Arnaud de Calavon Profile Picture
    308,288 Super User 2026 Season 2 on at

    Hi @AS-21081215-0,

     

    You discussed a lot of technical details with Patrick. I have some concerns. What are these 120 tables about? What number of records do you expect to be synchronized in total? What type of product is your “downstream application” or multiple applications? Do you really need data in an Azure SQL server? Is e.g. an Azure Data Lake also an option to consider?

    Have you considered using the Fabric link? Note that this is not real time syncing the data.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Season of Sharing Community Challenge Winners!

Congratulations to our community stars!

Women in Power Builds Momentum

Expanding mentorship, skilling, and AI innovation

Leaderboard > Microsoft Dynamics 365 | Integration, Dataverse, and general topics

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 39 Super User 2026 Season 2

#2
11manish Profile Picture

11manish 37 Super User 2026 Season 2

#3
Subra Profile Picture

Subra 24 Super User 2026 Season 2

Last 30 days Overall leaderboard

Product updates

Dynamics 365 release plans