web
You’re offline. This is a read only version of the page.
close
Skip to main content
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Finance | Project Operations, Human Resources, ... / Push Data Integrations...
Finance | Project Operations, Human Resources, ...
Unanswered

Push Data Integrations with Az services like ServiceBus/EH to receive live streaming data from D365 installed in Azure

(0) ShareShare
ReportReport
Posted on by Dynamic_365_ 10

Hello Good Folks

Looking for a way to receive data out of D365 F&O in Azure environment, continuous streaming of data to our Az services like EH/ServiceBus etc (not DataLake) preferably in json format. 

Not batch job format, but live streaming for all incremental updates

So far came across 

1. batch jobs which will send csv files for every hour or so

2. F&O module add-in to stream data into datalake (for reporting purposes). Good but no datalake and no csv format desired for our sol.

3. odata (again this is pull mechanism from outside env. ) and other apis. 

Desired solution

Push mechanism from D365 (in azure) for all incrementatl data directly to Azure services like SB/EH etc.

Thanks in advance.

I have the same question (0)
  • Umer Draz sheikh Profile Picture
    Umer Draz sheikh 15 on at
    RE: Push Data Integrations with Az services like ServiceBus/EH to receive live streaming data from D365 installed in Azure

    Your best bet here is to leverage business events and data events

    learn.microsoft.com/.../home-page

  • Martin Dráb Profile Picture
    Martin Dráb 236,863 Most Valuable Professional on at
    RE: Push Data Integrations with Az services like ServiceBus/EH to receive live streaming data from D365 installed in Azure

    If you want automatic export to an Azure service, Azure Data Lake seems to be exactly what you need. That it uses CSV internally doesn't mean that you can't extract data in another format.

    ServiceBus is for communication based on (short) messages; not for export of (relatively) large files.

    It'll be good if you tell us more about your business scenario, because you're talking many contradicting approaches and it's not clear what requirements the solution must meet.

  • Dynamic_365_ Profile Picture
    Dynamic_365_ 10 on at
    RE: Push Data Integrations with Az services like ServiceBus/EH to receive live streaming data from D365 installed in Azure

    Thanks Martin for the response

    This is the requirement.

    1) Data changes recived from D365 F&O module (in azure) needs to be transmitted to downstream systems - to be made available for business processes

    2) While data is transmitted , need to perform some transformations. downstream systems expect json format.

    3) Live streaming is needed.

    4) Might need to persist a copy of the data in sql. (optional)

    5) Earlier i implemented database mongodb changestream to be sent to EventHub, which was read by functions to persist in SQL.

    Thinking if we can model the D365 data streaming also in the same approach.

    Hence if there is a in-built streaming capacity to EventHub/SB the handshake between D365 and Az service will be neat.

    Also please confirm if there is only one out-of-the box integration with Az service, that is Datalake from D365 and no other integrations are available to EH/SB etc

    Thanks in advance

  • Martin Dráb Profile Picture
    Martin Dráb 236,863 Most Valuable Professional on at
    RE: Push Data Integrations with Az services like ServiceBus/EH to receive live streaming data from D365 installed in Azure

    You need to think harder about the actual requirements. For example:

    1) Consider what data you need. All you said so far was "all incremental data", but that's a nonsense. There is no point, for example, to inform external system every time when a RecId assigned from SystemSequences table. If you believe you need everything, it means that you haven't analyzed the requirements yet. Having some idea about the number of entities and the amount of data is important to be able to choose a solution.

    2) Think about time-related requirements. For example, if you receive information about data changes a minute after the transaction was committed, is it a problem for anything?

    3) You keep talking about streaming, but you didn't explain what you actually mean by that. Do you want to say that you don't want any queue mechanism and if the target system gets more data than it can process, it'll miss some data in the stream? You failed to provide any information about your business scenario, but I doubt this is what you want. If you process incremental changes and you miss some of them, you don't know the state of the system anymore. I think you'll likely need a queue and process everything. When you get more data than you can process at a time, it just means that the queue get longers and time between raining the event and processing it will increase.

    Regarding your last sentence, you seemed to ignore Muhammad's reply, otherwise you wouldn't ask such a question. Please look at busines events.

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Pallavi Phade – Community Spotlight

We are honored to recognize Pallavi Phade as our Community Spotlight honoree for…

Congratulations to the October Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > Finance | Project Operations, Human Resources, AX, GP, SL

#1
André Arnaud de Calavon Profile Picture

André Arnaud de Cal... 734 Super User 2025 Season 2

#2
CA Neeraj Kumar Profile Picture

CA Neeraj Kumar 636

#3
Martin Dráb Profile Picture

Martin Dráb 553 Most Valuable Professional

Last 30 days Overall leaderboard

Featured topics

End-of-life Announcement for Microsoft Dynamics GP

Product updates

Dynamics 365 release plans