Share
Report
16Hi,
I would like your thoughts on a suitable data integration process.
I'm a report developer and not a data integration architect. I work for a small company that doesn't have specialised expertise in data integration. As I know of the ETL/Warehousing principles, this matter is being assigned to me to drive.
Our transactional solution is Dynamics 365. We're a Microsoft house and perform all in the cloud. During the past year, around ten external third-party data sources have been highlighted as needing ingesting into our domain so we can report on the data. We'd only be looking to bring in a subset of third-party data. For example, for one source it would only be 15 columns at around 10,000 rows deep. The other sources wouldn't be that much different. Naturally, our third-party data source list may grow as well as the volume of data per source.
I'm suggesting to use a Dataverse (either the one our Dynamics 365 solution currently uses or a new Dataverse) with Power Platform Dataflows (not Power BI Dataflows). Our company is already familiar with the Power Platform. Our I.T. department supports a Dataverse (i.e. Dynamics 365) and I use Power Query which is the language of Dataflows.
I made this suggestion but our I.T. department (a single individual) suggests using Synapse. This individual is not ofay with the data world and I believe their suggestion is due to having attended a Microsoft seminar where Synapse was promoted (this is an assumption and I'm yet to hear reasons why Synapse is the correct route). If we needed to ingest large volumes of data or unstructured data, then I can see the use of Synapse but for our foreseen few, small datasets, I'm concerned to whether Synapse is suitable for us. I think we need something more tailored for our needs. I don't want our current skillsets or lack of to dictate the chosen data integration method as we can always learn what we don't know.
To help with my understanding, is anyone able to advise what they see suitable based on the little information I've provided?
Is anyone aware of possible difficulties on using either approach?
Does the Microsoft roadmap suggest to use one over the other?
Thanks.
I would like your thoughts on a suitable data integration process.
I'm a report developer and not a data integration architect. I work for a small company that doesn't have specialised expertise in data integration. As I know of the ETL/Warehousing principles, this matter is being assigned to me to drive.
Our transactional solution is Dynamics 365. We're a Microsoft house and perform all in the cloud. During the past year, around ten external third-party data sources have been highlighted as needing ingesting into our domain so we can report on the data. We'd only be looking to bring in a subset of third-party data. For example, for one source it would only be 15 columns at around 10,000 rows deep. The other sources wouldn't be that much different. Naturally, our third-party data source list may grow as well as the volume of data per source.
I'm suggesting to use a Dataverse (either the one our Dynamics 365 solution currently uses or a new Dataverse) with Power Platform Dataflows (not Power BI Dataflows). Our company is already familiar with the Power Platform. Our I.T. department supports a Dataverse (i.e. Dynamics 365) and I use Power Query which is the language of Dataflows.
I made this suggestion but our I.T. department (a single individual) suggests using Synapse. This individual is not ofay with the data world and I believe their suggestion is due to having attended a Microsoft seminar where Synapse was promoted (this is an assumption and I'm yet to hear reasons why Synapse is the correct route). If we needed to ingest large volumes of data or unstructured data, then I can see the use of Synapse but for our foreseen few, small datasets, I'm concerned to whether Synapse is suitable for us. I think we need something more tailored for our needs. I don't want our current skillsets or lack of to dictate the chosen data integration method as we can always learn what we don't know.
To help with my understanding, is anyone able to advise what they see suitable based on the little information I've provided?
Is anyone aware of possible difficulties on using either approach?
Does the Microsoft roadmap suggest to use one over the other?
Thanks.
All responses (
Answers (
