Personalized Community is here!
Quickly customize your community to find the content you seek.
Have questions on moving to the cloud? Visit the Dynamics 365 Migration Community today! Microsoft’s extensive network of Dynamics AX and Dynamics CRM experts can help.
2021 Release Wave 2Discover the latest updates and new features releasing from October 2021 through March 2022.
2021 release wave 2 plan
The FastTrack program is designed to help you accelerate your Dynamics 365 deployment with confidence.
FastTrack Community | FastTrack Program | Finance and Operations TechTalks | Customer Engagement TechTalks | Upcoming TechTalks | All TechTalks
Hope you all are doing well.
Here looking for a best solution to address one business need. Business want to send SMSs to customer's to participate in the Poll.
In CRM we have a custom entity to control SMS content, URL and reminder and an external database to manage the transactions.
From CRM side I need to process all records and update the SMS content and Shorten the URL and update the same to an external database. To shorten the url using 3rd party api.
My ask is how to design the solution which can process millions of records without any issue, currently thinking to develop a console app to handle all this requirement.
How to avoid performance issue, If I put this much population in a foreach loop/parallel foreach loop to process the records and perform update on CRM entity record, calling 3rd party api to get shorten url in response & updating in CRM entity and updating external database.
Appreciate your help to design a best solution.
If you are talking about millions of records, you could consider using a queue/service bus based approach to reliably process these records. With this approach, as and when the records become available, you could push it into the queue and multiple services can process items from the queue allowing you to scale as an when you need.
If you are completely On-Prem, and not ready to pay for cloud bases services, you could consider following option
01. You could use RabbitMQ as your queue management system and push all the SMS records that needs to be sent to the queue - you can push (publish) them in batches and to multiple queues - You can have this logic in CRM in plugin or workflow with CWA to push SMS record for processing as and when it becomes available
02. You could then use activity brokers like Hangfire Service(https://www.hangfire.io/) or MassTransit (masstransit-project.com/.../quickstart.html ) to process messages from the queue - This processing could involve updating content and shorting URL and sending to the external database. You can have multiple instances of these running, same server or different servers, all processing items from RabbitMQ queues.
03. Service mentioned I point 2, can also do the updating back to CRM
04. You could consider storing the SMS records in an intermediate database or some indexing and search technologies like Lucene (https://lucenenet.apache.org/)
One thing to consider, try and find ways to batch and push records to queues as and when they are ready for processing. Even if loads of them become available at once to be processed, you can batch them into in let's say 1000s or 10000s
Business Applications communities