Dynamics Community Forum Thread Details

Hi James,

there are several approaches; some by Out-Of-the-Box (OOB) technologies others by coding Developments (Plugins / Scripting).

Standard Solutions:

MANAGE DATA: you could take advantages by detect duplication: https://docs.microsoft.com/en-us/power-platform/admin/detect-duplicate-data
MANAGE DATA: you could leverage to duplicate detection rule during on create / update and also during import records: https://docs.microsoft.com/en-us/power-platform/admin/set-up-duplicate-detection-rules-keep-data-clean
MANAGE DATA: you could leverage to detect duplication jobs in order to detect duplicated record and manually merge them or choose other actions record by record (deactive, delete etc..): https://docs.microsoft.com/en-us/power-platform/admin/run-bulk-system-jobs-detect-duplicate-records
PREVENTION: you could find one or more field that keep unique value(s) e set them as alternate keys. This will prevent the creation of two records with the same key value field: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/define-alternate-keys-portal
PREVENTION: provide good views to your user to display the main data, this reduce confusion and bring data clean data. Use view also to monitoring data import.
MANAGE DATA: provide a KB procedure to your users in order to use merge button feature: https://docs.microsoft.com/en-us/powerapps/user/merge-duplicate-records
PREVENTION: clean data by calculated fields to check in parent-field hierarchy some duplicates. Also you can trim to clean the values: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/define-calculated-fields
MANAGE DATA: use aggregate statistics to find duplicates by roll up fields: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/define-rollup-fields
MANAGE DATA: monitoring your dato also with export Reports and Excel Templates. In Excel you can set rules criteria to see duplicates value by column.
PREVENTION: by a security you can limit the actions/rules per user based on his role inside the organizations. The granularity work well also to field level permissions by field levels security: https://docs.microsoft.com/en-us/power-platform/admin/field-level-security
PREVENTION: validation data BR might reduce the duplicates: https://docs.microsoft.com/en-us/powerapps/maker/model-driven-apps/create-business-rules-recommendations-apply-logic-form

These considerations can solve the points:

Need to be able to filter from a custom fields if necessary
Need to be able to manually review merge accounts initially to ensure dedupe logic functions
Need solution to ultimately be automated with scheduling rules

Process Solutions:

PREVENTION: take time to define Together your users one way to use the platform by functionals procedures and flows. This reduce errors import/managment data.
MANAGE DATA: you could leverage the Power Automate flows for schedule duplicate comparison criteria (OData query and fetchXML can help you during export a CDS List)
MANAGE DATA (ISV): release one SSIS ETL ad hoc, for example by the adapter KingswaySoft https://www.crmkonsulterna.se/handling-duplicates-with-ssis-and-kingswaysoft/

one old sample video: https://www.youtube.com/watch?v=mOP8F8vKrf8

PREVENTION: you shoud find on marketplace ISV solutions about intelligent dedupe, for example:

https://automerge.net/features/

https://appsource.microsoft.com/en-us/product/dynamics-365/cowia-solutions.62b0180b-ddc2-4453-86b3-a4edce5757a6?tab=Overview

Have you an Azure Subscription? You can dedupe your data by Azure FaaS functionalities: develop a cloud computing architecture according your dedupe criteria:

These considerations can solve the points:

Need to be able to filter from a custom fields if necessary
Need to be able to set rules for the record that wins vs loses
Need to be able to manually review merge accounts initially to ensure dedupe logic functions
Need solution to ultimately be automated with scheduling rules

Code Solutions:

PREVENTION: the UX on form developed by JS scripting; the show/hide of the only necessary fields or its validation according to the user behavior and his role inside the organization.
PREVENTION: you can register one plugin C# on create/update record event; this plugin detect duplicates and prevent the record creation/update: https://docs.microsoft.com/en-us/dynamics365/customerengagement/on-premises/developer/detect-duplicate-data-for-developers
PREVENTION: you can also create several Workflows custom activity and, or on-demand or by the proposal events, you could query all records via desired criteria by C# LINQ or lovable MS fetchXML
MANAGE DATA: you could create your custom UI panel/wizard detect duplicates by Power Apps. Create your app configure dedupe criteria and then launch respectives duplicates detection processes.

Development take more effort and time but resolve all your requirements pointed above.

I hope I was helpful,

David

Community site session details

Deduping/Merging Accounts and Contacts in bulk

Helpful resources

News and Announcements

Quick Links

Responsible AI policies

Abhilash Warrier – Community Spotlight

Congratulations to the September Top 10 Community Leaders!

Leaderboard > Customer experience | Sales, Customer Insights, CRM

Featured topics

Product updates