Issues with duplicate detection

(0) Share

Report

Posted on by GordonJohnston

I am trying to set up a robust proactive process for duplicate contacts.

In Dynamics we have 2 rules based on email address and the other on mobile phone. If I run this against active contacts then I fine 14 duplicates.

I also have a C# application which uses the BulkDetectDuplicatesRequest. For that I am supplying the email address rule only and I am receiving 55 duplicates.

When I open each record in turn from the 55 records. Some of them will pop up the Merge dialog while others do not.

Finally, thought I had a choice of solutions for sending an email that I could present: OTB or my Console App.

However when I showed the results, someone was able to show be an email address that had 15 duplicate copies. Thus I suspect there are loads of duplicates rather than the small number I am finding vie the two methods above.

Is the functionality broken? Or am I missing something in my approach?

As I recall, there is some weird index thing that is generated in the back end. Maybe this is wrong, stalled, broken, not covering all Contact records???

Any advice?

All responses (7)

Answers (1)

Verified answer

Nya on at

Like (0)

Report

RE: Issues with duplicate detection

Hi,

I assume that the 15 duplicate copies you are referring to are the effect shown below, right?

If yes, this pop-up screen will list all records that are duplicates of that record.

As an example, when saving A it shows duplicates with B, C and D. When saving B, it will also list A, C and D.

So the total number of duplicate records is not the sum of the duplicate records shown for each one, where there will be an overlap.
GordonJohnston on at

Like (0)

Report

RE: Issues with duplicate detection

That is useful to know.

So the duplicate total and the results from code represent only the top box and do not include all the other records that they are matching.

Good to know.

However, with the 15 duplicates that I mentioned. Not a single one of them are included in either the OTB results or my code.

All 15 records are active but I am not getting one of them reported in my total for some reason.

I am going to double-check my code this morning. Maybe I missed it.
GordonJohnston on at

Like (0)

Report

RE: Issues with duplicate detection

Just found that the OTB interface shows 1-12 but there are more pages so I think this is working fine. Pain that it isn't more clear in showing the total. Maybe my code is returning a limited data set as well. Will get to the bottom of it.
Nya on at

Like (0)

Report

RE: Issues with duplicate detection

Hi,

I look forward to hearing from you about your test results and if there is still a problem, please don't hesitate to let me know!
GordonJohnston on at

Like (1)

Report

RE: Issues with duplicate detection

Getting more results from my code today. I wonder if there was some indexing going on in the background because one of the rules changed yesterday.

I can see the record I am expecting in the list now. It is there twice and I guess I will be able to merge them together and then into the other 15 (likely that will be by hand anyway). Possible there are 2 because there are 2 contact rules?!

So my total will NOT be the total duplicates but the de-duped total of unique duplicates (hope that makes sense for the next person). Would like to get the full total as well so will research that.
Kristina_Skyline on at

Like (0)

Report

RE: Issues with duplicate detection

Hello,

You should take a look at a new user-friendly De-Dupe solution published on AppSource! Depending on your need it allows you to set up duplicate detection rules using standard Dynamics 365 functionality, but to review and merge duplicates in a completely renewed and comfy interface.

No more old, user-incomprehensible list of duplicates repeating with each merging step. Plus, since it has just been launched on AppSource, the solution is FREE of charge and includes support.

Feel free to visit Skyline Consultants website for more details skylineconsultants.com/.../.

Best,

Kristina
Inogic on at

Like (0)

Report
Issues with duplicate detection

Hi,

You're not alone. Dynamics 365's duplicate detection can behave inconsistently depending on how it's triggered and the data state.
A few things to consider:

Mismatch in duplicate counts often comes from how rules are applied. The out-of-box detection via UI applies all active rules, while your C# app using BulkDetectDuplicatesRequest may only apply one (email). That explains why you're seeing more hits programmatically.

The merge dialog not appearing could mean the record doesn’t match any duplicates under currently active rules, or the user lacks merge privileges. Dynamics also doesn’t always show the merge dialog if the duplicates aren't part of the same match set.

You're correct about backend dependencies; duplicate detection relies on system indexing and async jobs. If records were imported or updated in bulk, the detection index may not have caught up. Republishing the rules and triggering a manual duplicate detection job can help.

Lastly, if you're finding multiple records with the same email still slipping through, it might be time to look beyond exact-match rules. We’ve seen better results with tools that support fuzzy matching and bulk merge workflows. You can try DeDupeD by Inogic can help uncover and resolve deeper issues without interfering with the native system behavior. It lets you define your own merge logic (like keeping the most complete record or combining field values) and run cleanup jobs in batches, especially useful if you’re dealing with legacy data or complex imports.
Hope this helps, happy to share example rule setups if needed.

Thanks!