web
You’re offline. This is a read only version of the page.
close
Skip to main content

Notifications

Announcements

No record found.

News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Forums / Microsoft Dynamics CRM (Archived) / Duplicate Detection Jo...
Microsoft Dynamics CRM (Archived)

Duplicate Detection Jobs; more than 5000 duplicates

(0) ShareShare
ReportReport
Posted on by RaulOcana 8,270

Hi guys,

I have in my database around 150k leads and I want to check for possible duplicates.

I have 2 rules: E-Mail Match or Name Match.

But when I try to run the Job I got this message: "Bulk Detect Duplicate Limit Exceeded. The Bulk Duplicate Detection job cannot detect more than 5000 duplicates. Please review your duplicate rules or resolve existing duplicates and rerun the job"

Is there a way to solve this? Maybe through excel spreadsheet or something like that?

Thanks,

Raul

*This post is locked for comments

I have the same question (0)
  • jlattimer Profile Picture
    jlattimer 24,562 on at

    When running a duplicate detection job, besides the rule you create you can also specify criteria via an advanced find type search before running the job. I'd suggest trying to run the job on a subset of records first and eliminate those duplicates before running against the entire lot.

  • Hosk Profile Picture
    Hosk on at

    what's happening is the criteria you have chosen has brought back more than 5000 duplicates.

    the reason it's telling you is it would take you ages to go through all those duplicates.

    You either need to change your duplicate criteria so you don't find as many duplicates, you can do this by adding another field perhaps.

    OR you need to select a smaller subset of leads to check for duplicates.  You could do this by date or alphabetical.  

    To be honest if you have this many duplicates, it might be quicker to through these in Excel.  The only consideration you need to take into account is if you have done any activities around the leads (e.g. emails, phone activities), you don't want to get rid of those ones.

    Using the duplicate tool in CRM can take a lot of time and if you have this many duplicates I would definitely recommend cleaning the data outside of CRM.

  • RaulOcana Profile Picture
    RaulOcana 8,270 on at

    Hi guys,

    Yes it is possible that there are more than 5000 duplicates, since a time ago a massive upload (150k leads) was made without the duplicate rules published.

    I think that the CRM Online Duplicate Jobs wont work for me here, but how could I do this on an Excel Spreadsheet?

    Thanks,

    Raul

  • Suggested answer
    Hosk Profile Picture
    Hosk on at

    I would put them in an excel spreadsheet and then sort them by something like email address or something else and then see which ones look the same.

    I don't know what columns you have or the quality of your data

    What constitutes a duplicate in your data?

  • RaulOcana Profile Picture
    RaulOcana 8,270 on at

    Yes mostly of them are email duplicates. But once I have them in the excel spreadsheet, how could I delete the old duplicates?

    Thanks,

    Raul

  • RaulOcana Profile Picture
    RaulOcana 8,270 on at

    Is there a way to create a Plugin to make the duplicate (by email) search and update/delete the leads of certain criteria?

  • Verified answer
    Hosk Profile Picture
    Hosk on at

    Yes you could fire a plugin to run on Create of email.

    You could then do query in the plugin to check for duplicates

    I'm not sure you want a plugin deleting records, maybe you could deactivate them.  You could perhaps flag them as duplicates and then create a view for someone to check them and then delete them

    Auto deletion via a plugin sounds like a dangerous piece of functionality

    It sounds to me you want an console app to run through once and clean the current data.

  • RaulOcana Profile Picture
    RaulOcana 8,270 on at

    Yes I'm trying that, I guess this is the only way. Time to read the sdk.

    Thanks,

    Raul

Under review

Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.

Helpful resources

Quick Links

Responsible AI policies

As AI tools become more common, we’re introducing a Responsible AI Use…

Neeraj Kumar – Community Spotlight

We are honored to recognize Neeraj Kumar as our Community Spotlight honoree for…

Congratulations to the November Top 10 Community Leaders!

These are the community rock stars!

Leaderboard > 🔒一 Microsoft Dynamics CRM (Archived)

#1
SA-08121319-0 Profile Picture

SA-08121319-0 4

#1
Calum MacFarlane Profile Picture

Calum MacFarlane 4

#3
Alex Fun Wei Jie Profile Picture

Alex Fun Wei Jie 2

Last 30 days Overall leaderboard

Featured topics

Product updates

Dynamics 365 release plans