Our team at TechLabs London have recently faced a fairly complex and bewildering issue on Dynamics 365 v9. We have setup access team templates against the Contact entity to manage access to Contact records individually. If you haven’t used Access Teams before, you can read more here, but in essence it is a way to control which user can see which contact at a record level.

Our customer has converted from another vendor (a.k.a SF) to our iProperty Cloud solution (http://iProperty.Cloud) built on Dynamics 365 v9 and the new Unified Client Interface (UCI). As part of our data migration, we imported thousands of contacts to Dynamics 365 Customer Engagement and allocated a number of users for each contact record based on a predefined list and set of rules. This was a custom data migration process to create these record level access records using our own TechLabs London propriety migration solution.

Once our data migration into Dynamics 365 was complete, we checked Access Team security and everything looked absolutely fine. The next day after going live, Team access security disappeared for few thousand contacts – but NOT all contacts. Upon looking at the POA table (Principle Object Access Table) where security is set, we found that about 4K access records have disappeared. Literally disappeared!

Apologies for the long story but I want you to be aware of everything we tried to resolve this issue as one of these steps may help you fix your issue (which may or may not be similar to ours).

Basically, we spent the following few days trying lots of different resolutions to try to find out the issue.

  1. First we re-imported Access team records using our migration App – everything went back working fine until the next day in the morning – 4K access records disappeared again!
  2. We tried setting the record level security using “Share” not Access Team (manually and programmatically). While we know that Access Teams are “hidden share” and both get created in POA table, we thought this may solve the issue. It didn’t – next day, everything disappeared.
  3. One possible cause of these record level access security issues could be triggered by re-parenting. For example if contacts are associated to an account and you give ownership of this account to a team while the relationship behaviour is set to cascade all, then the child contact records will become accessible to this team. We removed any cascade relationship behaviour – but still didn’t help. Next day, all access disappeared.
  4. We then Disabled all our custom plugins that remotely affect security and access – we had few automation on security between different entities and teams. No luck.
  5. Also we disabled our own security roles synchronisation process (Azure function) which synchronises security between Dynamics 365 and SharePoint security. Nope! Not even that.
  6. We deleted and recreated the team access template then re-imported all access records. Still same issue. All 4k access records were deleted from POA table the next morning.
  7. We setup an hourly extraction routine to extract Access records from POA table to a separate Azure SQL. We monitored and tried to find out the issue or the cause – still nothing.
  8. After blaming ourselves and our code for good few days, we raised a ticket to the Dynamics 365 Support and Product Team.

Finally, the issue was found to be related to the Table: SubscriptionTrackingDeletedObject table and an associated cleanup process that runs every day in the morning (our UK time) in the Dynamics 365 Platform. Apparently this table includes a list of GUIDs for records that need to be deleted from the POA table (and other locations). This table is not exposed via API (or at least we are not aware of a way to).

Basically, it came down to the fact that we imported contacts before go live the first time with these contacts original GUIDs as they were coming from an older CRM system and we wanted to maintain the relationship between entities and records. This is normally fine, however, at one point before going live we had to make some changes to the structure so we deleted those imported contact records and re-imported them with the same GUIDs again. When we did that, apparently these records and all their related security access records were marked for deletion from the POA table. When we re-imported, these contact GUIDs and their related access records were still marked for deletion. Hence, every day in the morning the good old record deletion clean up process SubscriptionTrackingDeletedObject would delete our access records and drive us crazy.

The fix was a script created and run by the Dynamics 365 support team which permanently fixed the issue for us but not before we have learnt a few hundred ways of how to try to fix such issues!