Portal Troubleshooting, Part Four: Cache Invalidation and Web Notifications

Issue: Data is not being cached properly and/or the search index is not being updated regularly on my version 7.x portal, or the organization I'm using has version 7.x plug-ins firing when they shouldn't be. I'd also like to better understand how the portal refreshes the cache and search index, how I can be sure that my environment is operating efficiently in regard to these features, and how the associated mechanisms can affect the overhead of my portal and organization.

Important: A vast majority of this article pertains to version 7.x portals and to those Dynamics CRM/365 organizations that still contain solutions and other content from a previously deployed v7.x portal. If you’re unsure what version of portal you’re using, navigating to the portal in a web browser with the /_services/about/ path appended to it will take you to a page where the version number is prominently displayed.  This is the primary version of your portal, and the solutions installed in your organization may carry different version numbers.

In an earlier blog post in this series, I discussed one of the most common causes for stale data for the latest generation of cloud-based (SaaS) portals. While that continues to be an important configuration callout, today I want to expand on that discussion further and provide a better overview of how caching works, as well as other implications it can have on the behavior of your Dynamics CRM/365 organization and the portal itself.

When a portal is initially deployed and users begin to browse its contents, the data that is rendered on the website (Page Copy, Content Snippets, Entity Lists, and so forth) is directly retrieved from the Dynamics CRM/365 organization that the portal is associated with. Each time this information is retrieved, it is cached locally within the IIS Worker Process (W3WP) to ensure that subsequent requests for the same information are much faster.

However, this also introduces the challenge of making sure that the cache always reflects the latest information that is stored in Dynamics CRM/365. To address this, the portal is designed to listen for notifications that records have changed. When such a notification is received, the portal "invalidates" older cache entries, and then queries the organization for the latest information. Changes committed directly on the portal will immediately be reflected, but some mechanism is required for those records that are modified within the Dynamics CRM/365 application via the web browser, external SDK requests, and other means.

The components that maintain the cache differ depending on the version of the portal. The older portal release (version 7.x) uses a plug-in called Adxstudio.Xrm.Plugins.WebNotificationPlugin (hereafter referred to as the "Web Notification plug-in"), which fires whenever there are relevant record changes made within the Dynamics CRM/365 organization. The newer SaaS portal release (version 8.x or higher) eschews the plug-in approach entirely and instead subscribes to change notification messages being sent to an Azure Event Hub from the organization. When a modification is detected in either scenario, the relevant cache entries are invalidated by the portal, and the organization is queried for the latest data.

Both approaches are effective and accomplish the same thing, but the former scenario places a variable amount of overhead on the Dynamics CRM/365 organization. The plug-in can be enabled for any entity within the organization, and every time it fires, a system job is created that sends a web request to a user-specified URL within a Web Notification URL record that - if it's typed in correctly - references the HTTP handler Cache.axd. This handler is designed to receive these requests and update the cache accordingly, and a specific Web Notification URL value of http://[domain name]/Cache.axd should be used to accomplish this.

A SearchIndex.axd HTTP handler is also available, and it is highly recommended that a Web Notification URL record be present that calls this handler, as well. By having two Web Notification URL records present that feature the Cache.axd and SearchIndex.axd handlers, both the cache and search index are refreshed every time the plug-in fires and sends the relative notifications.

If you installed a version 7.x portal within a Dynamics CRM/365 organization at some point in the past, there are many changes that could have been made since that initial deployment that would be causing problems or reducing performance, and such issues may not be immediately apparent without further investigation.  I’ve summarized the key scenarios that can occur and how to address them in the resolution section below.

Resolution: Consider the following scenarios:

• Scenario A: Any version 7.x portals that were originally present have been moved to different organization(s), but since the Web Notification plug-in was never disabled in the original organization, it is still firing and causing unnecessary overhead.
• Scenario B: A Web Notification URL record was created that contains an incorrect value, and therefore the Web Notification plug-in is sending web requests to a URL that is not triggering any actions.
• Scenario C: The Web Notification plug-in was registered to fire on all entities (this is the default configuration for Adxstudio Portals Base solutions prior to version 7.0.0086) or on more entities than are required for portal use, which is causing unnecessary overhead.
• Scenario D: A version 8.x portal has been deployed to the same organization that a version 7.x portal was also deployed to, and therefore the Web Notification plug-in is still firing and causing unnecessary overhead. Technically, deploying v8.x on top of v7.x is unsupported, as having duplicate cache invalidation mechanisms will also impact performance.

If the Web Notification plug-in is firing unnecessarily, it has the potential to create hundreds, thousands, or even millions of asynchronous system jobs (depending on the activity level of the organization) that will negatively impact other system jobs and overall performance of the Asynchronous Processing Service. This will also lead to a proportionate increase in the size of the organization's AsyncOperationBase table, causing unnecessary storage consumption.

In all of these scenarios, the presence of any activated Web Notification SDK Message Processing Steps and the values of any Web Notification URL records should be validated to ensure that the organization is operating as efficiently as possible. The actions necessary to validate both sets of records are explained in greater detail within the sections below.

---

Web Notification SDK Message Processing Steps with Adxstudio Portals Base Solution Version 7.0.0086 or Higher

Note: These instructions specifically pertain to those customers who have the Adxstudio Portals Base solution version 7.0.0086 or higher installed.  This solution will be present in all v7.x portal environments, but version 7.0.0086 was first packaged with Adxstudio Portals version 7.0.0023.  This solution's update history can be found here.

1. As an administrative user, navigate to Settings -> Solutions and double-click the row containing the "Adxstudio Portals Base" solution to open the "Adxstudio Portals Web Notification Configuration" page.
2. If the configuration page contains the text "To configure web notification, you must first create Web Notification URLs", click the hyperlink to navigate directly to the Active Web Notification URLs view and create at least one record. Once a record is created, refresh the configuration page.
3. If Web Notifications have never been set up before, click the "Enable Notifications" button to activate the plug-in and to take the interface out of read-only mode.
4. The configuration page contains two lists of entities - "Changes Ignored" and "Send Notifications" - separated by directional arrows. Highlighting entities in either list and clicking the corresponding arrow will enable or disable that entity for Web Notifications, once applied.
5. Ensure that the "Send Notifications" list only contains those entities that are necessary for your portal to function properly by adding/removing them as needed. When finished, click "Save and Publish" to let the platform handle the SDK Message Processing Step enable/disable work.

Please keep in mind that the "Save and Publish" action stores the Web Notification URL values into each plug-in step, and any changes to the Web Notification URL values after this action won't be immediately reflected. If changes to the Web Notification URL records and/or their values need to be made at some point in the future, this configuration page should be re-opened and the "Save and Publish" action performed again.

---

Web Notification SDK Message Processing Steps with Adxstudio Portals Base Solution Version 7.0.0085 or Below

Note: If your organization contains the Adxstudio Portals Base solution version 7.0.0085 or below, we strongly suggest upgrading to the latest build of Adxstudio Portals version 7 to take advantage of performance improvements and the "Adxstudio Portals Web Notification Configuration" page that was added to streamline changes to the Web Notification plug-in.  At the time of this writing, the latest build of Adxstudio Portals version 7 was 7.0.0024, and it can be downloaded here.  If you upgrade your Adxstudio Portals Base solution to version 7.0.0086 or above, the previous instructions should be referenced.

1. If an upgrade is not feasible at the current time, as an administrative user, navigate to Settings -> Customizations -> Customize the System -> SDK Message Processing Steps.
2. In the SDK Message Processing Steps view, click the filter icon on the right side of the view's column headings.
3. Click the down arrow on the "Event Handler" column, choose "Custom Filter", select "Contains", and enter "WebNotification" as the filter value.
4. Click the down arrow on the "Status" column and check the box next to "Enabled".
5. Review the registered steps that appear in the view to ensure their accuracy. Note that versions of Adxstudio Portals that include an Adxstudio Portals Base solution of version 7.0.0085 and prior only included steps that were registered for "any entity" by default, and it is recommended to deactivate these given their substantial scope.
6. To register steps for specific entities, it is highly recommended to upgrade to the latest version of Adxstudio Portals to take advantage of the Web Notification configuration page. Alternatively, the Plug-in Registration Tool can be used to register and unregister steps within an organization. That is beyond the scope of this blog article, but a walkthrough on how to register a plug-in and specific plug-in steps can be found in this TechNet guide.
7. Beyond the "any entity" steps mentioned above, if any Web Notification plug-in steps are enabled for Create, Update, Delete, Publish, Associate, or Disassociate on any entity that is not exposed on the portal, highlight them and click "Deactivate" in the menu above the view.

In scenarios "A" and "D" mentioned earlier, all of the SDK Message Processing Steps for the Web Notification plug-in should be deactivated. In scenario "C", only those SDK Message Processing Steps related to entities that are not exposed on the portal should be deactivated.

---

Web Notification URL Record Values

1. As an administrative user, navigate to Settings -> Web Notification URLs.
2. In the "Active Web Notification URLs" view, check the "URL" column, specifically, to make sure that the same domain does not appear more than twice. An organization can have multiple portals associated to it, so the maximum number of Web Notification URL records would be twice that of the number of portals.
3. If two Web Notification URL records do not already exist for each portal associated with the organization, the missing records should be created. Each portal should have one record referencing the Cache.axd handler and another referencing the SearchIndex.axd handler.
4. If any "URL" columns contain text other than http://[domain name]/Cache.axd or http://[domain name]/SearchIndex.axd, the record should be deleted or updated to be accurate. Additionally, all "URL" values should use the fully qualified domain name, as other values such as localhost or a direct IP address will not function. If the portal supports SSL, then the "URL" values would begin with https:// instead.

This validation pertains mostly to scenario "B" above, but these records should be checked for accuracy in all scenarios, and on a regular basis.

As mentioned previously, if your organization has the Adxstudio Portals Base solution version 7.0.0086 or higher installed, the "Save and Publish" action must be performed again on the Adxstudio Portals Web Notification configuration page if any changes to the Web Notification URL records are required.

---

To verify that cache invalidation is working properly after activating the plugin and registering the desired steps, a user can perform an action within an entity that was scoped for notifications and is associated to the portal in some way (such as modifying a record exposed by an Entity Form), wait a few moments, and then refresh the respective page on the portal to validate that the change took effect. Using the key combination of SHIFT + F5 is recommended here, as it will ensure that no data is cached in the browser when this verification is performed.

I want to stress once again that creating Web Notification URL records and activating the Web Notification plug-in only applies to version 7.x or prior portals. If you have deployed a version 8.x SaaS portal, then cache invalidation no longer relies on the plug-in and instead uses an Azure Event Hub that requires no configuration on the user's behalf.

If cache invalidation is not working properly within your portal following these steps or additional assistance is needed with configuring Web Notifications, please contact support.

Interested in more troubleshooting tips for Dynamics 365 portals? Check out the first installment for an overview of all topics covered throughout this series.