Multiple AOS in a cluster can easily have different code or data in their local server caches, both with data (parameter tables set to EntireTable typically) and code. This is, of course, readily expected anytime parameters are changed or code is imported into the AOT on one AOS.
What is not expected is that the cache invalidation mechanisms built into AX seem broken out of the box.
First, there are several options under the Developer Tools menu, under Caches. These are Refresh dictionary, Refresh data, and Refresh elements. Unfortunately, the menu items behind each of these specify RunOn Called from, and the classes themselves also are set Called from, and the static main(..) methods of those classes do not peg them at server. Consequently, there appears to be no way to clear the AOS cache from the client. The solution to this, as one can quite readily find on blogs and forum posts, is to copy these menu items and force these copies to RunOn the Server, and testing shows that this indeed seems to work.
But, what is the value of clearing the client cache when simply closing and re-opening the client is so easy, and yet no mechanism is provided to clear the server cache, when stopping and restarting the server is so painful?
Second, there appears to be a cache invalidation mechanism between multiple AOS via the SysEvent table. For example, the SysFlushData\main method makes a call to SysEvent::fireEvent(SysEventType::FlushData) which inserts a record into the SysEvent table. While at first glance this may appear like transaction history only, I found a class called SysEventHandler which appears to poll this table and execute the various types of cache flush based on these newly inserted records.
However, SysEventHandler does not appear to get started anywhere. Monitoring an active AOS cluster with SQL Server Profiler reveals no polling against the SysEvent table by any AOS. It appears at first glance as though a built-in AOS to AOS cache invalidation mechanism exists, but is simply not activated or otherwise broken.
Does anyone understand why this situation is designed as it is and why managing an AOS cluster is made more complex due to apparently simple oversights such as leaving menu items set to Client and forgetting to start an event handler?