Model Store Deployment Best Practices

This question is answered

Hi all,

I have a build process that currently produces model files as an output. These can then be manually or automatically imported into our test/staging environments. While investigating this I noticed that there seems to be significant support for using model stores instead of individual model files. The ability to produce ready made application layers that don't require compilation or CIL builds is pretty attractive. I am becoming considerably confused as to what the best practice is, What scenarios exist that would influence the choice of one of the deployment methods (XPOs, Models, Model Stores) over the others? The Microsoft documentation suggests the following:

Scenario

Recommended installation method

Distributing a solution to customers

Model files

Deploying a solution in a development or test environment

Model files or XPO files

Deploying a solution to a production environment

Model store files

I cannot find any information to support these statements however. Why are model files and XPO files better suited for deployment between development and test environments? I would have thought the model store would always be the most convenient method.

If anyone can provide some further clarification on this it would be much appreciated.

Simon

Verified Answer
  • Forget XPOs. I'm appalled Microsoft even mentions it.

    Between dev and test, you need to keep in mind there may be discrepancies in object IDs, table IDs, etc. Deploying model files avoids this issue, as it will create new IDs for things that don't exist, re-use IDs for things that already exist. Moving a model store includes the IDs, so any discrepancy will result in data being dropped etc. So if you can tightly control so that no code is ever imported manually or added manually in anything but the one and only dev environment you should be safe to deploy model stores.

    Also, when we do builds we make sure to remove ALL previous code, so that our build is as accurate as possible and doesn't accidentally depend on stray code we didn't know about. Removing all the code first implies all IDs will be recreated, which makes it impossible to move that model store as it will most likely have ID differences with the target model store.

    To production you want model store because of the reasons you said. You minimize downtime by compiling somewhere else, plus you can import model store in different schema and just flip the switch when you're ready.

    Distrbuting to customers implies the customer may have their own model as well, they may have third party products etc. Unless you have the up-to-date models on your build environment, obviously you can't deploy the whole model store in such a scenario since you'll wipe out whatever code they may have that you do not in your build environment.

    Hope this helps.

  • Here is my recommendation:

    1. Production

    Always move changes to production from a Staging/Test/User Acceptance Testing environment. Move the entire ModelStore, precompiled X++ and CIL without any errors. Install it first to a temporary database schema (http://yetanotherdynamicsaxblog.blogspot.no/2013/02/deploy-modelstore-with-less-downtime.html), then take all the PROD AOSes down, then apply it, then start 1 AOS and run a data dictionary synchronize  Then start any other AOSes pointing at PROD.

    2. User Acceptance Test (UAT)

    Given a single test environment, you will install changes to this environment using models. There are scenarios where XPO is ok, which I will comment down below. 
    Given a separate build environment for test, you will move code to UAT using modelstore from build to UAT. In a perfect world, you will never install a model in UAT that has compile errors or dependency issues. Remember that whenever you install a model and need to run compile + synchronize  this process will take hours. Schedule it outside testing hours or consider setup a build environment.

    Code is moved from here to Production using modelstore. Data is updated from Production to UAT. Environment IDs are shared, except from any new IDs introduced in UAT waiting to be shipped to Production. :-)

    3. Development

    Some customers will accept having a customer specific developer environment, which allows us to develop customer specific customization without interfering with test. This is highly recommended. Here you will install updated models, your own and third party models, update customer specific models, merge code and compile. You can move code into this environtment using both models and XPO.

    You may schedule a restaging of the data and the code in Development, meaning you backup the UAT database(s) and restore them over Development. Remember to backup the models in Development (http://yetanotherdynamicsaxblog.blogspot.no/2012/09/using-powershell-to-backup-your-ax-2012.html), in case someone forgot they had super important code in Development that wasn't already shipped to UAT. 

    Over time the IDs in Development will stray away from Prod and UAT, so never move a modelstore from Development upwards without knowing what you are doing. 

    Models vs XPO

    There are scenarios where XPO is an easy and convenient way to move code from A to B.
    One clue is to know when introducing changes causes new IDs to be created. At this point, you need to know if this newly created ID will cause your UAT/Test and Prod to have different ID for the same element. If that happens, you've lost control and you need to restate UAT/Test again completely from Prod. 
    Another clue is when your XPO contains an element that contains changes bound to multiple AX models. Remember the XPO does not contain information of what change is bound to what model, so whatever you import using XPO will bind that change to the model you are currently working in. This may cause changes to be bound to a wrong model cause all sorts of havoc and issues later down the road.  

    So actually, you can move small changes to existing methods using XPO without introducing IDs if those changes all reside in the same model you exported the XPO from, if you catch my drift. You can also move SSRS reports with XPO safely. There are many scenarios supporting the use of XPOs, but you need to leave the mindset from pre-AX2012 where anything could be moved using XPO. Either you figure out when XPO can safely be used, or you refrain from using it altogether (like Joris recommends, sort of). ;-)

    Final note

    Moving a precompiled modelstore is only good when the source and targets share the same environment IDs. Production and UAT/Test should use the same IDs, allowing you to have things properly tested before putting it in production, PLUS allowing you to prepare a precompiled ready to launch modelstore having perfectly valid element IDs.

    When installing models you need to compile + synchronize. This is a time consuming process and more often you need to keep users out during. 

    When moving changes using XPO, you need to know how and why it will be ok to move changes using XPO. If you don't know the how and why, then don't use XPOs at all.

    Sorry for the wall of text... ;-)

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • You need to have a dedicated test environment, a separate AOS and a separate database, but based on production. No changes can be done directly in production without going through test, and only pushed to production using precompiled modelstores. If you skip this part, you will learn the hard way how to maintain an unstable and flaky production application.

    The notion of having two test environments is rather having one environment for test with ID equal to production, but still allow for compile time and such, and have another test environment that is not being compiled and is stable for consultants to test on. You don't need to have two test environments, but I would strongly suggest having at least 1.

    As for element IDs and element handles, you can read more about it here:

    technet.microsoft.com/.../hh352326.aspx

    You can also query the table ModelElement in the model database (AX2012 R2) or business database (Ax2012 RTM). Pay attention to the ID columns. These IDs are environment specific and you will essentially find dependencies to them in various places, even in userdata inside serialized containers. To put it another way; importing a modelstore from A into environment B is only doable if both A and B have the same element IDs and handles. Otherwise it will break.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Keep the questions coming, Simon. I am sure more people are asking the same questions. :-)

    Unless you're a magic worker, you will have logical bugs waiting to be discovered as soon as you test your code against the customers own data. Most likely, your development environment will be with less or outdated data. I reckon that is why there is this notion of fixing things in TEST and applying these fixes back to DEV. Since there is a high chance these changes will impose new element IDs, making each of these environments stray away from each other, it will not be easy to push the business data back and forth.

    If your customer/project allows for a dedicated STAGING or BUILD environment, then this would be the place you prepare your compiled modelstore.

    I have split this in PROD, TEST (or UAT) and DEV on the customer site. Then there will be in-house DEV environments for various solutions I work on. I keep element IDs equal in PROD and TEST, but DEV will over time get more and more different IDs. Reason is simply that completed code gets moved from DEV til TEST using models, and models will only contain the Origin ID (GUID), and not the element ID and element handle.

    I still recommend having  a BUILD environment for the customer, so you can build and prepare a modelstore without having to kick out the consultants from the customer TEST. This is probably more critical before a Go-Live, and after a Go-Live the cycle of pushing changes to Prod will slow down (hopefully), reducing the need for a dedicated BUIILD environment.

    As you probably know, one of the strengths in AX is the low threshold for making changes to the application (using the AOT). This is in the blood of the consultants, and they expect developers to be able to push changes to the business logic in a short and quick cycle.

    As developer, you know AX2012 can easily be broken if all dependent elements aren't properly compiled and CIL compiled. You also know a full compile takes hours.

    As responsible for the technical architecture you want the developers to work efficiently AND the consultants to be able to work efficiently. Having them to wait for each other will just stall everything. :-)

    Again, sorry for the wall of text.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

All Replies
  • Forget XPOs. I'm appalled Microsoft even mentions it.

    Between dev and test, you need to keep in mind there may be discrepancies in object IDs, table IDs, etc. Deploying model files avoids this issue, as it will create new IDs for things that don't exist, re-use IDs for things that already exist. Moving a model store includes the IDs, so any discrepancy will result in data being dropped etc. So if you can tightly control so that no code is ever imported manually or added manually in anything but the one and only dev environment you should be safe to deploy model stores.

    Also, when we do builds we make sure to remove ALL previous code, so that our build is as accurate as possible and doesn't accidentally depend on stray code we didn't know about. Removing all the code first implies all IDs will be recreated, which makes it impossible to move that model store as it will most likely have ID differences with the target model store.

    To production you want model store because of the reasons you said. You minimize downtime by compiling somewhere else, plus you can import model store in different schema and just flip the switch when you're ready.

    Distrbuting to customers implies the customer may have their own model as well, they may have third party products etc. Unless you have the up-to-date models on your build environment, obviously you can't deploy the whole model store in such a scenario since you'll wipe out whatever code they may have that you do not in your build environment.

    Hope this helps.

  • Wow. Thanks heaps for the quick reply.

    Joris dG

    So if you can tightly control so that no code is ever imported manually or added manually in anything but the one and only dev environment you should be safe to deploy model stores.

    When you say one and only one dev environment are you referring to the environment that is actually doing the build? We have multiple developers using TFS to work concurrently. If I understand what you are saying, as long as the model store is the only way that logic gets into the test system and it only comes from our build server we should be sweet.

    Or am I totally missing the point?

  • Here is my recommendation:

    1. Production

    Always move changes to production from a Staging/Test/User Acceptance Testing environment. Move the entire ModelStore, precompiled X++ and CIL without any errors. Install it first to a temporary database schema (http://yetanotherdynamicsaxblog.blogspot.no/2013/02/deploy-modelstore-with-less-downtime.html), then take all the PROD AOSes down, then apply it, then start 1 AOS and run a data dictionary synchronize  Then start any other AOSes pointing at PROD.

    2. User Acceptance Test (UAT)

    Given a single test environment, you will install changes to this environment using models. There are scenarios where XPO is ok, which I will comment down below. 
    Given a separate build environment for test, you will move code to UAT using modelstore from build to UAT. In a perfect world, you will never install a model in UAT that has compile errors or dependency issues. Remember that whenever you install a model and need to run compile + synchronize  this process will take hours. Schedule it outside testing hours or consider setup a build environment.

    Code is moved from here to Production using modelstore. Data is updated from Production to UAT. Environment IDs are shared, except from any new IDs introduced in UAT waiting to be shipped to Production. :-)

    3. Development

    Some customers will accept having a customer specific developer environment, which allows us to develop customer specific customization without interfering with test. This is highly recommended. Here you will install updated models, your own and third party models, update customer specific models, merge code and compile. You can move code into this environtment using both models and XPO.

    You may schedule a restaging of the data and the code in Development, meaning you backup the UAT database(s) and restore them over Development. Remember to backup the models in Development (http://yetanotherdynamicsaxblog.blogspot.no/2012/09/using-powershell-to-backup-your-ax-2012.html), in case someone forgot they had super important code in Development that wasn't already shipped to UAT. 

    Over time the IDs in Development will stray away from Prod and UAT, so never move a modelstore from Development upwards without knowing what you are doing. 

    Models vs XPO

    There are scenarios where XPO is an easy and convenient way to move code from A to B.
    One clue is to know when introducing changes causes new IDs to be created. At this point, you need to know if this newly created ID will cause your UAT/Test and Prod to have different ID for the same element. If that happens, you've lost control and you need to restate UAT/Test again completely from Prod. 
    Another clue is when your XPO contains an element that contains changes bound to multiple AX models. Remember the XPO does not contain information of what change is bound to what model, so whatever you import using XPO will bind that change to the model you are currently working in. This may cause changes to be bound to a wrong model cause all sorts of havoc and issues later down the road.  

    So actually, you can move small changes to existing methods using XPO without introducing IDs if those changes all reside in the same model you exported the XPO from, if you catch my drift. You can also move SSRS reports with XPO safely. There are many scenarios supporting the use of XPOs, but you need to leave the mindset from pre-AX2012 where anything could be moved using XPO. Either you figure out when XPO can safely be used, or you refrain from using it altogether (like Joris recommends, sort of). ;-)

    Final note

    Moving a precompiled modelstore is only good when the source and targets share the same environment IDs. Production and UAT/Test should use the same IDs, allowing you to have things properly tested before putting it in production, PLUS allowing you to prepare a precompiled ready to launch modelstore having perfectly valid element IDs.

    When installing models you need to compile + synchronize. This is a time consuming process and more often you need to keep users out during. 

    When moving changes using XPO, you need to know how and why it will be ok to move changes using XPO. If you don't know the how and why, then don't use XPOs at all.

    Sorry for the wall of text... ;-)

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Thanks so much to both Joris and Tommy. You guys are the business! The main concept I am struggling with at the moment is the IDs as I like to understand why I am doing something rather than just blindly following some rules. I am still a little confused by some about some of your advice:

    Tommy Skaue
    Given a single test environment, you will install changes to this environment using models. There are scenarios where XPO is ok, which I will comment down below.

    Given a separate build environment for test, you will move code to UAT using modelstore from build to UAT.

    What do you mean by single test environment? Are you suggesting that you should deploy to the test environment with models if you share a test environment but if you have a dedicated test environment then use model stores? Am I misunderstanding your point here?

    Thanks again to you both. You have definitely given me some great information that will make the learning process a lot easier.

  • You need to have a dedicated test environment, a separate AOS and a separate database, but based on production. No changes can be done directly in production without going through test, and only pushed to production using precompiled modelstores. If you skip this part, you will learn the hard way how to maintain an unstable and flaky production application.

    The notion of having two test environments is rather having one environment for test with ID equal to production, but still allow for compile time and such, and have another test environment that is not being compiled and is stable for consultants to test on. You don't need to have two test environments, but I would strongly suggest having at least 1.

    As for element IDs and element handles, you can read more about it here:

    technet.microsoft.com/.../hh352326.aspx

    You can also query the table ModelElement in the model database (AX2012 R2) or business database (Ax2012 RTM). Pay attention to the ID columns. These IDs are environment specific and you will essentially find dependencies to them in various places, even in userdata inside serialized containers. To put it another way; importing a modelstore from A into environment B is only doable if both A and B have the same element IDs and handles. Otherwise it will break.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Thanks Tommy, I get it now. Thanks for the hand holding. You da man...

  • Tommy,

    There is one more thing. After looking at the at all of the information you gave me and the document titled, Deploying Customizations Across Microsoft Dynamics AX 2012 Environments I still have some confusion in regards to best practice. I just don't understand why there seems to be this need to be able to update code on the test environment. The following diagram from the Microsoft deployment best practices mentions Deploy, test, make changes as actions to be performed on the test environment.

    In my experience development is better done in one place and deployed to test environments as built assemblies. Why do we need to be able to do more development in the test environment? If the build process produced a model store and it was used to deploy to the test, staging and product environments wouldn't that be preferable? Isn't the ability to modify code in other systems post-build the cause of the ID contention issues?

    Of course, you guys know better than me and there may be a valid reason for this strategy...

    Simon

  • Keep the questions coming, Simon. I am sure more people are asking the same questions. :-)

    Unless you're a magic worker, you will have logical bugs waiting to be discovered as soon as you test your code against the customers own data. Most likely, your development environment will be with less or outdated data. I reckon that is why there is this notion of fixing things in TEST and applying these fixes back to DEV. Since there is a high chance these changes will impose new element IDs, making each of these environments stray away from each other, it will not be easy to push the business data back and forth.

    If your customer/project allows for a dedicated STAGING or BUILD environment, then this would be the place you prepare your compiled modelstore.

    I have split this in PROD, TEST (or UAT) and DEV on the customer site. Then there will be in-house DEV environments for various solutions I work on. I keep element IDs equal in PROD and TEST, but DEV will over time get more and more different IDs. Reason is simply that completed code gets moved from DEV til TEST using models, and models will only contain the Origin ID (GUID), and not the element ID and element handle.

    I still recommend having  a BUILD environment for the customer, so you can build and prepare a modelstore without having to kick out the consultants from the customer TEST. This is probably more critical before a Go-Live, and after a Go-Live the cycle of pushing changes to Prod will slow down (hopefully), reducing the need for a dedicated BUIILD environment.

    As you probably know, one of the strengths in AX is the low threshold for making changes to the application (using the AOT). This is in the blood of the consultants, and they expect developers to be able to push changes to the business logic in a short and quick cycle.

    As developer, you know AX2012 can easily be broken if all dependent elements aren't properly compiled and CIL compiled. You also know a full compile takes hours.

    As responsible for the technical architecture you want the developers to work efficiently AND the consultants to be able to work efficiently. Having them to wait for each other will just stall everything. :-)

    Again, sorry for the wall of text.

    Tommy Skaue | Dynamics AX Developer from Norway | http://yetanotherdynamicsaxblog.blogspot.no/ | www.axdata.no

  • Simon,

    I do agree with you however. There should be no changes in TEST. Tommy's making a point of consultants expecting to have quick turnaround on fixes. This is the history of the beast known as AX, and is more a mindset than anything else. You can have quick turnaround doing things the proper way as well.

    I changed this around in our consulting firm about 2-3 years ago. It turned out to be less of an issue with the consultants than I thought. By now, I can honestly say we rarely do any emergency fixes directly in a TEST environment, maybe once every few months. And if we do, we do it in the model file that will be replaced later on, so we're sure we get the "proper" fix later. Also, those types of fixes should just be small fixes in code or on UI, which should not cause any ID problems. If what you're doing directly in TEST could cause an ID conflict (add a method, add a class, add a field, table, etc) you should be demoted for doing it in TEST since that would mean a big change, not just a quick emergency fix.

    In fact, on our client projects there's typically only one, sometimes two, developers that even have access to the client's network / systems at all. We do development in our offices and ship the model to the client who can do whatever they want with that. The only time we need access is when there's an issue we can't replicate reliably, bu even then we prefer a web meeting and someone show us.