Finance | Project Operations, Human Resources, ...

how to track new changes without having a limited period?

(0) Share

Report

Posted on by junior AX

1,552

So if i want to run a batch every 5 minutes where i want it to export an entity with new changes, so let's say every 5 minutes i want to detect all records that their fieldX newly changed to a certain value (query range)

Would change track help in this case? does it expire?... i mean after 1 month would i be getting old values for records with that fieldX value? or is there another way to do it?

I have the same question (0)

All responses (21)

Answers (0)

Suggested answer

nmaenpaa 101,160 Moderator on at

Like (2)

Report

If I understand correctly, you would like to export entity data, but only the changed values.

You can enable change tracking for the entity, and then use Incremental push in your data project. This way you only get new and changed records.

If your job runs once per month, and something was changed 3 weeks ago, you will still get the latest version of that record. Not the 3 week old version.

Was this reply helpful? Yes No
Blue Wang on at

Like (1)

Report

Hi Junior,

If you want only records that have changed are exported,you need to use incremental export.

To enable incremental export, you must enable change tracking on entities.

docs.microsoft.com/.../entity-change-track

Was this reply helpful? Yes No
junior AX 1,552 on at

Like (0)

Report

Thanks Nikolaos and blue,

So if i'm exporting entity by code, the change tracking must be enabled from UI not code right? but how can i change incremental push by x++? and what would happen if i enable change tracking without incremental push(what doses incremental push mean?)

and what guarantees that this change tracking will work forever without limited period? or is this how it works? because i read that there is limited period.

and if i run it for the first time it will take all records with that fieldX value right?

Was this reply helpful? Yes No
Suggested answer

nmaenpaa 101,160 Moderator on at

Like (1)

Report

Change tracking must be enabled via the UI.

I don't know how you are exporting the entity by code. But incremental push only works with data management projects. If you execute it by code, then it works.

Nothing guarantees that the change tracking will work forever, but I don't think that you need "forever". I would be surprised if the system even exists in 100 years. And forever is much longer :)

I'd say that it works with any time period that you would need in normal business processes. Earlier you mentioned that you want to run your code every 5 minutes. It definetely keeps the changes tracked for 5 minutes.

Was this reply helpful? Yes No
junior AX 1,552 on at

Like (0)

Report

Hi Nikolas,

I'm exporting by code.. Where i create a new project by code, so what code can i right to enable incremental push?

As for the period.. I read that there is a property on sql for change tracking.. So if let's say this value was 3 days.. My entity will start getting old values after 3 days? Let's say i want to run the batch every 5 min for 2 years maybe more.

Was this reply helpful? Yes No
Suggested answer

nmaenpaa 101,160 Moderator on at

Like (0)

Report

Just create a data project manually, and set the entity for incremental push. This way you can see how the change tracking works. This is the topic of the current discussion. If you want to create data projects in code, perhaps it's a good idea to post a new question about that.

If you run the job every 5 minutes, it means that the tracked change will become irrelevant in 5 minutes (once the data is exported). Then the change tracking status is "reset" from your point of view, and only the changes that happen in the next 5 minutes are relevant for the next export.

If you have read some documents, and have questions about them, you should tell us what documents you read. Otherwise we can't comment anything in those documents.

Was this reply helpful? Yes No
junior AX 1,552 on at

Like (0)

Report

Hi Nikolaos,

From what i understood: If change tracking is enabled and i run the export the first time i will get all records with range fieldX. Now when i run it after 5 minutes and there were records that changed from the first run then i will get only those records and so on.

Now there is a property on DB called changed tracking where you specify retention time and AutoCleanUp

Are these related to the tables property change tracking and the data entity change tracking, i mean if retention period was 3 days, this would mean when i run the batch after 3 days then it's going to give me all records with range fieldX just like the first time i run it? If yes does that mean i need to set Auto Cleanup to false? and what disadvantages would that make?

if you check this link, you'll find this paragraph www.gangsta.se/.../change-tracking-on-data-entities-in-d365fo

Also note that the retention period for the change tracking is configured in SQL Server. This means that if you are going to use change tracking and Incremental push for an integration, the frequency must be higher than the retention period, otherwise the change tracking is lost and the next export will be a full push. You can find the retention period on AxDB > Properties > Change Tracking in SQL Server Management Studio. On the development VM's from Microsoft, the value seems to be set to 7 days.

what does it mean that the frequency should be higher? does it mean that the time where i run the batch should be less 3 days each time (less than retention period) so this means i'm in the safe side? So the retention period is recalculated each time i run the batch? or i understood wrongly?

Was this reply helpful? Yes No
Suggested answer

nmaenpaa 101,160 Moderator on at

Like (0)

Report

You can't change this property in D365FO UAT/Prod systems, it's managed by Microsoft. So you don't need to worry about this setting. You can ask Microsoft what is the retention time in D365FO systems if you are worried if it's long enough.

But I would not invest too much time into it, I'm sure it's long enough for your integrations.

Was this reply helpful? Yes No
junior AX 1,552 on at

Like (0)

Report

But why ask them if i can see the property in DB? So you think this property (3 days) has nothing to do with the change tracking period in data entities? I need to know the answer because exporting the same data again could be a problem and i'm thinking if there is another way similar to change tracking that i can do where i don't have to worry about this?

Was this reply helpful? Yes No
Suggested answer

nmaenpaa 101,160 Moderator on at

Like (0)

Report

"Why ask them"? Because you can't see the property in D365FO prod database. So, if you're interested in this information, then you need to ask from the company who hosts the prod database.

What do you mean by "change tracking period in data entities"?

If exporting the same data again would be a problem I strongly recommend having some other control mechanism than only change tracking. For example some flag on the table that tells when it's exported.

Was this reply helpful? Yes No