0x200B is inexplicably appended to a JSON Value

Introduction:

In this blog post, I will share a tricky issue of an inexplicable zero-width space being appended to a JSON value.

Background:

This issue was encountered while integrating D365 and ERP systems that involved the translation of data in JSON format.

Integration Diagram:

The request body that was sent from D365 to the ERP system was in JSON format and looked like this:

{
"ProductCode": "P1001",
"ProductName": "ProductName001",
"ReasonCode": "A1"
}

Error Description:

However, the ERP system reported JSON format errors with the ReasonCode field. The length of ReasonCode was expected to be 2, but the length received by the system was greater than 2.

D365 sent body:

{

"ProductCode": "P1001",

"ProductName": "ProductName001",

“ReasonCode”: “A1”

}

ERP received body:

{

"ProductCode": "P1001",

"ProductName": "ProductName001",

“ReasonCode”: “A1<0x200b>”

}

Root Cause:

The root cause of this issue was the presence of the zero-width space in the data, which was not visible in the UI of the D365 system.

Upon comparing the sent and received message bodies, it was discovered that a zero-width space, represented by <0x200b>, was unexpectedly appended to the JSON value of ReasonCode.

Upon further investigation, it was found that the zero-width space is the Unicode value for a zero-width space. These characters are not usually visible in most editors by default, leading to confusing situations like this one.

To test this, an online tool can be used.

Solutions

Solve this issue, there are two possible solutions:

Solution 1: Since the value is master data, it can be manually removed and re-filled.

Solution 2: A regular expression can remove the zero-width space.

The following code can be used to replace the hexadecimal value of the zero-width space (i.e., \u200B) with an empty string to remove it:

str.replace(/\u200B/g,'');

In conclusion, zero-width spaces can be a tricky issue to deal with, especially when they are not visible in the UI of a system. It is essential to be aware of their existence and use appropriate methods like regular expressions to remove them from the data.

More about zero-width spaces you can find here: Zero Width Spaces

The End