Question Status

Verified
Arun B S asked a question on 21 Mar 2014 3:45 AM

I am generating a file in UTF-8 File format. When i open my file through notepad, i can see only my data. But a 3 byte sequence BOM (Byte Order Mark) is getting added at the beginning of the file to indicate that it is UTF-8 file format which is invisible. If i open the file through command prompt, i can see that 3 bytes.

reference:      msdn.microsoft.com/.../textio.aspx

                       msdn.microsoft.com/.../textio.new.aspx

My requirement is that , i don't want that 3 byte sequence BOM at the beginning of the file but my file should be in UTF-8 file format.

Is there any way to remove the Byte order mark through X++ code ? Or any other solution for this?

Thanks & Regards

Arun B S

Reply
Verified Answer
Martin Dráb responded on 23 Mar 2014 3:53 AM

This behaviour of TextIO class is documented ("UTF-8 files begin with a 3-byte byte-order mark sequence…") and doens't seem configurable. Therefore you shouldn't use it if you want generate files without BOM.

If you use .NET, you can exclude BOM by using properly configured UTF8Encoding. It's done by the parameter of UTF8Encoding's constructor in the following example (written in X++ using .NET Interop):

System.Text.Encoding encoding = new System.Text.UTF8Encoding(false);
;
System.IO.File::WriteAllText(@'C:\test.txt', "Data 123", encoding);

Martin "Goshoom" Dráb | Freelancer | Goshoom.NET Dev Blog

Reply
Suggested Answer
Denis Macchinetti responded on 23 Mar 2014 4:19 AM

Hi

In addition to what Martin say, try to use the code below :

System.Byte[]           bytes, bytesWithOutBOM;    

bytes = System.IO.File::ReadAllBytes( filename );

bytesLenght = bytes.GetValue(0);

bytesWithOutBOM = new System.Byte[bytesLenght-3]();

bytes.CopyTo(bytesWithOutBOM, 3);        

System.IO.File::WriteAllBytes(filename, bytesWithOutBOM );

Thanks & Regards

Denis Macchinetti

Senior Technical Architect

Reply
Martin Dráb responded on 21 Mar 2014 7:13 AM

Do you want generate the file without BOM or remove it afterwards? If the former, how do you currently generate the file?

Martin "Goshoom" Dráb | Freelancer | Goshoom.NET Dev Blog

Reply
Arun B S responded on 21 Mar 2014 11:31 PM

Hi Martin,

   Thanks for your reply. I am currently generating a UTF-8 file with BOM.

My ultimate requirement is to generate a UTF-8 file without BOM.

Regards,

Arun B S

Reply
Arun B S responded on 21 Mar 2014 11:59 PM

here is my code snippet:

      #define.UTF8Code(65001)

;

txioWrite = new Textio(fileName, #write, #UTF8Code);

txioWrite.write("Some Data");

txioWrite = null;

      ......

Reply
Arun B S responded on 22 Mar 2014 12:36 AM

Hi Martin,

My job

static void bank_test(Args _args)

{

   FileName                          fileName;

   InteropPermission          iopermission;

   Textio                                 txioWrite;

   #define.UTF8Code(65001)

   #define.write('W')

   ;

   fileName = "C:\\Documents and Settings\\abs\\Desktop\\test.CSV";

   iopermission = new InteropPermission(InteropKind::ClrInterop);

   iopermission.assert();

   txioWrite = new Textio(fileName,#write,#UTF8Code);     // #UTF8Code  is to generate UTF-8 file

   txioWrite.write("Data 123");

   txioWrite = null;

   CodeAccessPermission::revertAssert();

}

Thanks & Regards,

Arun B S

Reply
Verified Answer
Martin Dráb responded on 23 Mar 2014 3:53 AM

This behaviour of TextIO class is documented ("UTF-8 files begin with a 3-byte byte-order mark sequence…") and doens't seem configurable. Therefore you shouldn't use it if you want generate files without BOM.

If you use .NET, you can exclude BOM by using properly configured UTF8Encoding. It's done by the parameter of UTF8Encoding's constructor in the following example (written in X++ using .NET Interop):

System.Text.Encoding encoding = new System.Text.UTF8Encoding(false);
;
System.IO.File::WriteAllText(@'C:\test.txt', "Data 123", encoding);

Martin "Goshoom" Dráb | Freelancer | Goshoom.NET Dev Blog

Reply
Suggested Answer
Denis Macchinetti responded on 23 Mar 2014 4:19 AM

Hi

In addition to what Martin say, try to use the code below :

System.Byte[]           bytes, bytesWithOutBOM;    

bytes = System.IO.File::ReadAllBytes( filename );

bytesLenght = bytes.GetValue(0);

bytesWithOutBOM = new System.Byte[bytesLenght-3]();

bytes.CopyTo(bytesWithOutBOM, 3);        

System.IO.File::WriteAllBytes(filename, bytesWithOutBOM );

Thanks & Regards

Denis Macchinetti

Senior Technical Architect

Reply
Arun B S responded on 23 Mar 2014 8:51 PM

Hi Martin,

      I tried your code, file is getting generated without BOM but the file is in ANSI format.

Hi Denis,

    I am using AX 4.0 version, where System.Byte[] ...i am getting the syntax error at the first line itself.

Reply
Martin Dráb responded on 23 Mar 2014 11:34 PM

Regarding ANSI - it would be really strange if UTF8Encoding generated something else than UTF-8 (and it seems to work OK for myself). How did you distinguish ANSI from UTF-8? Can you give me an example?

Martin "Goshoom" Dráb | Freelancer | Goshoom.NET Dev Blog

Reply
Arun B S responded on 23 Mar 2014 11:44 PM

Hi Martin,

    I opened the generated file in notepad, File > save as > it is showing encoding type as "ANSI".

It should show UTF-8.

Reply
Martin Dráb responded on 24 Mar 2014 12:27 AM

Notepad isn't able to work with UTF-8 without BOM. If you want to use Notepad, keep BOM. If you want to continue without BOM, use a more capable text editor (my choice is Notepad++).

Martin "Goshoom" Dráb | Freelancer | Goshoom.NET Dev Blog

Reply
Denis Macchinetti responded on 24 Mar 2014 5:46 AM

Hi

Instead of use System.Byte[], use CLRObject like :

CLRObject               bytes, bytesWithOutBOM;

However, try to use :

System.IO.File::WriteAllText( fileName, System.IO.File::ReadAllText( filename ) );

In this way, seems the BOM disappear.

Thanks & Regards

Denis Macchinetti

Senior Technical Architect

Reply