Unable to properly convert a value due to decimal precision issue.

(0) Share

Report

Posted on by Sean The Michael

139

Hello all, good day. I would like to ask a question, i've been trying to import something into the database via job and I am having issues with some values, here is an example.

1000.002

Here is the line of code within the switch where it is supposed to assign the value:

case COMVariantType::VT_R8 : cellItemId = strFmt(num2Str(cells.item(record, 1).value().double(),0,0,0,0)); break;

Instead of assigning the value of 1000.002 it instead assigns just the value 1000.

What is wrong with my conversion here?

Thank you.

All responses (4)

Answers (0)

Sean The Michael 139 on at

Like (0)

Report

RE: Unable to properly convert a value due to decimal precision issue.

Ah hello thank you for this, but This is not the only record with that issue, there are about a hundred more records that have decimal values. But thank you for your input sir.

EDIT: It actually worked after some tinkering about. Thank you for your suggestion good sir.
kvnkk 800 on at

Like (1)

Report

RE: Unable to properly convert a value due to decimal precision issue.

can you try this ---info(strFmt(num2Str(1000.002,0,3,0,0)));
Sean The Michael 139 on at

Like (0)

Report

RE: Unable to properly convert a value due to decimal precision issue.

Hello sir Ludwig,

The item with 3 decimal places is an ItemID provided to us by our user via excel file. My job is supposed to check if the value 1000.003 exists in a table so my job can proceed to write the data associated with that ItemID, but it only returns the value of 1000.
Ludwig Reinhard Microsoft Employee on at

Like (0)

Report

RE: Unable to properly convert a value due to decimal precision issue.

Hi Sean The Michael,

The first question that I have is why you use 3 decimal places?

Is this something that you adjusted or a part of a localization package for your country?

Best regards,

Ludwig