Search My Ramblings

Sunday, March 14, 2010

Odd numeric issue with XSLT subtraction

On my project that automates a business process within Peoplesoft, we recently ran into an issue where we were updating a Component Interface with a number representing U.S. dollars and it complained about the length.

Upon investigation I could see that for some reason we were trying to insert the value 1.48999999997 into a field that only supports 2 digits past the decimal point, or a scale of 2. So this indicated to me 2 issues with our project:
  1. The Java web service we wrote to call the Peoplesoft component interface did not handle erroneous data being sent in, and
  2. Somewhere in our automation code before this we were creating a bad currency amount.
To solve the Java web service issue, I noticed our interface classes used both floats and doubles for the currency amounts. And since the Peoplesoft component interface classes required BigDecimals for all the amount fields, I figured the BigDecimal conversion was likely the issue. The code was converting these using the BigDecimal.valueOf(long) or BigDecimal.valueOf(double) methods. While stepping through a JUnit test case I wrote, I could see that the amount was not getting scaled or rounded at all, so we were writing 1.48999999997 into the field causing the error. After revisiting the javadocs for BigDecimal to refresh my memory, I could see we likely needed to simply call the setScale(scale, RoundingMethod) to ensure we didn't try to insert such values in the future. After the change I unit tested it again and could see that now we could handle all the various data values sent in and correctly scale and round the value accordingly.

The other issue was tracked down by my fellow developer to an XSLT that was doing a simple subtraction of 2 currency values, but for some reason was creating this strange value. In this test case the original numbers were similar to 2450-2448.51, which was resulting in 1.48999999997! I figured somehow this was related to the BigDecimal class as well, behind the scenes used by the XSL engine in OC4J (Xalan?). As a workaround, we multiplied the value by 100, rounded it using the XSLT round() function, then divided again by 100 to ensure the correct scale was applied. While ugly it at least functioned as expected.