Floating Point Accuracy

FPU Accuracy

Basic arithmetic

It is possible to make FPU's which perform the basic arithmetic functions (+,-,*,/,sqrt) perfectly, i.e. the result returned is the most accurate which is possible in the format used. Apart from the occasional bug, all (?) FPU's which use IEEE binary arithmetic achieve this performance.

Trig functions

Two factors affect the accuracy of the values returned by the basic trig instructions which compute sin(), cos(), tan(), etc. These are:
  1. The accuracy of the internal representation of pi.
  2. The accuracy of the instructions for reduced arguments.

Internal Representation of pi

To compute the value of a basic trig function (sin(), cos(), etc) for an argument which is larger than some small value (typically pi/2) the usual method is to first subtract an appropriate multiple of pi. This process is called argument reduction.

To accurately reduce an argument for a trig function, a very accurate value of pi is needed. For example, on the Intel 80x86 architecture the fsin instruction (used to compute sin()) takes a 64 bit precision argument and returns a result of the same precision. To accurately compute the sin() of the number which is closest to pi in this format requires a pi with more than 128 bits accuracy.

For larger arguments, even greater accuracy is required. For example the integer 

     8227740058411162616  =  2618971001542672383.9999999689975244 * pi
is exactly representable as a 64 bit precision number. The sin() of this number is 
     -9.73971495111e-08
and requires a pi with about 150 bit accuracy to be able to get a result with 64 bit accuracy.

Intel 80x86 processors (at least up to the Pentium Pro processors) have an internal pi which is accurate to about 69 bits. On these processors, the results of the fsin instruction therefore has reduced accuracy for arguments which are close to multiples of pi or are large in magnitude.

To determine to accuracy of the internal pi in Intel compatible processors, in May 1997 I posted a small test program to a Linux newsgroup. The program determines the accuracy of the result of the fsin instruction for 36 different arguments.

I received about 30 replies. The results were: 

-------------+---------+-----------+-----------+------------------------------
  CPU        |  Nr of  |    pi     | Incorrect |
  type       | reports | precision |   tests   | Notes
             |         |  (bits)   |           |
-------------+---------+-----------+-----------+------------------------------
486 DX2/66   |   5     |   69.40   | 17,27,29  | Intel, Cyrix and AMD
Pentium/PPro |  16     |   69.40   | 21,22,29  | Pentium-66, P133 and PPro-200
K6           |   2     |   69.40   | 21,24,29  | 166, 200
6x86         |   7     |   72.02   | 29        | IBM and Cyrix. P150+ to 200+
Emulator     |   1     |  129.48   | 17        | Linux version 1.22
K5 PR133     |   2     | >150 (2)  | 22,27     |
-------------+---------+-----------+-----------+------------------------------
Note: in each case where the result was incorrect, the error was one ulp.
(2) The program could not measure the pi precision of a K5, implying a
    precision of better than about 150 bits.

Trig Accuracy for Reduced Arguments