Digits of precision and software

One of the basic notions that kids are taught in school is digits of precision. Even going back to elementary school kids are taught as early as grade 2 about approximate numbers. This is done using rulers that are marked out in units of inches or centimeters. The student has to pick the closest line on their ruler. Later on reading dials covers telling time. The distance and time issues are extended to measurements of volume and even surface area.

The idea is covered again and again especially as the arithmetic is advanced. Computations involving measurements are reduced to the significant digits.

Somewhere along the line this concept is lost as computers are used to perform the calculations. It is unfortunately all too easy to find journal articles which report numbers with 5 or 6 digits of precision. The numbers are copied out of whatever software is being used and pasted into the article being written.

Most stereological work is limited to 2 digits of precision. There might be 3 digits at times.

The measurements themselves are limited to 3 digits. If the measurements are taken from a computer screen, then the screen is often limited to about 1000 pixels. That limits the precision to 3 digits. Due to imaging issues the number of significant digits might be less than 3. Slices and sections cut with a microtome are limited to 3 digits at best. The thickness measurement is limited.

Stereological estimates are often done to a CE of .05. That limits the precision of the results to less than 2 digits.

Report values properly. Do not copy and paste from the software.


Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: