Time for a little technical note…

Web page authors, in describing their layout (as per the CSS specification) expect to be able to use various units of length: in particular, millimeters, inches, points (a PostScript point being 1/72 inch), and, of course, pixels (the pixels per inch ratio being specified by the screen resolution, and is normally an integer, such as the conventional value of 96). Naturally, one would want ten boxes of five millimeters each to align exactly with one box of fifty millimeters. This sounds simple enough, but it really isn't: for if the browser computes sizes and positions at the pixel level, five millimeters might round off to 19 pixels, say, so ten boxes of five millimeters each would add up to 190 pixels, which is not the same as one box of fifty millimeters, giving 189 pixels. Doing all calculations with a pixel precision multiplies these irritating “off-by-one-pixel” at all levels. The problem is even worse with modern antialiased font rendering algorithms, which typically operate on subpixel scales; when words need to be broken in boxes by the renderer, it is unacceptable to round their position off to integer pixel positions.

Unfortunately, this problem has no solution: it is a basic fact of life that computers cannot handle real numbers efficiently without roundoff errors. Probably the best behavior is to subdivide the pixel in a fixed smaller unit, say in 1440 “pixies”. In this case, five millimeters might round off to, say, 18 pixels and 1293 pixies, and ten times that make 188 pixels and 1410 pixies, whereas fifty millimeters round off to 188 pixels and 1406 pixies, so the error is only 4 pixies, well below a pixel: thus, even after pixy positions are rounded to the closest pixel, there will be no overall roundoff error; certainly it is not impossible to produce roundoff errors still, but it is much more difficult. Donald Knuth's famous TeX typesetting system does this, using a basic unit, the “scaled point”, of roughly 5.362851 nanometers (the exact value being, 2368143360 scaled points in 127 meters). Essentially, the smaller the unit and the more divisible by common units, the better the precision; but if it is made too small, typical lengths and positions will be large: TeX cannot handle lengths of 1073741824 scaled points or more, that is, roughly 5.76 meters. Web pages may reach quite frightening lengths, however, since they are typically viewed by scrolling. I believe a pixy of 1/1440 screen pixel is a reasonable value: it is larger than TeX's basic unit, but more nicely multipliable by 3's and 5's. Of course, floating point types could be used, but that is simply a way of pushing ashes under the rug, since the roundoff errors will continue to exist, and will be more difficult to control.

In theory, it would be possible to compute with, say exact rational numbers (by remembering their numerator and denominator). That is, after all, what I have done above in stating that there are 2368143360 of Knuth's scaled points in 127 meters (5000 inches). Exact rational arithmetic, however, is costly in terms of memory storage (think of Web pages containing many thousand boxes) and CPU time, since the size of numerator and denominator tends to explode rapidly. Besides, there is no reason to stop at rational arithmetic: exact real algebraic numbers can be handled in principle, so why shouldn't a user be allowed to specify, say, a length equal to the diagonal of a square of side 1 inch, so sqrt(2) inches, and put 408 such lenghts end-to-end to compare them with 577 inches ? There is no limit to the madness one can allow except the limits one decides to set.

In practice, I have been rather annoyed with Mozilla's bug 63336 these days: the text of very long pages tended to be slightly corrupted by off-by-one-pixel errors, and some words jumped around by one pixel when the mouse was moved over them because the CSS “hover” quasi-element was used. I applied the workaround described in comment 12 of that bug discussion, however, fixing the screen resolution at 96dpi, and it seems to have solved the problem for me, while in fact improving the overall layout quality (I'm not sure I understand why). This is not very satisfactory, intellectually, perhaps, but at least it works.