Archiving family photographs: a format statement

I am starting to scan the family’s photographs on an Epson Perfection 4990 for i.bytestone.uk, where each person gets a page. A photograph scanned once should never need scanning again, so the scan has to be an archive and not just a picture for a web page. This post sets out what that means in practice: a statement in the spirit of the Library of Congress Recommended Formats Statement for still images, with capture targets taken from the US federal FADGI guidelines and the format choice checked against the British Library’s preservation assessment of TIFF.

One choice is still open: where the offsite copy lives.

Principles

A master is captured once and never altered. FADGI is blunt about this: no dust removal, no colour or tone enhancement, no restoration on a master. Everything that makes a photograph look good on a screen happens to a rendition, which is derived from the master and can be thrown away and made again when the processing improves. Improving the pictures means re-running the processing, never re-scanning.

The archive must outlive the tools. In fifty years nobody will have the scanning program or its database. Every photograph is kept as a directory that explains itself: images in a common format, metadata in plain text, and a checksum for every file. The database only records the work done; it holds nothing that the archive needs.

Boring formats. The British Library’s assessment of TIFF found its risks come from the unusual corners (rare compression schemes, multiple images in one file, private tags), not from the format. We stay in the well-trodden middle.

Say how good it is. FADGI grades images from one to four stars. Where we fall short of a level, the statement says so instead of implying otherwise.

What we capture

The back is captured in colour because ink colour distinguishes one writer, or a lab’s stamp, from another.

Small prints get 1200 ppi because a 3½ × 2½ inch print at 600 ppi is only 2100 × 1500 pixels, and old contact prints can hold more detail than that.

Every scan uses fixed settings: no automatic exposure, sharpness neutral, no dust removal, the scanner’s colour correction off. They are recorded with the master. That is what lets a colour profile be added later.

Colour

There is no calibration yet, so the masters are in the scanner’s own colour space and fall short of FADGI 3-star colour, which asks for a measured profile in Adobe RGB (1998), ProPhoto or eciRGB v2. When an IT8.7/2 target arrives, the scanner gets an ICC profile (made with ArgyllCMS), and because the settings never changed, that profile describes every master already made. It is assigned to them in their metadata, not used to convert them, so the pixels stay as captured.

Noise

Two scans of the same print, subtracted, leave only the scanner’s noise. Measured that way, the 4990’s settings matter more than the scanner:

Noise is in counts of 65535, measured over USB 2.

The scanner’s built-in colour correction is a matrix that mixes the channels and amplifies noise, blue worst; it also makes the channels’ noise correlated. It is turned off, and colour is left to the ICC profile, which does the same job properly. The 4990’s own 600 ppi mode is noisier per pixel than its higher resolutions, so the firmware is evidently not averaging when it reduces resolution. We scan higher and average blocks of pixels down to 600 ppi, a fixed linear step recorded as part of the capture. At 1200 ppi that gives about 150 counts, a little over 8½ bits of signal per pixel, close to Roger Clark’s measurement of the same scanner. 2400 ppi gives about 90 counts, but the gain is less than the 16 pixels averaged would suggest. Neighbouring pixels at 2400 ppi share noise, because the lens resolves less than 2400 ppi. It costs time, not storage.

The scanning resolution is chosen per batch: 1200 ppi for bulk work, 2400 ppi for the photographs that matter most, for about ¾ of a bit less noise at twice the time. It is recorded with each master.

Subtracting repeat scans cannot see fixed-pattern noise, such as faint streaks that repeat identically in every scan; that needs a flat target.

The scanner’s sharpness setting does what it says. +2 doubles the noise by sharpening, and −2 halves it by blurring along the scan line. Both change the image, so it stays at 0, which measures as neutral.

The masters keep the noise that remains. Removing it means guessing which fine texture is the print’s grain and which is the scanner’s, and FADGI rules that out for masters. Renditions may be denoised, and the measured figure tells the denoiser how hard to work. Averaging repeated scans is a legitimate way to reduce noise: two passes cut it by the expected √2. We don’t do it yet, because a pass that shifts by a fraction of a pixel would blur the average.

That noise sets how many of the 16 bits carry information. Rounding the 1200 ppi averaged scan to coarser steps, still stored as 16-bit TIFF with Deflate:

Rounding to a step under half the noise adds almost nothing, because the noise is already larger than the step. Detail averaged over an area survives as well, since the noise randomises the rounding. So 10 bits would halve the storage at a cost of under 1% more noise.

We keep all 16 bits anyway. “Lossless” is a claim anyone can check; “lossy, but only below the noise” depends on trusting a measurement and on the step being re-chosen whenever the settings change. At family-archive scale the difference is a few hundred gigabytes per copy.

FADGI asks for a reference target in every exposure. We will scan the target once per session instead, and say so.

What we keep

FADGI and the Library of Congress accept Deflate (lossless “ZIP”) compression for TIFF. The British Library notes many institutions insist on uncompressed, but traces that habit to the old LZW patents rather than to any risk in lossless compression. Every TIFF reader decodes Deflate with the horizontal differencing predictor. Measured on a 7 × 5 inch print at 600 ppi, 16-bit:

The saving is a modest 19%, because the scanner’s noise fills the bottom eight or so bits of each 16-bit sample and noise does not compress. LZW makes 16-bit scans bigger, so it is ruled out.

Why not PNG, which is lossless, simple and on the Library of Congress’s preferred list? FADGI lists only TIFF and JPEG 2000 as master formats for photographs. PNG records resolution in pixels per metre, so 600 ppi can only be stored approximately. Its embedded metadata is also less widely read by archive and photo tools. PNG’s per-chunk checksums would be a bonus, but the bag’s SHA-256 manifest already covers fixity.

A bed scan holds several prints. Each print gets its own master, cut from the bed scan with a margin so the print’s own edges, borders and any stamp on them survive, and not rotated: straightening resamples the pixels, so it belongs to the rendition. The uncut bed scan is not kept.

Metadata

A photograph’s metadata is mostly claims: the date on the back, the names written under it, the studio that took it. Each claim records where it came from, because a date written in pencil in 1960 and a date read off the paper’s backprint are not the same kind of evidence.

Each claim has a value, a source (which side, which person, which program) and how it was read: transcribed by a person, machine-read, or inferred. A machine reading is never published until a person has confirmed it. Dates use ISO 8601 prefixes (1952, 1952-07) with an approximate flag, the same as i.bytestone.uk’s person records.

Metadata lives in two places, and the split follows the rule that masters never change:

• Embedded in the master as XMP: only what is fixed at capture: the photograph’s id, which side, the batch, the scanner and its settings, the capture date. A master separated from its directory still says what it is.
• In the sidecar photo.yaml: everything that can change: the claims, captions, who appears, which person’s album it belongs to, and the rights, which are rarely settled when a print is scanned. This is the authority.

The Library of Congress asks for title, creator, date, place and publisher where the format supports them. For a family photograph most of those are claims, and “unknown” is an honest value.

Packaging

Each photograph is one BagIt bag (RFC 8493): a directory with a manifest of SHA-256 checksums that any BagIt tool can verify.

<photo-id>/
  bagit.txt
  bag-info.txt
  manifest-sha256.txt
  tagmanifest-sha256.txt
  data/
    front.tif
    back.tif
    photo.yaml

Renditions sit outside the bag. They are access copies, made again whenever the processing improves, and are not part of what is preserved.

A photograph’s id is its own, in its own series, written in lower-case Crockford base32 like i.bytestone.uk’s person ids. It does not depend on which person’s album the photograph is in, so moving it to another album changes a link, not its identity.

When a claim is corrected the manifest is regenerated, so the bag alone would lose the old value. The sidecars are also kept in a git repository, which keeps their history at little cost.

Copies and fixity

Photographs whose rights do not allow publishing are still archived, which is why the site cannot be the archive. A scheduled fixity check recomputes every checksum on every copy and reports any that differ.

Fixity runs monthly.

Open: where the offsite copy lives.

Rights

Old family photographs rarely have a clean answer on copyright: studio portraits, unknown photographers, and UK transitional rules for photographs taken before 1957. So each photograph records three things:

The rights statement decides whether a photograph is published; nothing is published on a guess.

Where we stand

References

• Library of Congress, Recommended Formats Statement: Still Image Works
• Federal Agencies Digital Guidelines Initiative, Technical Guidelines for Digitizing Cultural Heritage Materials, 3rd edition, May 2023
• British Library Digital Preservation Team, Preservation Assessment: TIFF Format, version 1.3, 2015
• Roger N. Clark, Drum vs flatbed scanners
• Pennock, Wheatley and May, Sustainability assessments at the British Library: formats, frameworks and findings, iPRES 2014
• The National Archives, PRONOM fmt/353: Tagged Image File Format
• RFC 8493: The BagIt File Packaging Format
• RightsStatements.org