FineBooks benchmarks 14 open OCR models on historical texts to fix LLM

A joint initiative by Hugging Face and EleutherAI titled FineBooks evaluated 14 open-source OCR models across 2,165 historical book pages to solve data degradation in LLM pre-training corpora. Legacy OCR errors on public domain books have been shown to degrade language model efficiency by up to 30% compared to clean text.

The benchmark revealed that modern vision-language models achieve over 97% character accuracy at a processing cost under $2 per 1,000 pages. The top performer was dots.mocr, a 3-billion parameter model that outperformed models three times its size, followed by OvisOCR2 operating at just $0.46 per 1,000 pages.

The project aims to re-process massive open archives like the Biodiversity Heritage Library's 64 million pages, making high-quality historical texts available for open-source AI training without relying on degraded scans.