A survey of OCR evaluation tools and metrics

Neudecker, C, Baierer, K, Gerber, M, Clausner, C ORCID: https://orcid.org/0000-0001-6041-1002, Antonacopoulos, A ORCID: https://orcid.org/0000-0001-9552-0233 and Pletschacher, S 2021, A survey of OCR evaluation tools and metrics , in: HIP '21: The 6th International Workshop on Historical Document Imaging and Processing, 5th-6th September 2021, Lausanne, Switzerland.

PDF - Accepted Version
Download (680kB) | Preview


The millions of pages of historical documents that are digitized in libraries are increasingly used in contexts that have more specific requirements for OCR quality than keyword search. How to comprehensively, efficiently and reliably assess the quality of OCR results against the background of mass digitization, when ground truth can only ever be produced for very small numbers? Due to gaps in specifications, results from OCR evaluation tools can return different results, and due to differences in implementation, even commonly used error rates are often not directly comparable. OCR evaluation metrics and sampling methods are also not sufficient where they do not take into account the accuracy of layout analysis, since for advanced use cases like Natural Language Processing or the Digital Humanities, accurate layout analysis and detection of the reading order are crucial. We provide an overview of OCR evaluation metrics and tools, describe two advanced use cases for OCR results, and perform an OCR evaluation experiment with multiple evaluation tools and different metrics for two distinct datasets. We analyze the differences and commonalities in light of the presented use cases and suggest areas for future work.

Item Type: Conference or Workshop Item (Paper)
Additional Information: ** From Crossref proceedings articles via Jisc Publications Router **History: published_online 31-10-2021; issued 05-09-2021; published 05-09-2021
Schools: Schools > School of Computing, Science and Engineering
Journal or Publication Title: HIP '21: The 6th International Workshop on Historical Document Imaging and Processing
Publisher: Association for Computing Machinery (ACM)
ISBN: 9781450386906
Related URLs:
Funders: German Research Foundation (DFG), Federal German Ministry of Education and Research (BMBF)
SWORD Depositor: Publications Router
Depositing User: Publications Router
Date Deposited: 10 Nov 2021 11:19
Last Modified: 15 Feb 2022 17:17
URI: https://usir.salford.ac.uk/id/eprint/62335

Actions (login required)

Edit record (repository staff only) Edit record (repository staff only)


Downloads per month over past year