Optical Character Recognition (OCR) of contemporary printed fonts is widely considered as a solved problem for which many commercial and open source software products exist. However, the task of text recognition on early printed books is still a challenging task due to a high variability in typeset, additional characters, or low scan quality. To master OCR on early printed books, a book or font specific model must be trained to achieve CERs below 2% [Reul et al. 2017a] [Springmann and Lüdeling 2017]. For this purpose, highly performant individual models must be trained in a short period of time using as less manually annotated GT files as possible. Currently, there exist several Free Open Source Software (FOSS) attempts for such programs: OCRopy, OCRopus 3, Kraken, or Tesseract 4, each with its own advantages or drawbacks.

Calamari[1] is a novel software for training and applying OCR models in order to predict[2] text lines including several up-to-date techniques to highly optimize the computation time and the performance of the models. The usage of Tensorflow allows to design arbitrary DNNs including CNN and LSTM structures that are proven to yield improved results compared to shallow network structures [Breuel 2017] [Wick et al. 2018]. These networks can optionally use CUDA and cuDNN (on a supported GPU) which results in a highly reduced computation time. Compared to other FOSS Calamari supports various techniques that minimize the CER including voting and pretraining (see Reul et al., 2018a, Wick et al., 2018). The software is not designed as a full OCR pipeline which includes tasks such as layout analysis, or line segmentation, but is the topic of a separate publication (in preparation), instead it focuses solely on the OCR step that transcribes line images to text. However, Calamari as Python-Package[3] can easily be integrated in existing pipelines to manage the OCR part. Thus, by design without any changes it can directly replace the OCR-Engine of OCRopy.

In the following, we give a short list of the existing open source OCR programs OCRopy, OCRopus 3, Tesseract 4, and Kraken. All of these systems are designed to support the full pipeline from plain page to text. Since the intention of Calamari is solely to handle the OCR of line images, only this functionality of the other programs will be described and compared.

Currently, there exist several versions of OCRopy which was originally published by Breuel (2008) (see also Breuel et al., 2013), that are still maintained. OCRopy[4] is the first software that allowed a user to train custom LSTM based models incorporating the CTC-Loss function [Graves et al. 2006]. By default, it uses a slow numpy based implementation of the computation, which can be exchanged by a faster C-based clstm one. However, neither the GPU nor Deep CNN-LSTM models can be used. For training it requires a list of images and their GT and outputs a model, which then can be used to automatically transcribe the written text of other text lines.

Kraken[5] is a fork of OCRopy, which has a different API, uses clstm as default, and adds support for bidirectional and top to bottom text. Currently, the usage of PyTorch[6] as Deep Learning engine supporting GPU training is under development.

While OCRopy is still maintained OCRopy 2[7] seems not to be developed anymore, probably due to the introduction of OCRopus 3[8] which changed all major OCR components to Deep Models using PyTorch. OCRopus 3 supports variable network architectures including CNNs and LSTMs and allows training and applying of the models on the GPU. The resulting models yield state-of-the-art results and can be trained with minimal effort in time.

Tesseract[9] was initially released as open source in 2005 and is still under development. The newest version 4 of Tesseract adds support for deep network architectures such as LSTM-CNN-Hybrids, however GPU support is not offered. To prototype network structures Tesseract proposes a Variable-size Graph Specification Language (VGSL) which is similar to the network prototype language of Calamari.

Calamari comprises several techniques to achieve state-of-the art results on both contemporary prints and historical prints. Beside different DNN architectures (see Appendix A), it supports confidence voting of different predictions and finetuning with codec adaption. These methods will be presented in the following.

Calamari supports easy instructions to use any of the listed methods to train various models, and to apply them on existing lines. The software is implemented in Python3 and uses Tensorflow for Deep Learning of the neural net. In doing so, Calamari supports usage of the GPU including the highly optimized cuDNN kernels provided by NVIDIA for a rapid training and an automatic transcription of multiple lines (batches) simultaneously.

To compare the performance of Calamari to OCRopy, OCRopus 3, and Tesseract 4 we use the datasets UW3 and DTA19. All final results of Calamari were achieved by using early stopping. Hereby we check every 20,000 iterations for a smaller CER on 20% of the training data (hence 80% are used for actual training), after ten checks without a better model, we interrupt the training. Similarly, we chose the best OCRopy or OCRopus 3 model based on the same 20% of training data. Tesseract 4 itself chooses its best model on the provided test data set. For Calamari and OCRopus 3 we use a batch size of 5, the OCRopy and Tesseract 4 do not support batching.

The main focus of this paper was the presentation of Calamari as new line based OCR engine to replace OCRopy or Tesseract due to its low CER and fast computation times.

The conducted experiments clearly demonstrate the capabilities of Calamari on both contemporary and historical OCR tasks. Not only does Calamari yield outstanding performances compared to other OpenSource OCR software but also requires a minimum of time for training and prediction due to the usage of Tensorflow as Deep Learning framework including cuDNN. As already shown by Breuel (2017) and Wick et al. (2018), Deep Hybrid Convolutional-LSTM architectures increase the accuracies both on contemporary and historical prints. Our results have revealed significant differences of Tesseract 4, OCRopus 3 and Calamari due to variations of the network architectures and different sets of hyperparameters. It is to be expected that an optimization of these hyperparameters might further improve the accuracies of the OCR models. However, a very high amount of the remaining errors on UW3 and DTA19 are GT inconsistencies or impure lines, which are nearly impossible to predict correctly.

Voting and pretraining are two important mechanisms to increase the performance of newly trained models, especially if only few data is available. Voting has shown improved results on all conducted experiments, however on the cost of a higher prediction and training time, since several different models are independently used. Pretraining is most useful if the original model is similar to the new data and reduced both the CER and the training time. Especially, when training new individual models as required for OCR on historical prints pretraining should be used to receive the best possible results if only a few manually annotated lines are available. In general, the best OCR results are to be expected if the targeted fonts of a pretrained model are similar to the material at hand. This of course correlates with the period of publication of the various books lies in the same epoch.

Although Calamari supports many features such as voting, or pretraining, plans for extensions exist. A first major work is data augmentation during training which is expected to significantly drop the CER especially if only a few lines are present. The augmentations basically are different types of noise, degradation, or generated background. Obviously, synthetic data based on existing fonts can also be incorporated for data augmentation.

Tesseract's language to define network topologies (VGSL) has a very simple and compact syntax. The current syntax of Calamari should also support this language to define networks.

Finally, since Calamari is designed very modular, it shall be extended to support other sequence-to-sequence tasks, such as monophonic Optical Music Recognition (e.g. Gregorian chants) or Speech-To-Text. All these tasks fundamentally share the tasks of processing two dimensional sequential data and output a sequence of classes. Only the data preprocessing e.g. of audio files and postprocessing is different. Calamari is designed to easily exchange these steps but keeping a generic structure for training, evaluation, and application.

The task of the DNN and its decoder is to process the image of a segmented text line and to output simple text. This sequence to sequence task is trained by the CTC algorithm published by Graves et al. (2006) that allows to predict shorter but arbitrary label sequences out of an input sequence that is the line image regarded as sequence. Hereby, the network outputs a probability distribution for each possible character in the alphabet for each horizontal pixel position of the line. Thus, an image with size hxw with a given alphabet size of |L| will result in a matrix of shape P(x,l)∈Rwx|L| with P(x,l) being a probability distribution for all x. Since the network needs to see several slices in width to be certain about a single character, most of the time it does not yet know what to predict. Hence, the CTC-algorithm adds a blank label to the alphabet that is ignored by the decoder but allows the network to make empty or uncertain predictions with a high probability. In fact, the used greedy decoder that chooses the character with the highest probability at each position x mostly predicts blank labels, and only with one or two pixel widths the actual character is recognized. Afterwards, the final decoding result is received by unifying neighbouring predictions of the same characters and removing all blank labels. For example the sentence AA--BB--CA--A-¿ of length w is reduced to ABCAA.

The network is trained by the CTC-Loss-Function that basically computes the probability of the GT label sequence given the probability output of the network. This probability is computed by summing up all possible paths through the probability matrix P that yield the GT using an efficient forward backward algorithm. Fortunately, this computation can be derived to receive the gradient that is required for the learning algorithm.

The supported network architectures of Calamari are CNN-LSTM-Hybrids that act on a full line in one step. CNNs are widely used in image processing because they are designed to detect meaningful features (e.g. curves, circles, lines, or corners) that can be located anywhere in an image. These features are processed afterwards by a (bidirectional) LSTM based recurrent network to compute the probability matrix P. Max-Pooling is a common technique in CNNs to reduce the computation effort and keep only the most important features. In general image processing settings pooling is applied both in vertical and horizontal direction. This means that the processed lines get shortened. Thus, it is important that the final layer of the CNN in the full network has an image width, that is long enough to produce the full label sequence. For example, if the GT has a length of 40 characters, a minimum of 80 predictions are required to allow for a blank prediction between any pair of adjacent characters. Thus, if two 2x2 pooling layers are used in the CNN the width of the image lines should be at least w=2·2·80px=320px .

The default network consists of two pairs of convolution and pooling layers with a ReLU-Activation function, a following bidirectional LSTM layer, and an output layer which predicts probabilities for the alphabet. Both convolution layers have a kernel size of 3x3 with zero padding of one pixel. The first layer has 64 filters, the second layer 128 filters. The pooling layers implement Max-Pooling with a kernel size and stride of 2x2 . Each LSTM layer (forwards and backwards) has 200 hidden states that are concatenated to serve as input for the final output layer. During training we apply dropout [Srivastava et al. 2014] with a rate of 0.5 to the concatenated LSTM output to prevent overfitting. The loss is computed by the CTC-Algorithm given the output layer’s predictions and the GT label sequence.

Calamari uses Adam [Kingma and Ba 2014] as standard solver with a learning rate of 0.001. To tackle the exploding gradient problem of the LSTMs we implement gradient clipping on the global norm of all gradients as recommended by Pascanu et al. (2013).