Palladia
Beta
A Benchmarking Tool for Vision LLMs
Introducing a benchmark for comparing the performance of various SOTA Visual Language Models (VLM) on historical image documents, based on the GT4HistOCR dataset.
DISCLAIMER: This tool is currently in development and provided as a beta version. Results may be incomplete, inconsistent, or subject to change. Use at your own discretion, and do not rely on the tool for critical evaluations or decisions.
Evaluation Metrics
Accuracy
Percentage of characters that match exactly between the OCR output and ground truth text. Higher values indicate better performance.
Character Error Rate (CER)
Ratio of character-level errors (insertions, deletions, substitutions) to the total number of characters in the ground truth. Lower values indicate better performance.
Word Error Rate (WER)
Ratio of word-level errors (insertions, deletions, substitutions) to the total number of words in the ground truth. Lower values indicate better performance.
Execution Time
Average time taken by each model to process and transcribe an image, measured in seconds. Lower values indicate faster processing.
Leaderboard
Results by Category
Public API
Access the available public benchmarks directly through the JSON API. You can see all the available files at:
https://dassoo.github.io/Palladia/data/json/GT4HistOCR/corpus/{category}/{document}/{filename}.json
https://dassoo.github.io/Palladia/data/json/GT4HistOCR/corpus/{category}/{document}.json
Usage Examples
const category = 'EarlyModernLatin';
const document = '1564-Thucydides-Valla';
const filename = '00363.bin';
const url = `https://dassoo.github.io/Palladia/data/json/GT4HistOCR/corpus/${category}/${document}/${filename}.json`;
fetch(url)
.then(response => response.json())
.then(data => {
console.log('OCR Data:', data);
})
.catch(error => console.error('Error:', error));import requests
category = 'EarlyModernLatin'
document = '1564-Thucydides-Valla'
filename = '00363.bin'
url = f'https://dassoo.github.io/Palladia/data/json/GT4HistOCR/corpus/{category}/{document}/{filename}.json'
response = requests.get(url)
if response.status_code == 200:
data = response.json()
print(data)
else:
print(f'Error: {response.status_code}')curl \
-H "Accept: application/json" \
https://dassoo.github.io/Palladia/data/json/GT4HistOCR/corpus/EarlyModernLatin/1564-Thucydides-Valla/00363.bin.json
Data Structure
{
"model_name_1": {
"gt": "Ground truth text for this specific image...",
"response": "Model OCR output for this image...",
"wer": 33.33333333333333,
"cer": 4.838709677419355,
"accuracy": 95.16129032258065,
"time": 1.988757848739624
},
"model_name_2": {...}
}{
"model_name_1": {
"source": "Path to the document",
"images": 58,
"avg_wer": 32.248594317559835,
"avg_cer": 8.362065842702526,
"avg_accuracy": 91.99533595687504,
"avg_time": 1.548311759685648
},
"model_name_2": {...}
}