How we evaluate our parser

Current Status: completed, with next (not urgent) steps.

Update logs

• 24th April 2025:
  • fixed: a bunch of errors and initiated initial run (still failures unclear where from)
  • fixed: big bugs and added back parallelisation, for now will workaround it by pruning the problems and fixing after we have more data.
  • added: gemini-2.5-flash-thinking + cell by cell updates to local .csv + local logdir
  • added: two evaluation runs
  • added: exploration of the results via features/metadata and added next steps
• 23rd April 2025:
  • added further features for the KO dataset on the page level (see rationale)
  • light testing with llamaparse and compared it against our parsing
  • currently setting up the llm-as-a-judge script so we have comparables between text_variants (still here)
• 22nd April 2025:
  • progress updates, rate limit problems and questions on sampling
• 21st April 2025:
  • added the initial specification of how we will process the parsing evaluations

Added on 21st April 2025

Overview

The most important metric in parsing is the ability of the system to convert the information in a given page into markdown text, accurately. This is because it's then used to answer questions (qualitative & qualitatively) and also impacts retrieval (although impact is to be tested).

For a given model configurations,

• [ configurations = {model_v + prompt_v} ]
• [accuracy process = average of (i to n for {img} vs {parsed_content} -> {llm_as_a_judge} -> is_accurate (bool)) ]

To get the accuracy of a {config} for a given page

To get the average accuracy of a given {config} over a set of pages:

To then compare the average accuracy across different configurations

Caveats

• assumes that the llm-as-a-judge is accurate / provides an accurate assertion which requires manual checking / verification over a stratified sample
  • ideally there is a ground truth - this can be curated for each row that is verified via a human to be accurate

[to add, 'test set distribution' / 'statistics' ]

Configurations to be tested and tested

prompt versions

v1

You role is to be an world class image to text markdown converter. You must transcribe the image you are given into markdown text as accurately as possible.

Note:
- Graphics: if there are graphics that conveys products, services, or customers, please do your best to accurately describe and represent that information
- Tables: for any tables that are shown, please convert them to proper markdown tables with | column | separators and a header row separator. Ensure there are spaces between content and the | characters.
- Charts: where there are bar charts, graphical charts, where possible, please ensure that you convert that information into a markdown table and leave a note that it is a conversion. In this conversion, please ensure that the columns and rows are correctly notated so it's clear what the numbers mean.
- Text: any text that is shown, please ensure with 100% accuracy that you capture it directly from the source image 
- Edge cases: If the formatted itself when trying to replicate mutates the meaning of image, please format the text (ideally in markdown) so that the information is still accurately conveyed. 
- General: you do not need to give your interpretation if the image is purely textual/numerical, you just need to ensure you accurately replicate it into text. Ensure you make use of markdown formatting so that it is obvious what is a subpoint, if it's a table how things are categorised, etc.
- Markdown Syntax: Ensure all code blocks are properly closed. Use appropriate heading levels (# for main headings, ## for subheadings, etc.), and proper markdown syntax for lists, tables, and emphasis.

Test set table set-up details

A curated test-set will be inside of LanceDB as it's optimised for very wide data as they're a columnar database.

Given that we'll be using llms-as-a-judge which can bring some variability in the metrics (i.e. confidence rates needs to be asserted), we will separate out the two tables between test set and actual evaluations (so that evaluations itself can have a different methodology, however, we can keep the test set the same and append to it.

It is unlikely that we'll have many different variants of evaluators, however we will make the assumption that we may have to hill climb on that. And to allow flexibility for that, each time we 'run' the evals', by default it will generate a new table on LanceDB which is different to the test set which is only going to be appended/updated.

Naming wise, the eval tables will be eval_parser_{eval_run_uid}_{timestamp}

And then those individual runs, which serves as a backlog to check over the data, can then be appended to a summary table as per below.

quick notes:

• v1 prompt is already 'hillclimbed': v1 of the prompt, some hillclimbing was already performed to enable it to take images that doesn't directly have 'text' translations. see 'Edge cases: ...' bullet point
• using appendix for prompts: as newer versions of the prompts are created, it'll be added to some appendix with some rationale on version deltas. there should be an appendix for the eval prompts too.
• should we return text_docling or text_gemini_2_0_flash? these are the two alternative parsing methods (and both are 'ideal' depending on the page composition), however needs to be validated through some evaluations.
  • text_docling will work better in text_dense + image-less pages + text_gemini_2_0_flash better for where images are present. we have a has_image condition to allow for more 'dynamic' returns however need to validate if this is true.
• lancedb is where we'll curate the test sets that are to be evaluated e.g. multiple different parsing run + different embedding runs (text variants)
  • [the metrics store is TBD]
• what is the parsing_runs table + summary table? to be answered, less open question, just need thinking

Added on 22nd April 2025

Progress update - 22nd April 2025

• Rate limits: for KO we have 3723 pages which is part of the test set, as such running 'quick' evaluations is likely to be a longer running job which will require a model with a higher rate limit to get results returned faster
• Sample or Not: I don't think taking a stratified sample is a good idea at this point as: (1) it assumes that we have enough information on the right stratas to construct; (2) it is important that all the pages are parsed correctly; (3) there are LMs that can handle X000s of RPM meaning we don't necessarily need to sample.
• Enabling LM judge vs Human alignment: there needs to be an easier to way to check through (stratified sample) the evaluations to ensure that the LM is aligned with what a human would assert
• Next steps
  • Add more text_{variant}: there are few other variants that we want to test out whilst holding the prompt fixed for now
    • v1 prompt + gemini_2_5_flash without thinking
  • Set-up LM as a judge system:
    • initial hill-climb: initially hill climb on a selection of LM as a judge as recently hit a bug on the LMs as a judge
    • pick high throughput model: pick a model that can handle X000s RPM and > 1m TPM so that results can come faster
    • think -> execute: think about the best interface/storage solution for all these rows i.e. locally (.csv) in a /dump/ then send that into a google sheets? if we add 'reasoning' then we can also leverage google sheets native AI features to interpret the results + underlying dataset
• Extract more 'features' to the pages that are being parsed: there are more features that impact parsing that should be extracted earlier which will help with both interpreting failure/successes, downstream for creating stratified samples that are representative, and also potentially for routing different parsers depending on the feature e.g. if something is not directly parsable and/or is diagram heavy, then we may need to utilise a customised prompt and/or a more powerful reasoning model + also potentially 'omitting' certain pages from being returned e.g. placeholder pages

Page-level feature mapping & feature engineering

• contains (categorical, multi-choice): [ "text", "table", "diagram" ]
  • how is information represented
• has_leakover_from (categorical): [ "page_before", "page_after" , "page_before_and_after", "no_pages" ]
  • some pages are leakover from pages before, after, both.
• sizing (bool): "vertical" | "horizontal"
  • ppts comes in horizontal and reports comes in vertical - sometimes mixes
• heavy_on: [ "text","diagram", "table", "text_and_table" , "table_and_diagram", "text_diagram", "text_and_table_and_diagram"]
  • weighting behind contains
• is_directly_parsable (bool) : true | false
  • some diagrams aren't directly parsable and some are directly parseable to markdown text examples includes: (1) tables with nested columns; (2) diagrams with multiple plots; (3) diagrams with no clear y-axis
• has_image (bool) : true | false
• has_footnotes (bool): true | false
  • whether or not the page has footnotes
• report_type (categorical): ["transcript", "presentation", "report"]
• is_valid_page (bool): true | false
• is_placeholder_page (bool): true | false

Utilising Google's AI studio to test out initial feature engineering

Defining structured output schema on Google's GUI

model = gemini-2.5-flash-preview-04-17 with thinking

system_prompt="""You are a world class machine learning engineer. Your role is to take a the given structured json and classify an image according to the structures. Below is an explanation of each.
- contains: this is a multi-select to add tags around how information is represented
- has_leakover_from: this is a variable to indicate if it seems like the page (text/diagram/table) leaks over to the next or has leakovers from before or both or none
- sizing: this is to indicate the shape of the page i.e. is it vertical or horizontal
- heavy_on: you can see this as the weighting behind `contains` i.e. what takes up most of the page space
- has_footnotes: boolean to indicate if this page contains footnotes
- is_directly_parsable: this is to assert if the page is directly parsable e.g. raw text is directly parsable but tables and diagrams with no direct text translation and/or tables with layered headings may not be directly parsable
- is_valid_page: is valid page i.e. not corrupted
- is_placeholder_page: you will be receiving a collection of financial documents, and there typically are placeholder pages 
- is_empty_page: is this page empty i.e. no text etc.?"""

Looks like it works. Will now need to run this across 3.7k rows.

Process Notes

• Replaced is_directly_parsable with parsing_difficulty - this needs to be defined even better. the concept is to see whether there is a relationship with how 'hard it is to directly parse' and the accuracy.
• Removed sizing - if it's important downstream, we don't need an LM to assert that

Running the feature extraction process

Notes:

• script gets stuck: it sometimes gets randomly stuck / fails (may be because of rate limits and/or limits of running it locally) -> added logic to allow me to re-run same script except only for the rows that have null values for the page features.
• small error: error 4 pages to run feature extraction, will ignore that for no

Revisit / Use later: now we have features per page, it'll be a lot easier to interpret failure rates and also the distribution of the dataset. More thinking needed here but now is available.

Testing LlamaParse and quick comparisons to our parser

It has an AI: I need to use... which doesn't work for us and we're using the highest

Good

Both balance and premium wrong - see % Fav/(Unfav.)

Our versions - still needs hill-climbing but it is still closer to the right output compared to LlamaParse. Surprising Flash-Lite outperformed though.

Added on 23rd April 2025

Setting up run.py for the lm-as-a-judge (latest)

problems

• odd bug: returning a very odd error which is '' for when judging the llms. will need to walk through problem more slowly to ensure it works as intended.
• still need to...: still need to add more variants first but we can already generate "variants" evals baesd on the two variants we have which is text_docling , text_gemini_2_0_flash

Current steps...

• tpm / rpm handling: need to fix this odd bug so that we can actually run the llm-as-a-judge. it's likely because we are sending 3 (number of text_variants) calls per row, and we have 3723 rows which means around ~10k requests needs to be made. On average each request should be 2.5k tkns which means around 25m tokens (mostly input as output is a bool).
• time: if we assume we use gemini's models, (i think we're on t2), we should be able to to complete these runs with 25m tokens then we should be able to complete it in around
  • 5-10 minutes with gemini-2.5-pro-preview and costs us $31.25 (25m * $1.25/mtkns)

Added on 24th April 2025

Running LLM as a judge (not parallelised)

We take the evaluation test set from supabase and for each text_* variant, we using an llm-as-a-judge to assert whether the accuracy was true or false. As each row gets updated, it'll append to a .csv locally. eval_uid will be how we can get back to the page level features + doc level metadata.

Notes:

• odd bug: there are still bugs - i.e. the gemini model returning ' ' instead of true or false. unclear if this is a model behaviour or created as a product of code.
• learnings with lm coding: models can't handle this level of complexity of task so need to build it from scratch (too much logic). although it helps with knowing what functions they mess up on as such can ensure those functions are very well built out

Next steps

• fix/bug: odd bugs and/or add retry logic AFTER full set is run i.e. update existing CSV to patch
  • 2025-04-24 09:36:54,332 — ERROR — Unexpected response format from Gemini for https://pub-64fdd2e26c484a62a0b7c73c5283b28d.r2.dev/page_images/KO_KO Q2 2021 Press Release_1725a031-cbf3-4e29-bd64-35dfaa2008c0_page_28.png. Expected 'true' or 'false', got raw:
• add: parallelisation on _variants
• quick hillclimbing: will need to quickly improve the prompt so that the true/false assertions fair from a quick sample
• run: will run two variants for now to get some initial results

Notes:

• added enforcement but still bug? added schema enforcement on gemini but still having raw empty string error which is very odd
  • 2025-04-24 09:47:06,044 — ERROR — Failed to parse JSON response from Gemini for https://pub-64fdd2e26c484a62a0b7c73c5283b28d.r2.dev/page_images/KO_Consumer Analyst Group of New York Conference 2025 - February 18%2C 2025_e8c401ca-de08-4662-a2c8-2722d0799c4a_page_5.png. Raw: ''. Error: Expecting value: line 1 column 1 (char 0)
  • 2025-04-24 09:52:59,401 — ERROR — Gemini returned an empty response for https://pub-64fdd2e26c484a62a0b7c73c5283b28d.r2.dev/page_images/KO_KO Q2 2022 Transcript_e4dd8c48-37a6-4fca-b149-b39a43884b5d_page_16.png.
• workaround: for now, to get some results, we'll prune the errors and at least initially construct an evaluation testset

Updates: added thinking to gemini-2.5-flash evals, parallelisation, and cell by cell updates to allow for resumability (see below)

parsing eval results w/ pruned LM call errors)

#1 - first run

test set description

to add further but 1k rows ran

evaluation set-up

• model: gemini-2.5-flash + thinking
• prompt_version: EVAL_SYSTEM_PROMPT_V1

EVAL_SYSTEM_PROMPT = """
Your role:
You are a world class expert in evaluating the accuracy a vision models ability to transcribe an image into markdown text.

Your input:
You will receive a single page image and a candidate text string.

Your task:
Your task is to decide whether the candidate text string is a faithful and accurate transcription of the image.

Note:
- numbers needs to be 100% accurate, so any incorrect numbers should be considered "false"
- some images cannot be directly translated into markdown as such on those, please use your judgement to assess if the meaning was preserved
- small formatting mistakes are fine, as long as the FULL meaning is preserved 

Output structure
Respond in a JSON boolean with the key "is_accurate" and a boolean value. Only use invalid if you cannot answer the question.
{
    "is_accurate": true | false | invalid
}
"""

results

caveats

• there is still for some reason a 30% failure to call LM rate, this is problematic if those calls are a product of actual 'falsities'
• we still need to manually sample the lm-as-a-judge to ensure human alignment on judgements

next steps

• run full evals: run full evals
• add: add feature data into .csv so that provides us flexibility with interpreting results
• diagnose bugs:
• sanity check: check why there's issues currently in /.log/ file

#2 - second run (1.4k rows)

same structure, 1.4k rows, too many errors though.

You can see full results sheet with preview, page level data, input+variants, etc. all in here.

There are two rows (1) Full; (2) Cleaned. There are some errors due to per day rate limits of the gemini-2.5-flash-thinking models. Should consider utilising LiteLLM as default to avoid these limits again and/or reintroducing the revolver logic.

Utilising metadata + features to explore results (1.4k rows, omitted rate limit failures)

Overall result

See above for the the 'features' and breakdowns

by event_type

event_type definition: periodic are your fiscal events (inc. 10-Ks, 10-Qs, earnings transcripts, etc.), the others are self-explanatory.

comment: docling seems to do pretty poorly on on periodic information so will need by content_type to diagnose where/why (breakdown of periodic)

by content_type

content_type definition: this is a more granular breakdown (and likely to be more helpful), the naming shold be informative enough.

comment: this doesn't make sense - earnings_transcripts docling should be excelling at as it's text heavy, it's suprising that it does best on earnings_press_releases. probably need some breakdown on release vs report vs presentation vs transcript

by has_footnote

has_footnote definition: does it have footnote definition

comment: seems like footnotes causes accuracy loss although may be due to llm-as-a-judge asserting problems due to wrongly positioned footnotes rather than

by heavy_on

heavy_on defintion:

comment: i'm very surprised that table_and_diagram are ones that has near 100% accuracy. there must be a bug as, intuitively that should be the hardest.

by parsing_difficulty

comment: this makes sense that as it gets harder, the results goes down. would need to know what constitutes as 'hard' / 'medium' / 'easy' based on the feature classification that we did. assuming that the llm-as-a-judge assertion is correct, the classification of difficulty is clearly 'right' for the lm-based parsing.

for a given pasring_difficulty, what is the heavy_on mix?

comment: this is consistent i.e. text_and_table_and_diagram are the ones that have the biggest problems. would be good to know the mix s

test set distributions based on heavy_on and parsing_difficulty

comment: none

next steps

• manually check: need to manually check through some of the discrepancies to sanity check if we have an overly lenient llm-as-a-judge:
  • is docling really that bad on text tasks?
  • are both really that good at table_and_diagram based tasks?
  • is gemini-2_0-flash really that actually that good?
• understanding the omitted/errors: does the omitted sample cause issues in the results? are the 1.4k rows that are run a representative enough sample?
• please note: we've hit our daily rate limit for gemini-2.5-flash thinking rate limits!!!!!!!!!
• add some variants: can/should start adding some variants or move onto search asap.
• clean this file: should make this file cleaner so it's easier to read
• clean-up the codebase: the codebase is a bit so hard to read, should clean-up a bit so can be re-used and/or take learnings to structure the codebase for future evals

Manual checks on LLM as a judge, stratified sample

Rationale

Goal: Select a representative sample of 100 items for manual verification, ensuring coverage across different content_types and heavy_on categories, including less frequent ones.Analysis & Strategy:

1. Dominant Categories:
   1. content_type: earnings_transcript, quarterly_report, and annual_report make up a large portion (~63%)
   2. heavy_on: text is overwhelmingly dominant (81.7%).
2. Less Frequent but Important
   1. content_type: earnings_presentation seems diverse in its heavy_on mix (stacked bar chart) despite being smaller overall. Other types like company_conference_transcript, earnings_release, etc., are also significant
   2. heavy_on: Categories like table, text_and_table, table_and_diagram, diagram, etc., are much less frequent but represent different data structures we need to verify. The none category also warrants inspection.
3. Sampling Approach: Stratified Sampling
   1. We'll use stratified sampling based on both content_type and heavy_on to ensure we capture the variety.
   2. We'll aim for proportional allocation initially but adjust to ensure a minimum number of samples (e.g., at least 1 or 2) for each existing stratum (combination of content_type and heavy_on), especially the rarer heavy_on types. This might mean slightly oversampling rare groups and slightly undersampling the most dominant (text) group compared to pure proportionality, which is acceptable for verification purposes.

Proposed Allocation Logic (Implemented in Code):

1. Group the DataFrame by content_type and heavy_on.
2. Calculate the size of each group (stratum).
3. Calculate the proportion of each stratum relative to the total dataset size.
4. Determine an ideal sample size for each stratum by multiplying its proportion by the target total sample size (100).
5. Adjust this ideal number: Take the ceiling (ceil) and ensure a minimum of 1 (clip_min(1)). This guarantees every existing combination gets at least one sample, rounding up favors smaller groups slightly.
6. Sample the calculated number of rows from each stratum. Ensure we don't try to sample more items than exist in a stratum.
7. Combine the samples from all strata.

Stratified Sample

See here for the stratified sample, 121 examples, the next step will be to manually annotate each sample, then see the correlation to the llm-as-a-judge.

Manual Annotation Results

The LLM is definitely more lenient on results than the annotator as you can see LLM >> Trues than Human although disparity is much larger for docling. This can be adjusted as the LM prompt can be hill-climbed on.

Points of Optimisations

• Prompt optimisations for v1
  • order aware: it should know where things bucket into i.e. if it studies things more carefully, it should be able to identify exactly where the positioning should be.
  • complex diagrams: when there's a complex diagram e.g. waterfall chart that cannot be directly placed into a table, it needs more guidance. there must be a way to get all the different diagrams and its associated mapping then have that as part of it's context load and/or a tool for when the classifier identifies that this is the type that exists.
  • complex tables: specifically multi-columned and/or multi-tabled, again if it had some guide around how to do this then it can be done
• Bug
  • empty values: if there are any empty values, then we should patch them for gemini flash, but if the model still asserted that as true, then that's. abug
• Odd
  • docling it seems like docling gave more semantically complete work, but that wasn't its mandate...
• Learnings
  • gemini > docling: docling seems to like missing footnotes on big tables whereas gemini captures that clean
  • gemini surprisingly is good with accuracy on numbers: it's surprisingly good at getting the numbers/content out near perfectly, but the part it fails at is when it's complicated and translating that into markdown which can be hill-climbed.
• The known error:
  • not an urgent problem now: most of the errors seems to be attributed to an API call error (for the lm-as-a-judge) e.g. rate limits rather than a problem with the parsing. this is more of an infrastructure issue which we don't need to resolve for now given we have 1.4k results + 100 human annotated datasets.

Next steps on parsing (not urgent):

• Ground truth: We have a curated list of ground truth answers here of 100 test sets and then around 1.3k tests (not human validated). That can be used to test further variants.
  • Continued curation: this dataset can be sent to a team to complete the full set of annotated parsed dataset. As it will take time, it'll be worthwhile. I think given complex tables + diagrams are tricky, may be worthwhile to great subsets from equities to annotate + get GT answers so that we have a test set for complex parsing jobs.
• Hill-climbing on gemini parser and llm-as-a-judge: there's some hill-climbing to do for complex tables + complex diagrams (i.e. context on how to parse those) AND llm-as-a-judge, to reduce the leniency
  • Adding more variants: there are a few variants which would be good to test against once we have have hill-climbed on both LMs, such as llama-parses' parser, mistrals-ocr etc. this would help to prove out if our current state of the art.l
• Errors on running 'full evals': much of the errors on running 'full evaluations' comes from
• Clean-ups: should clean-up this document at some point so that we have the option to share it with others and/or utilise for marketing material (blog / even paper)

To be revisited when we are async.

Note: the empty values in the gemini parsed values will need to be fixed (however less about evaluations so will not be bucketed here)