Reducto - testing as a potential provider for parsing

Context

Before we avoided utilising reducto as:

• they hadn't natively supported charts (diagrams) which were important to us
• they werent' as well-capitalised as such depending on them is fragile

Much of those priors have changed, they raised $25m recently and also support varying types of parsing.

Open questions:

• embed is a parameter that will be given. I think this is pretty good to ensure that we have consistently the right text (and some that enters) into our retrieval system
• tables in md vs html: reducto makes a pretty good point on html vs md for tables where a more natural format may just be in html (esp for more complex/layered tables)
• patching where reducto fails: they are pretty weak at company_presentations etc. but ideally we're able to conform to the same formats as them and/or to us if we do employ our own.

Old

View

• Saving efforts on engineering end: we won't need to scale a lot of the infrastructure for processing information (which is a common area for 'bad parsing')
• We are likely on-par/slightly better than them: I think we is on-par/slightly better, and we can focus on areas they fail to make sure we're best (i.e. we can ride the beta from them)

Questions

• general
  • cap on pages? do they have a cap on the number of pages one can process per month?
  • is it only a monthly commitment i.e. we can cancel after a month?
  • accuracy assurances: what are the accuracy assurances they can provide? how does that vary from simple > complicated (tables / diagrams)?
  • processing 100k pages: if we send X00k to process as jobs, do we need webhooks live to ensure the data doesn't get lost and/or is there some temporary storage?
  • can we use GUI? is there a way we can use your GUI to read over the some of the JSONs (helpful for manual evals)
• specific
  • row column attribution: we found quite a few documents where the row column that describes the rows is detected separately to the table, is this a known bug/trade-off? and how can we fix that?
  • complete meaning: there are some diagrams/charts that needs to be bundled together and/or in order for it to be complete e.g. heading -> text -> numbers etc. - how can we ensure that

Thoughts on parsing process:

1. If we get any new content_types that are very different to what we know, then we can use their GUI to manually build conviction
2. There are certain types of content_type(s) where it'll require more checking through per equity e.g. company_presentations industry_presentations as the format/portrayal of information can vary whereas 10-Ks/10-Qs has much less volatility

Current Status

• currently cannot test it across all the tests I want to run, but will book a call with them.
• current view:
  • time saved: i think it is worth working with them to get our equities processed as they can manage the scaling infrastructure + failures and also have a pretty good interface view for extraction jobs
  • focus: this means we can place more time onto retrieval reasoning whilst spending the money on areas where we will likely have low alpha for now.

ideas

• each page is a part of a 'story' that the company is trying to explain. a document a container for that story. if you can capture the story, that is most important as the 'first' dive people care about.(?)

Reducto Tests Set-up.

Below are the content_types we want to test on reducto, ratings will be out of 10 (i.e. 10 being a perfect extraction and 1 being crap, and 9 being nearly there). Comment will provide some more context on what is failing. Each content_type will have it's own section so that we have a backlog area.

Even if it's the case that they have failings, if we can get confidence on 80% of it being handled well, we can focus our parsing efforts on more tricky items and/or patch what reducto fails on.

Note to self: will need to add pricing somewhere...

Reducto's Pricing

We have around 3.7k pages for 5 years worth of information with KO. Let's assume that we have around 4k pages per equity.

Thoughts

At $1825/month plan, we can process ca. 40 equities per month (150k pages). Their interface is also suitable for quick testing and checking over what PDFs seems to have problems but it will most definitely save us time in terms of scaling efforts e.g. having to process 100ks worth of pages at once.

It'll also let us focus on my high leverage tasks such as retrieval + agentic flows rather than on parsing. As Reducto gets better + better captialised, we can assume they'll get better and/or we can patch temporary issues.

Tests on Reducto

Annual Report (10-K)

Note: there are other annual_report(s) than just 10-Ks. 10-Ks are regulatory such as NVDA's annual report. They can produce more/different information that in the filings that are required.

Overall: 10/10

Comment: gets everything and also in the right 'borders'. From checking over the parsing, it sees to be accurate. Quite good for us that they treat tables as a separate thing (makes it easier for us to aggregate tables and/or facilitate 'copying' from tables.

Quarterly Report (10-Q)

Overall: 10/10

Comment: same performance as with 10-K.

Earnings Presentation

Overall: 5/10

Comments:

Chart (Waterfall) - Pretty good

I think if they could add metadata and/or we add metadata that it's a waterfall chart, then that'd be helpful downstream to attempt reconstruction and/or for LM as context on what story is the table trying to communicate.

Tables - bad

wrong row label column: this is important because the numbers are given meaning by the row label column. even the reducto's ai table summary is

Earnings Call Transcript

Overall: 10/10

Comment: gets it done well.

Earnings Press Release

Overall: 8/10

Comment: would be good if they could recognise footnotes individually rather than as a bundle. The row label attribution is smth we can probably detect and handle ourselves. And we can push them to do better.

Bad

They seem to have trouble with the row label column when it's further away to attribute it as part of the table. This can be problematic as we're not getting the full meaning of tables and/or ordering of the 'titles' can create confused meaning.

Good

Industry Conference Presentation (CAGNY)

Overall: 7/10

Comment: A lot of the issues are around semantic completion rather than textual completion. Caveat'd that this isn't on their agentic mode which will be the next test.

[BAD] It'd be good if the figures and numbers were more semantically complete e.g. held via a table.

[ODD] Different way of recognising the page (i.e. this page it recognises the full screenshot(ish))

[BAD] How do you attribute those figures if not via ordering? Tbh this should be relatively easy to detect & post-process as we need to.

[BAD] Extracted figure content is a little stupid.

[BAD] Missess out on the legend date as such is not complete. Row label attributions e.g. RED is Coca Cola, etc.

[GOOD] Pretty Impressive. And also getting the image too is super helpful for sourcing these figures directly. Also manages to get individual logos out.

Extra Reducto Research

Great if they can handle the background processing of PDFs and other types of information

Sometimes you have a huge number of documents to parse. Reducto will autoscale to handle even the largest of jobs (millions of documents) with ease. (source)

Extra configurations - but can't see the delta after setting it to 'agentic'