Our journey thorugh retrieval

Status: [STILL BEING WORKED ON DONT READ INTO JUST YET PLS]

Before we start, why is retrieval important?

Retrieval is important for both LMs and humans as it serves as an access point for information. That access point needs to make it possible for whoever's interfacing it to (1) find the information they desire; (2) find the information is a useful time frame.

If we look at many of our predecessors, they have been either data providers (Bloomberg), search providers (AlphaSense), and/or or data structuring providers (BamSec).

Improving retrieval itself is valuable because it allows people to find information they want at the point of wanting the information.

Why is it fundamentally hard?

Varying representations of information makes standardisation hard: information can be communicated in different methods e.g, bar charts, waterfall charts, tables, text, footnotes within presentations, long pdfs, podcasts and more. Unifying that into a machine-readable format is hard.
Most technology were built for humans not LMs: if we look at the existing search providers such AlphaSense, BamSec, and more, they were build for humans i.e. specifically to find specific documents. When even one document can have 50-500 pages, it becomes difficult for LMs to 'reason' over it correctly. Humans also naturally have more tools and knowledge to navigate through the document space, LMs do not. As such a lot retrieval today is about enabling an access point that points to more specific sets of information than just entire documents.
The information people want changes: the questions that humans and/or LMs will want to ask will change as time goes on. This makes sense because as more information gets produced, the right and relevant questions to ask changes, however, this means that retrieval systems needs to be able to handle for the constantly changing inputs. It would be too capital efficient and unscalable to constantly handle queries.

Consensus: text-based embeddings + full-text search

In the cannonical RAG paper, the premise of Retrieval Augmented Generation ("RAG") was really the idea that if at the point of a given query, if you can give the right context to an LLM, it will perform much better. The paper was bsaed on the premise of a text-based embeddings approach.

This way of approaching solving retrieval carries a lot of baggage and assumptions because of what the implementation itself assumes.

parsing: to get to the text that can be searched over, you need to have all documents in a text-form. this assumes that all information can be accurately parsed e.g. see this presentation. this assumes a dimension of unsolved tech debt and questionable workarounds. it also raises the question of whether text is the. best way to interpret that information.
chunking: chunking is required because of the input context limitations which is around 6.1k tokens (text-embedding-3-small) to 32k tokens (voyage embedding models). When the search is run, you are returning chunks of text and those text chunks needs to be semantically meaningful and complete. This is tricky if information doesn't come in a standardised format you know what semantic completion really means and if that semantic completion is also a moving target. Moreover, are chunks really the best way by which one should get information?

Note: I intentionally do not mention FTS. These approaches are not mutually exclusive rather there are certain implementations that requires teams to 'build around it'. Building around text-based embeddings means introducing potential issues that one doesn't need to face now and/or at all is the view this piece articulates.

If we look at our domain, where we have information represented in the form diagrams, text, tables (in combination) and semantic completeness itself can vary, if one decides that the text-based approach is the optimal way forward one would need to first solve pdf parsing then solve semantic completion which itself is a seperate business.

The question that I asked was: does it really need to be the case? Which is where alternative implementations came around.

Instead, let's index based on images

You can see here for the rough architecture of our retrieval system.

There are a few things that we want: (1) the ability to search over information that is important to us; (2) not taking the trade-offs a text-based approach captures.

An image-based indexing method naturally solves a lot of the assumptions that the text-based approaches takes

semantic completion: most PDFs have already been worked in a way where pages are mostly semantic complete if viewed on a page-to-page level (and was also the chosen way by which the person who produced the information the way they wanted it to be communicated)
decoupling search & parse: this means that we can decouple search and parsing - i.e. you can still search over to find a smaller unit of information than a full document without needing to assume an world-class parser because you are just looking at images from a pdf
extensibility as a necessary property: as queries change, the concept doesn't need to be as specific to a class of texts semantically rather it just need to be semantically similar to different types of pages with different information representations. this itself is helpful as one can leverage more abstract basis (and improving on that abstract basis) without building in future fragility.

It's important to not forget that the purpose of technology is scale which means ideally as one improves a component, it raises the level of others.