FinRetrieveBench v0.1
Updated as of 1st May 2025
Note: this document will be evolving as our benchmarks evolve and scoring mechanism evolves. It is still currently WIP.
This is a follow on from How we evaluate our search. In this piece, we'll be setting up the experiments to benchmark our search system against alternative search implementations.
What we are testing
We are testing different search set-ups ability to surface up relevant information given a set of financial queries. Specifically, we're testing the ability of a given search system to surface up relevant information given a fiscal_event which is defined.
There are four dimensions by which we are comparing ourselves against others: avg_recall , avg_precision, avg_latency, and source_verifiability. source_verifiability is a metric that I added after using OpenAI's RAG implementation where I realised that verifying where chunks came from was not trivial.
Test set-up
Who - who are we comparing ourselves against?
We'll be testing against out of the box implementations of RAG (taking the default configurations where possible) and also current search providers e.g. Alpha Sense.
Note: we should probably also test out using the web search route to elucidate the idea that 'ChatGPT' or Perplexity are non starters.
For each, we'll do two types of search: (1) simple; (2) per content type (dependent on if the provider allows for metadata filtering).
| entity | metadata | parsing_method | chunking_method | embedding_model |
|---|---|---|---|---|
| openai | fiscal_quarter + fiscal_year | uncertain | auto | uncertain |
| llamaindex | fiscal_quarter + fiscal_year | balanced | page | cohere |
TODO: Each one will need a method name to identify the unique way of approaching RAG. The main that is tricky is metadata (which we can probably use the filter sets concept).
What - what's the metrics that we'll judge all the systems on?
You can read into How we evaluate our search for more details (and above), however below are four dimensions.
| method | avg_recall | avg_precision | avg_latency | verifiability |
|---|---|---|---|---|
How - how will we in practice normalise any implementations to get our 'What'?
In practice, as varying implementations won't naturally lend to easily comparing against our search benchmark - the main metric is the assertion of recall. There will be a few workarounds we need to make e.g. adding an LM to assert what page it came from to compare against whether it got the right page_number(s).
OpenAI RAG
Current:
- page_attribution workaround: using an lm to essentially attribute where a page came from so that we can get the recall metric.
- comment: the llms can't actually one-shot identify the page_targets well which is frustrating.