Semantic Drift

Sign in Subscribe

King Choo

Evaluating at Wissen

Updated as of 7th May 2025 [ ] = section * = points within section that will be mentioned Skeleton [purpose of evaluations generally, its importance in our domain, and what it means from 'using an ai product' pov] * evaluation is simply observing whether a system, given an input, produces what the user/

FinAnswerBench - v0.1

Updated as of 5th May 2025 General: There will be three tests that matters * FinParse - assessing the parsing accuracy of our system vs others * FinRetrieve - assessing the search performance of our system vs others * FinAnswer - assessing the answer performance of our system vs others FinAnswer is the initial 'final

FinRetrieveBench v0.1

Updated as of 1st May 2025 Note: this document will be evolving as our benchmarks evolve and scoring mechanism evolves. It is still currently WIP. This is a follow on from How we evaluate our search. In this piece, we'll be setting up the experiments to benchmark our

Vault - v.0.1

Updated: 28th April 2025 Feature Context This is a build out of vault inspired from our previous demos where we take an opinionated view on the ontology on behalf of the users to make the navigation experience of the documents much more delightful & faster. Future states of Vault In

Our journey thorugh retrieval

Status: [STILL BEING WORKED ON DONT READ INTO JUST YET PLS] Before we start, why is retrieval important? Retrieval is important for both LMs and humans as it serves as an access point for information. That access point needs to make it possible for whoever's interfacing it to

Reducto - testing as a potential provider for parsing

Context Before we avoided utilising reducto as: * they hadn't natively supported charts (diagrams) which were important to us * they werent' as well-capitalised as such depending on them is fragile Much of those priors have changed, they raised $25m recently and also support varying types of parsing.

How we evaluate our search

Disclaimer: our journey to the current retrieval system is part of a larger topic area and it will have it's long form post. This part will be talking about how we evaluate our search engine in an ecologically valid manner and also posting the results. Current Status: write-