Sign in Subscribe

Evaluating at Wissen

Updated as of 7th May 2025

[ ] = section

  • = points within section that will be mentioned

Skeleton

[purpose of evaluations generally, its importance in our domain, and what it means from 'using an ai product' pov]

  • evaluation is simply observing whether a system, given an input, produces what the user/developer functionally expects
  • traditional software is able to capture 'evaluations' determinstically whereas the introduction of LMs means that evaluations are inherently statistical due to to the stochastic nature of lms
  • evaluations are important as it is what: (1) what the system is expected to handle; (2) what the system has been optimised towards.
  • functionally, this means expecting to handle {user_queries} and achieving {system_outputs}

[quick summary/tldr of 'evaluations' in our domain and how they're not ecologically valid]

  • existing benchmarks are not ecologically valid: the queries themselves are not representative of real valuable/useful queries analysts would ask a system
  • the difficulty comes into appreciating the shape of queries that analysts would ask and ensuring that evaluations do accurately represent that

[what's important to evaluate? finparse > finretriev > fingeneration]

  • there are three areas of evaluations that are interlinked sequentially - parsing > retrieving > generating
    • parsing - this is how documents are converted: (1) a machine-readable format for generation; (2) a format that can be searched over
    • retrieving - this is how information is surfaced for a given query for generation and/or directly to the user
    • generating - this is where the retrieved parsed content is passed over to a system of LMs to generate a response
  • relations:
    • parsing -> retrieval: subpar quality in parsing impedes on retrieval and if not parsed (most likely case), it cannot be searched over
    • parsing -> generation: subpar parsed outputs e.g. either missed diagrams/tables means higher chance of hallucination
    • retrieval -> generation: subpar retrieval means LMs are more likely to hallucinate AND/OR unable to handle wider sets of queries

[how we create the benchmark]

  • we've curated a list of queries from the domain utilised a novel method to increase the representation of those queries in the benchmarks
  • this is to ensure that we cover fully the different representations a query can take shape e.g. lots of queries are the same but linguistically different.
  • as we get more queries, we extract the features behind those queries to ensure we can trace performance across different categories

[examples of finparse/finretriev/finanswer]

  • teasers of it each benchmark with some commentary in a table

[finanswer results - us vs others performance AND why the delta]

  • we only showcase finanswer for now as that is what 'most of the work leads up to' and we'll do the others later
  • showcase results us (variants) vs openai vs contextual ai vs {other}
  • conclusion: we attribute three core reasons why our system outperforms
    • parsing: our system is created & expects to handle figures & complex tables as the base case as such at generation, it is handled as it does with simple text
    • retrieval: our system can search for diagrams on the same comparable metric to text using a new indexing method. this means that when lms are generating, it is 'actually considering' the full set of important information rather just text
    • generation: we've buit the system to talk, answer and reason as closely to how world-class analysts would with evidence driven reasoning and brevity in answers

[cta - some call to action]

  • partnering with you on benchmarking your providers to see if they perform adequtely
  • adding your queries to the dataset so that it ai systems built for finance is as representative as possible
  • working with our white-glove service as we build out the platform