FinAnswerBench - v0.1

Updated as of 5th May 2025

General:

There will be three tests that matters

• FinParse - assessing the parsing accuracy of our system vs others
• FinRetrieve - assessing the search performance of our system vs others
• FinAnswer - assessing the answer performance of our system vs others

FinAnswer is the initial 'final destination' where Parse / Retrieve are big stages that gets towards an LM producing an accurate, domain-specific and quick answer. as it's a combination of implementation choices that leads to instances of answers. We will go back to FinParse/FinRetrieve at a later point.

• For each answer we'll be considering the following metrics:
  • accuracy (true | false): we'll be comparing the answers produced by a given {system} to a ground truth answer produced by a query
  • verifiability (low | medium | high): this is to assess how verifiable are outputs which is impacted by how they build the parsing+retrieval function
  • latency (s): for a given answer, how long did it take to come back
• We'll be focussed on fiscal events QA (equity_ticker + fiscal_year + fiscal_quarter) for our evaluations.

• Contextual AI
• OpenAI

Testing Contextual AI's Platform

Current View: it's a pretty neat GUI to set-u

p 'agents' abstracting away a lot of the details to essentially get an endpoint where an LM will do QA on a dataset. I think there is a 'place' for them for user e.g. they'd be valuable even for POCs... to think a bit more about it. See here for runs

Note: Contextual AI does provide page_numbers, so it'd be good to test how well correlated the answer to recall.

Note: there is also probably still some hill-climbing we can do on Contextual to reach even higher answer accuracy, but this is what we get out of the box.

To test it on a fiscal_event by fiscal_event basis, I set-up separate stores (which serves as a proxy for the fiscal_year and fiscal_quarter filtering).

This in comparison, is similar to the simple version of search in How we evaluate our search. Another test that'd be interesting is to split content_type by query, then feeding it into their Grounded Language Model to answer (replicating our simple-per-content implementation).

I also added metadata_configurability as a dimension - this is particularly important for our domain as in a given time period, the data is very deep, as such metadata filtering helps with cutting a lot of information that isn't necessary and/or ensuring periods are considered fully when the LM is generating (e.g. to do QoQ comparisons, it must've had QoQ data for each quarter)

Parsing impacts downstream tasks...

Tables

Comment:

• incomplete meaning preservation: it doesn't preserve the table's full meaning as it chunks it at odd stages -> this impacts generation as it means the LM gets context that is not fully contextualised.
• incorrect column attributions: the tables aren't parsed to markdown correctly, either i.e. numbers are under another column. This is extremely bad from a hallucinations perspective when generating answers.

Figures

Comments: It doesn't even recognise charts which means it won't also consider the that within search + generation. This limits the downstream queries contextual ai can handle. (assuming they've taken a text-based approach)

Random to consolidate notes

Consistent dimensions where we differentiate: complex diagrams, verifiability, diagrams, metadata filtering

Our idio alpha: choices we make upstream impacts downstream performance e.g. metadata filtering, choice of embedidings model (and info that can handle), parsing style (complex tables & diagrams) and finally how we steer LMs ('AGL') to get to the output quality.

I also produced a {variant}_reasoning for each result which we can feed into another LM (more powerful one) to get an understanding on what it seems to be particularly good at vs bad at and essentially surface up initial patterns. lm-as-a-results-interpreter. [To be thought of]. Hill climbing on contextual?

Patterns includes:

• query vs accuracy : what is the answer system particularly good and bad at?
• query vs latency : what queries takes longer and queries that are faster
• query_type AND query_subtype vs accuracy : what of the query tyeps, what seems to fail more than others.

config breakdown.

parse>search>answer

• parse_configs
• search_configs
• answer_configs

[to be cleaned for a better way to view 'configs' globally for any given variant']

see here for the riuns

• OpenAI Configs
  • Chunk size: 800 tokens
  • Chunk overlap: 400 tokens
  • Embedding model: text-embedding-3-large at 256 dimensions
  • Maximum number of chunks added to context: 20 (could be fewer)
  • Ranker: auto (OpenAI will choose which ranker to use)
  • Score threshold: 0 minimum ranking score
  • Generation Model: gpt-4o
  • Systems_prompt: None

OpenAI

The evaluated system demonstrates strong capability in extracting specific, factual financial data points and high-level drivers directly mentioned in the source documents (earnings releases, 10-Ks, transcripts). It reliably retrieves KPIs (revenue, margins, EPS, cash flow), guidance figures, non-GAAP adjustments, balance sheet items, and strategic commentary when the information is explicitly stated.

However, its performance degrades significantly when queries require:

1. Completeness: Ensuring all parts of a multi-faceted query are answered.
2. Numerical Accuracy: Performing precise calculations (e.g., margin bridges, reconciliations) or retrieving exact figures consistently.
3. Analytical Depth: Explaining the how and why behind financial results, comparing metrics directly, or synthesizing information beyond simple factor listing.
4. Scope Adherence: Strictly following query constraints regarding sources, analysis type (qualitative vs. quantitative), or specific contextual framing.

Common failure modes include omitting requested data points or factors, numerical errors in calculations or retrieved values, superficial explanations lacking causality or comparison, and failing to address the specific nuances or full scope of the query. Technical stability (timeouts) was also an issue.

Actionable Suggestions for Improvement

Based on the observed failure patterns, the following improvements are recommended:

• Enhance Completeness Verification: Implement a pre-generation check or post-processing step to explicitly verify that every distinct clause or question within the user's query has been addressed in the generated output.
• Improve Numerical Reasoning and Calculation:
  • Fine-tune the model specifically on arithmetic tasks involving financial data.
  • Implement internal consistency checks where components should sum to a total (e.g., margin bridge contributions, reconciliation items).
  • Improve precision in extracting specific numerical values from text, potentially using structured data extraction techniques.
• Deepen Analytical Capabilities:
  • Train the model to generate explanations that focus on causality (e.g., why refranchising impacts margins) and comparison (e.g., explicitly stating if Capex > D&A when asked to compare).
  • Focus on synthesizing relationships between different metrics (e.g., how currency-neutral growth translates to reported growth considering headwinds).
• Refine Query Scope Adherence: Improve instruction following, particularly regarding:
  • Using all specified source documents.
  • Distinguishing between requests for qualitative summaries versus quantitative data.
  • Addressing specific temporal or contextual framing (e.g., "how has guidance changed?", "commentary following an event").
• Improve Contextual Factor Identification: Enhance the model's ability to identify and include all relevant contributing factors mentioned in the source for a given phenomenon, not just the most obvious ones (e.g., ensuring all key risks or working capital drivers are captured).
• Address Technical Stability: Investigate and mitigate the causes of gateway time-outs to ensure reliable response generation.

Interpretation of Experiment Results - Contextual AI

Overall Assessment:

The evaluated system demonstrates a strong capability in answering a wide range of financial queries related to Coca-Cola, particularly in extracting specific quantitative data points and definitions from the provided documents (likely earnings releases, transcripts, 10-Q/K filings). Accuracy is high for straightforward data retrieval, KPI definitions, and summarizing stated strategies or outlook figures. However, weaknesses emerge in handling complex queries requiring complete synthesis of multiple components, precise breakdowns with accurate quantification of all drivers, and nuanced contextual understanding.

Strengths:

1. Quantitative Data Extraction: The system excels at accurately extracting specific financial figures, including revenue growth rates (organic), EPS figures, margin percentages, cash flow numbers, balance sheet values, and guidance metrics. Reasoning for accurate results frequently confirms the correct extraction of specific numbers.
2. Handling Diverse Query Types: Accuracy is demonstrated across various financial domains: Revenue, Margins, Balance Sheet, Cash Flow, Outlook, and Other (KPIs, Strategy, Risk).
3. Definition and Explanation: The system accurately defines key financial terms and non-GAAP metrics (e.g., Unit Case Volume, Concentrate Sales, Price/Mix, Structural Changes, FCF) and can explain their calculation or treatment.
4. Basic Synthesis & Multi-Part Queries: It often successfully answers multi-part questions, retrieving several distinct pieces of information requested within a single query. It can also perform basic synthesis, like explaining the difference between reported and organic growth by identifying key drivers.
5. Qualitative Information Retrieval: The system shows competence in summarizing qualitative aspects like management tone, strategic pillars, and key commentary points from earnings calls or press releases.

Weaknesses:

1. Incompleteness in Complex Queries: A primary failure mode is omitting one or more parts of a multi-faceted query. This often occurs when asked for multiple metrics, comparisons, and drivers – the system might provide some but not all requested elements (e.g., providing organic but not reported revenue, missing a comparison to depreciation, failing to address alignment with strategy).
2. Precision in Breakdowns and Quantification: The system struggles occasionally with providing fully accurate and complete breakdowns, particularly for margin drivers (e.g., quantifying basis point contributions from underlying vs. structural vs. currency) or segment performance. Numerical data provided for these breakdowns sometimes mismatches the ground truth.
3. Contextual Nuance and Scope: The system can miss subtle but important context or nuances. Examples include:
   • Failing to distinguish drivers of underlying vs. comparable metrics.
   • Not fully capturing the change aspect when asked how guidance or impacts have evolved.
   • Misinterpreting the specific impact requested (e.g., impact of working capital accounts vs. overall OCF change).
   • Occasionally including unrequested information or failing to use all specified data sources (e.g., ignoring transcript when requested).
4. Omission of Key Items: In some inaccurate cases, the system failed to identify or mention significant items crucial to the answer, such as major impairments (BodyArmor, fairlife), key risks (IRS dispute), specific refranchising locations, or major cash outflows (dividends impacting equity).
5. Calculation and Consistency Errors: While less frequent, there were instances of incorrect calculations (e.g., summing non-GAAP adjustments) or internal inconsistencies within an answer (e.g., debt components not matching stated total debt).

Patterns:

• Accuracy is highest on queries asking for specific, well-defined data points or definitions.
• Inaccuracy increases with query complexity, particularly when requiring detailed, multi-component breakdowns, comparisons involving subtle distinctions (e.g., underlying vs. comparable), or the synthesis of all contributing factors with precise quantification.
• Failures often involve omission rather than outright factual hallucination, suggesting issues with retrieval completeness or attention to all query constraints.

Actionable Improvement Suggestions

Based on the observed patterns of failure, here are specific suggestions for improving the system:

• Enhance Query Deconstruction & Completeness Check:
  • Implement a more robust query deconstruction mechanism to explicitly identify all distinct questions or data points requested within a complex query.
  • Before finalizing the answer, perform a checklist-style validation against the deconstructed query components to ensure every part has been addressed.
• Improve Numerical Reasoning & Consistency:
  • Strengthen internal checks for numerical consistency within the generated answer (e.g., ensuring components sum to totals, reconciling reported vs. comparable figures correctly).
  • Refine the model's ability to accurately extract and attribute specific quantitative impacts, especially for multi-driver analyses like margin bridges (basis point contributions). This might involve better grounding in tables or structured data extraction.
• Refine Contextual Understanding & Nuance:
  • Fine-tune the model on examples highlighting subtle distinctions common in financial analysis (e.g., reported vs. comparable vs. underlying; timing impacts vs. structural impacts; growth drivers vs. margin drivers).
  • Train the model to specifically recognize and address comparative language (e.g., "how did it change?", "compare X to Y", "relative to Z").
  • Improve understanding of non-GAAP reconciliation logic, emphasizing why items are excluded and their impact on the reported figures.
• Boost Salience Detection for Key Items:
  • Improve the system's ability to identify and retrieve financially significant events or items mentioned in the source documents, even if they are one component among many (e.g., large impairments, specific legal contingencies like the IRS dispute, major M&A impacts like fairlife). This could involve better entity recognition and financial event tagging.
• Strict Scope Adherence:
  • Reinforce instructions to adhere strictly to the user's query scope, avoiding the inclusion of excessive, unrequested information.
  • Ensure the system attempts to use all specified source types (e.g., transcript, 10-K) if available and relevant to the query. Address any underlying data access limitations.