Agent: Financial Metric and Commentary Extraction

This agent workflow extracts quarterly (Q1-Q4) and full-year (FY) financial figures and commentary for specified metrics of a given company and fiscal year.

Mermaid Diagram:

Claude Artifact

Try out Artifacts created by Claude users

Workflow:

1. Input & Preprocessing: Takes user input (equity ticker, fiscal year string, raw comma-separated metrics). An LLM structures the metrics into IDs/names and parses the fiscal year integer.
2. Verification (Parallel per Metric):
   • Searches company filings (10-K/10-Q) via the Wissen API using the metric name as a query.
   • An LLM analyzes the search results to verify if the metric is mentioned, extracts the exact name used in filings and a concise definition/context snippet, marking verification as successful or failed.
3. Human Approval (Pause): The graph interrupts, presenting the verification results (including successes/failures and extracted names/definitions) to the user for approval ("yes/no") before proceeding with data extraction.
4. Data Extraction (Parallel per Verified Metric):
   • For each metric the user approved and which was successfully verified:
     • It loops through Q1, Q2, Q3, Q4, and FY periods.
     • Initial Attempt:
       • An LLM generates period-specific queries for both the numerical value and commentary.
       • Wissen API is called twice: once querying 10-K/10-Q for the value, and once querying 10-K/10-Q plus earnings transcripts for commentary.
       • Separate LLM calls attempt to extract the structured value (number, unit, context) and summarize the commentary (focusing on quantitative details and drivers).
     • Retry Logic (up to 3 attempts): If the initial attempt fails to find the value or adequate commentary, a specialized "Retry Reasoner" LLM analyzes the failure, suggests an improved query if feasible, and triggers a retry (another Wissen call + LLM extraction/summary).
     • The results for the period (value + commentary, potentially from a retry) are stored.
   • Finally, it aggregates the Q1-FY results for the processed metric.
5. Summarisation (Reduce): Collects the detailed results from all processed metrics and uses a final LLM call to generate a Markdown table presenting the data with columns (Metric Name | Q1 | Q2 | Q3 | Q4 | FY) and two rows per metric (Value, Commentary).

Essentially, it's a verify-then-extract process with human oversight, leveraging parallel execution and an embedded, LLM-guided retry mechanism within periods to enhance data retrieval robustness.

Technical Details:

• Model: OpenAI GPT-4o-mini
• Orchestration: LangGraph
• Observation: LangGraph Studio
• Name: financial_extractor_v2
• GitHub Repo: langchain_academy: w_simple_analyst
• Code Status: Good shape

Latency:

• Total latency: 130-150 seconds
  • Latency pre-validation: 8 seconds
  • Latency post-validation: 120-140 seconds
• Key sources of latency:
  • Process Metrics: 90 seconds
    • 10-12 seconds per metric per quarter / fiscal period (50 seconds overall) processed sequentially
    • Occasional call (1 in 10) to openAI 4o-mini takes 30 seconds instead of 1-2 seconds
• Summarise results and produce output: 10-30 seconds
• Potential Latency:
  • Total: 30 seconds, with user input 8 seconds in and streaming from <1 seconds
  • Pre-validation: 8 seconds
  • Post Validation: 20 seconds
  • Key source of latency reduction:
    • Running each quarterly / fiscal period call per metric in parallel rather than sequentially
    • Solving the issue of the occasional call to OpenAI (1 in 10) taking 30 seconds instead of <2 seconds

Cost per run:

• GPT-40-mini: $0.05
• GPT-4o: $1.10

Key Problems:

1. When looking at Q4 metric values and commentary, the LLM typically, is on occasion (roughly 20% of the time), unable to differentiate between Q4 and full year
2. The metric extracted by the LLM are wrong 30-40% of the time - this is a prompting fix / cured by using different LLMs. I suspect changing to Gemini 2 flash will solve the issue for the metrics should eventually be present in the retrieved context given we have 3 retries for each metric
3. Per metric per year there are 20-25 calls an LLM being made (5-6 calls per metric per quarter/period). If you're looking at 10 metrics across 5 years you end up calling an LLM 1000 times. This feels unstable.
4. Retry logic adds latency if the answer doesn't exist in the first 5 chunks.

Output Visualisation Example:

Overall Output:

Claude Artifact

Try out Artifacts created by Claude users

Verification Output:

Value Extraction Output:

Commentary Extraction Output:

Next Steps:

1. Set up an evaluation:
   1. Golden Truth Dataset for metric value extraction and definition
   2. LLM as a judge for commentary extraction
2. Hill-climb on output quality with Evals in place:
   1. Improve prompts (big one)
   2. Change models for certain nodes (inserting in o3 mini for commentary extraction, or gemini 2 flash for value extraction)
   3. Play around with changing top k chunks for different source types from 5 for earnings calls and 10q's to 10 for 10ks. This should be guided by the performance of extraction from different source types in the evals.
3. Better define the actual use case for this, this will provide more colour on actual latency constraints, output quality requirements, and that will allow for architecture revamp

Timelines:

• Historic: I've spent probably 4 high quality hours on this (in addition to roughly 8 hours spent on a much simpler v1). The quality of the code is good, no state management issues, and easily interpretable by me.
• Expected: To get this up to a decent serviceable quality this I will probably spend an additional 3-5 hours setting up and creating an initial eval data set. 2 hours prompt engineering. 1-2 hours code engineering and deploying it as a API. This will not get the agent workflow to 100% but it will get it 80%+ of the way there.
• What is stopping me: I don't really have a tangible sense of ROI for this project at the moment which is what is holding me back on executing it to completion and spending the time putting together the eval sets and hill climb. Part of this is a function of not knowing how this will be interfaced by the user, and what role this plays for us in the near term. Another part of it is a sense that even if it is valuable, we don't have a clear idea on how we want to push agents and workflows to production. As such I am shelving additional work on this right now until I have a better sense of those variables and a better defined sense of the scope of this specific project. Also I have a few other competing projects I'd like to work on (Document Summary, Quarterly Summary, Company Initiation Document Creation, etc.) that will surface up use cases for this agent workflow as an input node.
• Why did I start it: KPI / commentary extraction on the fly is inherently valuable and is the bread and butter of many of our tasks and requirements.