Wissen QA Agent

Outline of Project

Core Objective
Build a tool-calling reasoning agent that sits on top of our existing Wissen retrieval API and produces analyst-grade answers (plus citations) to equity-research questions (Answers that are typically less than 1 page of output content).  The same agent will serve two caller types: (1) humans via chat UI (rich Markdown + inline citations) and (2) upstream AI agents via a strict JSON schema.

Agent Reasoning Loop
1 . Breadth pass – single call with default top_k (5) and minimal filters.
2 . Self-reflection & scoring – LLM rates answer completeness (scale 1-10).
3 . Up to 5 refinement calls – adjusts free-text query and/or any metadata filters.
4 . Early stopwhen score ≥ 8 or five iterations exhausted.
5 . If a call returns 0 results → try an alternative query once, else surface an “insufficient evidence” flag in the agent state.

Output Contract
JSON for AI callers:
json { "answer": "Text with {{C1}}, {{C2}}…", "citations": [ { "id": "C1", "doc_label": "NVDA 10-Q Q3 FY-24", "page_no": 17, "page_uid": "2191bfb-…" } ] }
Markdown for humans:
Sentence-level inline markers (¹, ²…) mapped to a footnote block without page_uid.

Orchestration
Implemented in LangGraph:• Nodes: User ↔ Agent, Search Tool, Self-Reflexion, Formatter.• Persistent checkpointing only per live session (no long-term fact store).

Non-Goals / Out-of-Scope (for now)
• Compliance guard-rails & disclaimers (handled elsewhere).
• Token-cost optimisation & hard latency SLAs.
• Long-term knowledge base of equity facts.

Open Items Needing Final Confirmation
- Exact field names / nesting we'd like in the AI JSON
- Md citation style - superscript numbers are assumed
- Target scale for the self-score threshold
- Evaluations

Key Principles

1. Separation of concern: The LLM should reason about what it needs; the schema and tool wrapper should only worry about how to express that to the API.
   1. This means that the Pydantic models and other validations should only validate things that the retrieval endpoint cannot safely reject or auto-correct

Decisions Made for Stage 2 (excl. Reflection)

Key Decision 1: How to engage with the Wissen Search API

1. Wissen Search Space is a multidimensional grid (ticker × fiscal year × quarter × content type × event type). A FilterSet is one tiny cell in that grid. And Wissen Args is the full request the tool receives, it carries the user's natural language query plus one or more FilterSets to explore.
   1. Filter Set:
      1. class FilterSet(BaseModel):
         tickers: Optional[List[str]]
         fiscal_years: Optional[List[int | str]]
         fiscal_quarters: Optional[List[int]]
         content_types: Optional[List[str]]
         event_types: Optional[List[str]]
         content_uids: Optional[List[str]]
      2. Scope of one - describes exactly one logical slice
      3. Light normalisers - e.g. "FY22" -> 2022, quarters validated 1-4
      4. Enum gates - invalid content_types / event types are rejected locally
      5. UID override - if content_uids is present, the other fields are ignored
   2. Wissen Args:
      1. class WissenArgs(BaseModel):
         query: str
         filter_sets: List[FilterSet] = [FilterSet()] # default = open search
         top_k: int = 5
      2. query - the semantic search string from the user
      3. filter_sets - list of FilterSet objects; empty list means no filters
      4. top_k - max pages per query/ combination
   3. Parallel fan out:
      1. If filter_sets has n entires, the tool wrapper fires n API calls (one per FilterSet), merges the distinct pages, and returns a single JSON blob. That lets the agent:
         1. Breadth pass - filter_sets=[{}] => 1 broad call
         2. Refine - append a second or third FilterSet without rewriting the first
         3. Multi-ticker / multi-year queries - supply several FilterSets at once and dedupe the client-side instead of looping synchronously
2. Why this abstraction was made:
   1. Composability - filter_sets is just a Python list, the LLM can .append() a new FilterSet for each refinement iteration
   2. Isolation of validation - All enum and range checks happen inside the Pydantic model before any HTTP request
   3. Parallel breadth - Need ticker-only + ticker year + ticker event in the same iteration? Pass three FilterSets at once
   4. Readable tool schema - LangChain exposes the field descriptions including the allowed enum values to GPT to guide its function calls

Key Decision 2: Enum-validated filters

• Content_types and event_types are validated against the allowed lists before the API calls to document the contract and surface mistakes early
• enum lists are exposed in field descriptions sent with the tool schema which gives GPT the exact vocab, reducing hallucinations with filter names

Key Decision 3: Breadth-pass guard rail

• First tool call is pruned to only tickers + fiscal_years. This guarantees a cheap, wide search before refinement to provide some quick context

Key Decision 4: Page-merge logic inside the tool

• All /new_search responses are deduped by page_uid then returned as raw JSON

Decisions Made for Stage 3 (incl. Reflection)

• Back-up relax rule: fiscal_quarters → content_types → event_types → fiscal_years → tickers
  • We basically drop the most specific filter in the above sequence in-case we have patchy data or in-case the company just doesn't provide that information in that specific category idiosyncratically
  • After relaxing once if zero results are retrieved, the stop-condition that triggers the "insufficient evidence" path is pushed
• After every search call the LLM will pause and self reflect, the output will look like:
  • {"score": 6, "plan": [...]}
  • Where the score is a number from 1 to 10 saying how complete the current answer would be if we stopped now
  • The plan is an instruction for how the agent should improve its next search. The decision has been made for the plan to be a structured output which we put through the FilterSet model and check that every content_types / event_types entry is one of the allowed enums.
• For sourcing we will be using a markdown table for the footnotes in addition to subscript citations in-line in the answer payload. The order of the sourcing is (label . page_no . page_uid) where label = (equity ticker . fiscal period . fiscal year . document type)
• Stopping Criteria:
  • We’ll stop when score ≥ 9 or after 5 refinement passes (breadth + ≤5 more tool calls, within the overall 12-turn ceiling).

Project Overview — Stage 3 Actor-Critic LangGraph Agent with Structured Reflection

1 · Core Objective

Deliver an equity-research reasoning agent that can:

• Answer analyst-grade questions in ≤ 6 LLM iterations.
• Automatically decide when to call the Wissen retrieval API, refine its own queries, and cite primary documents.
• Serve two caller types out of the same loop:
  • Humans → rich Markdown (inline superscripts + footnote block).
  • Up-stream AI agents → strict JSON that passes Pydantic validation.

2 · Stage-3 Enhancements (vs. previous Stage 2)

3 · Reasoning Loop (Actor side)

1. Inject system prompt – Adds a single evergreen SYSTEM_PROMPT describing the JSON contract and scoring rubric.
2. Inject ticker (placeholder) – Hook point for later ticker extraction heuristics.
3. Call LLM – Model sees full conversation + tool schema and emits JSON.
4. Parse reflection – Validates JSON;
   • If score < 9 and a tool_call is present → wrap it in LangChain‐style tool_calls to trigger the Tool node.
   • Else → format final Markdown answer with build_markdown.
5. Call tool – Executes wissen_equity_search once per FilterSet (with retry and client-side dedupe).
6. Increment & route – turns += 1; router decides END vs call_model.

4 · Output Contract

• Markdown (human UI) — superscript numbers¹ mapped to a footnote list generated by build_markdown().

JSON (AI callers)

{
  "answer": "Plain text …",
  "score": 1-10,
  "tool_calls": [ { "name": "wissen_equity_search", "args": {...} } ]?,
  "citations": [ { "id": "C1", "doc_label": "NVDA 10-Q Q3 FY-24", "page_no": 17, "page_uid": "2191bfb-…" } ]?
}

5 · Tool Abstraction — wissen_equity_search

@tool(args_schema=WissenArgs)
def wissen_equity_search(query, filter_sets=[FilterSet()], top_k=5): ...

• FilterSet granularity mirrors a single “slice” of the Wissen grid.
• All enums (content_types, event_types) are validated before any HTTP call.
• Parallel fan-out: one POST per FilterSet, merged & deduped by page_uid.
• Retry once on failure; surface JSON {"error": ...} if both attempts fail.

6 · Graph Orchestration (LangGraph)

  inject_system → inject_ticker → call_model
          ↘︎                         ↘︎
        END ← router ← increment ← call_tool ← parse_reflection

• Entire state lives in a TypedDict {messages, turns, score} — easy to hydrate / persist.
• The compiled graph is exposed via get_wissen_agent_graph() for external import or CLI testing.

7 · Guard-rails & Heuristics

• Enum gates: invalid content_types / event_types rejected locally.
• Fiscal normalisation: "FY22" → 2022; quarter must be 1-4.
• Relax-on-zero (back-up plan): if a highly-specific FilterSet returns zero pages, the LLM is instructed (in the system prompt) to drop the most restrictive filter next round.
• Breadth pruning: On turn 0 only tickers & fiscal_years survive.

8 · Non-Goals / Out of Scope

• Compliance & disclosure banners (handled by hosting layer).
• Long-term fact cache or learning.
• Token-level cost optimisation beyond fixed iteration cap.

9 · Open Items for Confirmation

1. Citation style — current footnote format: “label p.# (UID…)”; any tweaks?
2. Score threshold — staying at ≥ 9 feels conservative; adjust?
3. Ticker-injection node — keep as no-op or implement regex extraction (TICKER_RE exists).
4. Relax-on-zero policy — sequence & aggressiveness of dropping filters.

10 · Key Principles (unchanged)

• Separation of concerns — LLM decides what it needs; tool layer decides how to express it.
• Fail fast — Pydantic validation & enum checks catch bad requests before hitting external APIs.
• Deterministic orchestration — all side-effects happen in dedicated nodes, making the graph observable and unit-testable.

TL;DR
Stage 3 wraps the original breadth-then-refine idea in a disciplined actor-critic loop, adds airtight JSON validation, and pushes all orchestration into a LangGraph state machine. The result is a self-grading, tool-calling agent that stops early when confident—or gracefully concedes when the evidence just isn’t there.

Updated Design Snapshot (v 0.2)

Let me know if any of the above needs tweaking; otherwise I’ll start refactoring the graph to:

• (a) collapse the JSON/Markdown formatter into one Markdown-only node,*
• (b) wire in the self-reflection scorer,*
• (c) add the zero-hit fallback branch.*