Slice and Probe is all you need

Slice:

Simple Definition: A slice represents a strictly defined context within which evidence is sought and questions are answered. It acts a filter or boundary for the research process. The triplet <Subject, Scope, Context Boundary> is key to defining this boundary.

E.g. <KO, Q3 FY2023, Earnings Transcript>

Formal Definition: A slice represents a fundamental, uniquely identifiable, and computationally tractable unit of analysis within the equity research domain. It encapsulates a specific Subject (the entity being studied, typically a company) within a defined Scope (typically a fiscal time period), potentially further refined by a Context Boundary that guides the interpretation and focus of investigation within that scope. Each Slice serves as the bounded container and contextual anchor for a series of discrete investigative questions (Probes).

Operational Definition: Operationally, a Slice is defined by a set of Core Identifiers (Equity and Fiscal Period) that establish its primary boundaries, mapping directly onto the underlying knowledge retrieval system. Associated with these Core Identifiers is a Context Boundary, consisting of metadata that provides nuanced guidance for analysis performed within the scope defined by the Core Identifiers.

Core Identifiers: These components define the primary retrieval boundary of the slice. They are mandatory, immutable for a given Slice instance, and dictate the negotiable filters applied during evidence retrieval. All evidence considered for Probes within a Slice must originate from within this boundary.

• Subject: Ticker, Industry, etc. This is the entity under investigation. There is extensibility for this to become more/less granular but for our purposes right now the subject is the ticker / equity. The subject defines what we are looking at.
• Scope: This defines the extent or dimension along which the Subject is being examined within the Slice. For our purposes fiscal year + quarter is canonical because financial reporting follows this cadence. However it could also be:
  • Time-based (beyond fiscal periods): E.g. DataRange (2024-01-01 to 2024-07--01)
  • Geographic: GeoRegion: EU

Context Boundary: The Context Boundary is a set of descriptive metadata associated with a Slice. Its purpose is to refine the focus for research conducted within the Primary Retrieval Boundary defined by the Core Identifiers. Critically, the Context Boundary does NOT alter the {ticker, fy, fq} filters used for initial evidence retrieval, but provides essential parameters for downstream agents or users.

Purpose:

• Align generated Probes with the specific user intent or required analytical angle (margins, revenue, balance sheet, management)
• Provide the necessary background from the originating user query to guide the retrieval agent in prioritising or filtering evidence

Potential Components:

• data_source_preference: (Type: List[String]; e.g. ["10-Q", "Earnings Transcript"]. This provides a confinement of the retrieval to a list of document types when retrieving or formulating answers. This enables emphasis on primary sources or specific document types. Does not necessarily exclude other sources but strongly guides attention.
• probe_focus (analytical_angles): (Type: List[String]; e.g. ["Margin Analysis", "Volume Growth Drivers"]. These are keywords derived from the user query, intended to steer the Probe Generator toward creating questions covering the most important dimensions.
• user_goal: (Type: String). The original user query that prompted the Slice's creation, providing direct contextual grounding for all downstream agents.

E.g.
slice_object = { 
"slice_core_identifiers": {"ticker": "KO", "fy": 2024, "fq": 4}, 
"context": {
 "user_goal": "...", # The original user query snippet relevant here 
"probe_focus": ["Revenue Growth Drivers", "Margin Analysis"], # Optional hints for Probe Generator 
"data_source_preference": ["10-Q", "Earnings Transcript"], # Guides Researcher's query priority 
"reporting_basis": "as-reported", # Explicit aspect - Default or from query. 
# Potentially: "event_tags": ["Product Launch X"] # If an event happens within the FQ 
},

System Role
The slice serves as the atomic, parallelisable unit of work. Its precise definition ensures:

• Bounded Retrieval: Limits evidence searches, improves efficiency and relevance
• Verifiability: Enables tracing findings back to a defined data subset
• Comparability: Allows meaningful cross-sectional (ticker comparison) and inter-temporal (fy/fq comparison) analysis. Especially as it allows you to hold all else (including probes) equal and change one variable in the core identifiers
• Contextual Relevance: Ensures the analysis pipeline remains aligned with the user's specific and required standards (document types, etc.)

Probe:

Simple Definition: A Probe represents an atomic, directed and specific question formulated to investigate a distinct analytical angle concerning the information contained within exactly one Slice. It serves as the fundamental unit of task execution and within a research workflow, and within our deep equity research workflow it initiates, and drives an adversarial loop between Researcher (Analyst) and Reviewer (PM) agents to generate a verifiable answer based solely on evidence within the parent Slice's defined boundary and context.

Operational Definition: A Probe is instantiated by the Probe Generator for a specific slice. It encapsulates a single, well-defined question and associated metadata that manages its progression through the Researcher-Reviewer loop. Each Probe is tightly coupled to its parent Slice, inheriting its boundaries and context.

Key Attributes:

• probe_id: (Type: String) A unique identifier for the Probe instance. Deterministic and ideally structured to reflect its parent Slice and its specific angle (e.g. {slice_id_string}/{probe_title_slugified}). Mandatory, system-wide unique.
• probe_title: (Type: String) A concise, human-readable label summarising the probe's investigative focus (e.g. "Organic revenue growth YoY"). Used for identification, logic and potentially in synthesised outputs. Mandatory.
• probe_question: (Type: String) The precise, unambiguous and formal question posed to the research agents that will execute the Probe. This question defines the core task: it must be specific enough to guide the Researcher's investigation within the Slice and ideally be answerable using evidence expected within that Slice's boundary and context. Mandatory.
• Status: (Type: String, Enum) An indicator of the Probe's current position int he processing lifecycle. Typical states:
  • pending: Generated, awaiting processing
  • researching: Actively being worked on by a researcher agent
  • reviewing: Draft answer submitted, awaiting reviewer agent assessment
  • revising: Reviewer provided feedback, returned to researcher for modification
  • solved: Reviewer accepted the answer, loop successfully completed
  • insufficient_evidence: Loop completed (e.g. max rounds reached), but the Reviewer determined a satisfactory, evidence-backed answer could not be produced from within the Slice.
  • Mandatory - the state transitions are governed by the adversarial loop logic

Relationship to Slice:

• Strict adherence: A Probe operates exclusively within the confines of its parent Slice. Its associated Researcher agent cannot query data outside the Slice's Primary Retrieval Boundary ({ticker, fy, fq}). We may need to update this at some point to allow reviewers and researchers access to multiple time periods or equities if they want to run a quick inter-temporal or cross-sectional from within the probe. Though for simplicity sake this strict adherence exists for now to allow for entailment checks easily.
• Contextual guidance: The Probe's execution is informed and guided by the parent slice's Context Boundary. But it is permitted to write over or further specify context boundaries to drill into certain document types etc.

Purpose and Design Principles:

• Task Decomposition: To break down the potentially complex user goal associated with a slice into focused, manageable and individually verifiable sub-tasks. The system constraint limits this to a maximum of five Probes per Slice for now.
• Analytical Focus: To direct the research process towards specific, predefined analytical dimensions relevant to the Slice and the overarching user query.
• Orthogonality and coverage: The set of probes generated for a Slice should ideally cover the necessary analytical angles comprehensively and with minimal overlap, ensuring the synthesised answers effectively address the user's requirement for that Slice.
• Workflow Initiation: to serve as the trigger and the central tracking objet for each instance of the research-reviewer loop which is at the core of our research system.

So we have a Slice and a Probe, what does this enable?

The S/P architecture (Slice/Probe) isn't just a novel abstraction but I will argue gives rise to several foundational capabilities.

• Scalability and Parallel Processing:
  • Inherently supports massive parallelism, allowing us to scale test time compute and automate large / complex workflows. Slices by their definition represent logically independent units of work (this is a very important point). The analytical tasks associated with one Slice do not depend on the real-time processing of another. This enables parallelisation. Further, Probes operate strictly within the boundaries of their parent slice, this structural independence allows the system to distribute the workload horizontally. You can process multi-year historical analysis of an entire sector.
• Retrieval Evaluation and Targeted Improvement:
  • A cornerstone of the Slice definition is the Primary Retrieval Boundary (ticker, fy, fq). Evidence retrieval systems (i.e. our Wissen equity search API) are constrained to query only within these strict limits for a given Slice's Probe. This contrasts sharply with approaches that require searches across vast undifferentiated corpora (Multi-Year Single Equity, Multi-Equity Single Year, etc).
  • The immediate consequence of this decision is a significant reduction in the search space, reducing computational overhead and fundamentally reducing the scope dramatically of our retrieval system. All we care about is how high is our accuracy rate among One ticker in One Time Period (compressed down to one quarter). We just need to hill climb on this.
• High quality rigour and verifiability:
  • The S/P architecture embeds rigour and audibility throughout the workflow. Each probe poses a specific analytical query at a bounded evidence set. The researcher searches that specific bounded evidence set to draft answers. Each answer is validated by a reviewer agent. There are citations at a Researcher Draft level, at a Probe Answer level, at a Slice Answer Level, and at a Portfolio of Slices Answer Level. The Portfolio of slices is effectively a research report. And the user can drill into the slice level answers and see evidence and summaries and citations. Can drill further onto a probe level and see the probe answers, the evidence, and summaries and citations. And can then drill further down into an individual researcher-reviewer loop turn level. There is a structured audit trail and we can decide how much to reveal, or how to package it, if to package it. But on our end it exists. Further, the explicit insufficient_evidence status provides a formal pathway to acknowledge epistemic uncertainty when verifiable answers cannot be found within the defined scope. This can become a quick human in the loop refinement of a query or expansion of the Slice.
• Improved consistency and ability to run low bias inter-temporal and cross-sectional analysis:
  • By systematising the research process, S/P enhances consistence and reduces subjective variability (by agents that is). The Slice condition ensures uniform criteria are applied when defining analytical units across companies and periods. The Probe Generator can deploy standardised sets of questions (Probes) across comparable slices, ensuring key analytical dimensions are constantly investigated. In practice this means that a set of Probes (questions around organic revenue growth analysis) can be set to be run across an entire sub-sector via altering the slices ensuring consistent analysis. And can also be set to run historically for one company by the same alteration.
• Modularity and Composability:
  • A Slice has a deterministic identifier and structured outputs (culminating in a Slice Summary which contains verified Probe answers and consolidated citations), which can function as a standardised reusable "knowledge component". This modularity allows the S/P architecture to serve as a foundational layer for many higher level applications. This can be a research report, it can also be automated dashboards, alerting engines monitoring specific metrics, etc.

Applications of the S/P Architecture

• Systematic cross-sectional and inter-temporal analysis:
  • The standardised structure of Slice outputs allows comparative analysis to become trivial to conduct. e.g. comparing key metrics YoY, QoQ, benchmarking a company against peers by running identical Probes across their respective Slices for the same time period.
• Intelligent Monitoring and Anomaly Detection:
  • The ability to consistently generate Slice outputs and Probe answers over time enables sophisticated monitoring applications. By automatically comparing the latest Slice data against historical trends, peer group benchmarks or pre-defined consensus expectations the system can pro-actively identify anomalies, significant deviates, or emergent risks that warrant attention. E.g. A Management Quality Slice runs every quarter with the same probes, and we analyse the sequential delta. One quarter that delta is 2 sigma vs historical norm. That's a flag.
• Foundational for sophisticated research:
  • The verified, structured and citation backed outputs from individual Slices serves as a high quality building blocks for more complex, higher level research involving multiple Slices. The Report Composer agent (in our Equity Research System) leverages this foundation to construct detailed research reports spanning multiple Slices and constructing coherent narratives stringing together disparate data points across industries or time periods.

Evaluation of S/P Architecture:

The fundamental principle guiding the design of the S/P architecture has been the explicit consideration of evaluability. Instead of being treated as an afterthought it has been the seed of the whole process.

Core Metrics simplified version:

Will it fucking lie to me

We can at each stage run entailment tests, Research/Review Turn, Probe, Slice, Portfolio of Slices to rigorously test for deviations from core context.

Is that the full story? Like is it capturing everything?

Given each Probe is confined to a Slice, we can either on our end test for recall by searching wider / longer through Slice. Or if a user wants, we can re-run the Probe through the Slice with more k's to double check something or add more detail. And further we can allow the user to edit the Slice themselves and re-run a probe that led to an answer they consumed.

Software / Agent Evaluation:

• The evaluation occurs at multiple levels:
  • Can you create Slices accurately given a user query
  • Can you create Probes accurately given a Slice and a user query
  • Can you research and validate (using the reviewer) the answers provided for each Probe
  • Is the retrieval system providing the right result to the Researcher in each Probe?
  • Can you synthesise well at a Probe level and then at a Slice Level.
• Each of those can be hill climbed on independently. But more importantly, we can trace the problem in a sub-optimal output down to structured and defined components.
• Further we can test each and every probe collectively across all tasks. Same with Slice, same with Researcher, same with Reviewer, same with Retrieval System, etc.

Explicit Uncertainty Handling: 

The formally defined insufficient_evidence pathway provides a crucial mechanism for the system to declare epistemic uncertainty. This is not merely an error state but a defined outcome that can be evaluated. Test cases can be designed where necessary information is intentionally absent from the Slice's boundary, allowing assessment of the system's ability to correctly identify and report these knowledge gaps, rather than hallucinating or providing unfounded answers. Evaluating the accuracy of this uncertainty reporting is key to building trust in the system's reliability.

Overall Comment on Evaluation

• This is not a complete mapping of the evaluation potential, but a scoping out of how it allows one to now decompose the process

Additional Points

• The S/P architecture extends immensely neatly over to quantitative analysis also - will run that at a future date.
• Can we let user's author their own probes? Can we make specific probes for them?

Critiques

• The current definition strictly prohibits a single Probe from accessing data across multiple Slices, potentially limiting the types of questions that can be directly addressed within the core loop. 
  • Yes, will experiment with this later - currently adds complexity without necessarily proving that the complexity is worth it.
• The effectiveness of the entire S/P system hinges significantly on the ability of the Probe Generator agent to formulate questions that are genuinely atomic, unambiguous, specific, and, crucially, answerable using only the evidence contained within the parent Slice's strict core_identifier boundaries. 
  • Yes, it does. That's part of the trade / hill climbing / evaluation. If it can't ask the right questions we still have ways to improve that i.e. add more context to the Probe Generator, allow the user to evaluate Probes before they're run etc. Even train our own reasoning model to generate high quality Probes.
• What about cross slice reasoning?
  • This is outsourced at the moment to a Slice_Synthesis agent which weaves intra-slice narratives. I heed the criticism that it does not allow for further research upon surfacing the intra-slice insights. This could easily be inserted if deemed necessary.
• Architectural tension between rigid slices and probes to allow for scalability and simplified evaluation, and understanding intra-slice narrative, trends, etc.
  • Noted, is it material enough to underwrite a significantly less scalable and complex to evaluate system? Idk yet.
• Looking at quarters in isolation at the first point of LLM processing may lead to bias. We cannot fetch prior-period comps without creating a new Slice and sticking it together in another step.
  • May be valid, not immediately obvious to me.
  • Mitigant / Check: allow a read only context window of adjacent periods (+/- 1FQ/FY) and have them be flagged distinctly also
• Sometimes you don't need probes, the initial question in the slice can just be run without all this breakdown stuff
  • Well then the probe = 1 in that context, and the probe_generator will need to be refined to not generate unnecessary probes. Maybe we do a coarse "scan" to see if there is an immediate answer to the question?
    • Feels like this is only worth addressing if it really is a problem in practice.
• Need Evaluation Method Specifics:
  • Yes, agreed
• Can you just display the Slice and Probe Summaries, do you need a report?
  • Probably, think things will tend to this tbh
• Can we get automated evals ASAP?
  • Yes, was thinking of running citation-level entailment tests and maybe we build a nightly job to run entailment tests in bulk on a series of questions. To answer the "Will it fucking lie to me?" - if we can pull it up to 95%+ entailment we're a huge set of the way there in terms of headline stats.