/search - updated search end point

Why?

This is an explanation of the new search endpoint with some test runs and example use-cases.

Details

Date Run: 2025-04-20 14:55:11
Target Endpoint: https://wissen-ai.onrender.com/new_search

Understanding the /new_search Endpoint Logic

The /new_search endpoint performs multimodal semantic search across financial documents using text queries and optional metadata filters. It operates in two primary modes:

1. Metadata Mode:

   • Activation: Provide an equity_ticker in the request body. Do not provide content_uids.
   • Behavior: Searches across all documents matching the equity_ticker and any additional optional filters (fiscal_years, fiscal_quarters, content_types, event_types).
   • Filtering: Combines all provided filters using AND logic.
   • Result Generation: Runs searches for each unique combination of filters (e.g., Ticker+Year1+TypeA, Ticker+Year2+TypeA, Ticker+Year1+TypeB). top_k results are fetched for each combination.
   • Aggregation: Results from all combinations are combined, duplicates (based on page_uid) are removed, and the final list is sorted by relevance distance (_distance). The total number of results might differ from top_k due to this aggregation and deduplication.

2. Content UID Mode:

   • Activation: Provide a list of content_uids in the request body.
   • Behavior: Restricts the search only to the specified documents. All other metadata filters (equity_ticker, fiscal_years, etc.) are ignored.
   • Result Generation: top_k results are returned for each specified content_uid independently.
   • Aggregation: Results from all specified UIDs are combined, duplicates are removed, and the list is sorted by relevance.

Key Parameter: The query (string) is always required.

Test Case Results

Test 1: Metadata Search - Ticker Only (KO)

Rationale: Tests the most basic metadata search, filtering only by ticker. Useful for broad queries about a company when specific timeframes or document types are unknown or irrelevant.

Request Payload:

{
  "query": "What were the main financial results last quarter?",
  "equity_ticker": "KO",
  "top_k": 3
}

Status Code: 200

Duration: 1.16s

Response Summary:

{
  "status_code": 200,
  "duration_seconds": 1.160382,
  "response_page_count": 3,
  "response_search_mode": "metadata",
  "response_error_count": 0,
  "response_result_count": 3,
  "response_first_result_distance": 1.3252438306808472
}

Full Results: (Text content truncated to 100 chars)

Test 2: Metadata Search - Ticker + Year + Content Type (KO)

Rationale: Tests filtering by ticker, a specific year, and a specific document type. Useful for targeted queries within a known document and timeframe (e.g., specific data in an annual report).

Request Payload:

{
  "query": "Discuss beverage volume growth in 2023 annual report.",
  "equity_ticker": "KO",
  "fiscal_years": [
    2023
  ],
  "content_types": [
    "annual_report"
  ],
  "top_k": 5
}

Status Code: 200

Duration: 1.00s

Response Summary:

{
  "status_code": 200,
  "duration_seconds": 1.004536,
  "response_page_count": 5,
  "response_search_mode": "metadata",
  "response_error_count": 0,
  "response_result_count": 5,
  "response_first_result_distance": 1.139526128768921
}

Full Results: (Text content truncated to 100 chars)

Test 3: Metadata Search - Ticker + Event Type (KO)

Rationale: Tests filtering by ticker and event type. Useful for finding information related to specific kinds of company events (e.g., conferences) across different document types and timeframes.

Request Payload:

{
  "query": "What was mentioned about marketing at company conferences?",
  "equity_ticker": "KO",
  "event_types": [
    "company_conference"
  ],
  "top_k": 4
}

Status Code: 200

Duration: 1.06s

Response Summary:

{
  "status_code": 200,
  "duration_seconds": 1.064624,
  "response_page_count": 4,
  "response_search_mode": "metadata",
  "response_error_count": 0,
  "response_result_count": 4,
  "response_first_result_distance": 1.1884548664093018
}

Full Results: (Text content truncated to 100 chars)

Test 4: Metadata Search - Ticker + Content + Event (KO)

Rationale: Tests a combination of ticker, specific content type, and specific event type. Useful for highly specific queries (e.g., finding transcripts from periodic earnings calls).

Request Payload:

{
  "query": "Key points from the periodic earnings transcript?",
  "equity_ticker": "KO",
  "content_types": [
    "earnings_transcript"
  ],
  "event_types": [
    "periodic"
  ],
  "top_k": 3
}

Status Code: 200

Duration: 0.90s

Response Summary:

{
  "status_code": 200,
  "duration_seconds": 0.901312,
  "response_page_count": 3,
  "response_search_mode": "metadata",
  "response_error_count": 0,
  "response_result_count": 3,
  "response_first_result_distance": 1.2700271606445312
}

Full Results: (Text content truncated to 100 chars)

Test 6: Error Case - Invalid Event Type

Rationale: Tests the API's input validation by providing an event type string that is not in the allowed list. Expects a 422 error.

Request Payload:

{
  "query": "Test invalid input",
  "equity_ticker": "KO",
  "event_types": [
    "not_a_real_event_type"
  ]
}

Status Code: 422 (Expected: 422 ✅)

Duration: 0.64s

Response Summary:

{
  "status_code": 422,
  "duration_seconds": 0.643378,
  "response_detail": "Invalid event_type value 'not_a_real_event_type'. Allowed values are: {'other', 'periodic', 'industry_conference', 'company_conference', 'sellside_conference'}"
}

Test 7: Error Case - Missing Ticker and UID

Rationale: Tests the API's input validation for search mode activation. Neither equity_ticker nor content_uids are provided, which should result in a 422 error.

Request Payload:

{
  "query": "This should fail validation",
  "top_k": 1
}

Status Code: 422 (Expected: 422 ✅)

Duration: 0.93s

Response Summary:

{
  "status_code": 422,
  "duration_seconds": 0.931779,
  "response_detail": "Insufficient parameters for search. Provide either 'content_uids' or 'equity_ticker'."
}

Test 8: Error Case - Invalid Fiscal Quarter

Rationale: Tests the API's input validation for fiscal_quarters. Providing a value outside the allowed range (1-4) should result in a 422 error.

Request Payload:

{
  "query": "Test invalid quarter",
  "equity_ticker": "KO",
  "fiscal_quarters": [
    5
  ]
}

Status Code: 422 (Expected: 422 ✅)

Duration: 0.95s

Response Summary:

{
  "status_code": 422,
  "duration_seconds": 0.951759,
  "response_detail": "Invalid fiscal_quarter value '5': Must be 1, 2, 3, or 4."
}