Sign in Subscribe

Wissen's Equity Data Provision

TLDR:

Between S&P and Quartr, 90% of all documents LS Equity Analysts need that are publicly accessible are available and provided within minutes of an event with an incredibly high level of accuracy.

What neither of them provide is the ability to make that data LLM ready.

As a Hedge Fund, you either already are or will be in the near future looking to build / use AI in your process.

What you need before you can even think of the term "Agent" is:

  1. All of the data in PDF format
  2. Data Ingestion Pipeline and Expertise in designing and choosing the right configurations for the data ingestion
    1. Further an ability to ingest data rapidly upon release
  3. A data ontology that facilitates retrieval
  4. A retrieval system that allows you to conduct cross sectional and inter-temporal analysis with sufficiently high recall and precision rates to render downstream retrieval use cases positive value add

Analysis of Wissen as a Data Sourcing Company

Why Wissen got involved with Data Sourcing

We began sourcing our own data under the assumption that there would be no other "cost effective" method of experimentation. Hence we focussed on onboarding historical data to unblock experimentation with our retrieval system and our equity research agents, whilst also laying down the groundwork to be able to scrape this data on an ongoing basis to provide it to our users

    1. This thought process was further validated by the cost of data provision to an enterprise from S&P (unless the fund already has a contract with S&P) at around $20-50k. We didn't investigate thoroughly the provision costs from Quartr. But in the paradigm of Wissen charging $10k per user per year as a research platform, the cost of data alone was going to 2x our headline cost to the user, as such potentially pricing out customers unless we provided the data + platform at $10k. In retrospect this was a misguided belief for several reasons which we will address.

Updating those priors:

  1. We have a cost effective method of experimentation, S&P is willing to cut us up to 6 months of unlimited access to ALL of their data
  2. Quartr provides the vast majority of documents one would need from a company's IR website, but at very low provision cost. $15-20k for provision.
  3. We can charge a hell of a lot more than $10k per user for our service. Using Winnowing Labs as an example, small funds are willing to pay $30/month for some automated workflows and foot the entire data cost as pass through. Rendering our months of toiling in data sourcing as a superfluous exercise to attract the early adopters. We should not be targeting those who are price sensitive by $25k at an enterprise level, they aren't our target market.

Where are we at the moment

After around 2 months of having 2 data scrapers out of Pakistan, we have backfilled data from scraping 3rd party sites and have some backfill from scraping IR websites in Europe directly.

We only have historical data and no real set-up to get the data on an ongoing basis - "live". The historical data we have itself is patchy and incomplete. It is approximating the data one would get if they signed a contract with Quartr, which our competitors have done. Further, practically free provision of Quartr level data has been done by fin tool, finchat, and recently Perplexity.

This became a huge time cost in reviewing the scraping team's outputs, with each equity's historical data onboarding costing cognitive time and resources.

Where we could go if we continued trying to source and provide data

Something to outline here is there is incredibly limited value in only having historical data when it comes to servicing customers. The key reason for this is that in financial services, data becomes outdated very quickly, and users demand up-to-date data. The vast majority of research requests are processing new information, using historical context. If your data has a cut-off point in May 2025, but the user is using it in August 2025, there is a strictly limited utility to the service you're providing. A lot can change.

Therefore any consideration of Wissen as a data company must focus on being able to provide up-to-date information - in other words "live". This means that in order to continue providing data we must consider the following challenges:

  1. Always on Discovery: A real-time feed requires infrastructure that polls thousands of IR sites, exchange feeds, RSS endpoints every few minutes, plus change-detection logic so we don't ingest duplicates
  2. Latency targets: HFs expect new 8-ks, earnings presentations, etc. within <5 minutes of release. Meeting that SLA demands a robust systems infrastructure with very low fail rates.
  3. Quality Assurance at scale: "It downloaded" is not equal to "It is usable". We would need automated checks for completeness and usability. In addition to ensuring that any system/infrastructure errors on our side are able to be addressed irrespective of timing of event - Swedish Earnings get released at 5am GMT, US earnings can be released 10pm GMT. We would need a round the clock team to address and handle any issues.
  4. Ops headcount and management: Though we have a lot of AI based scrapers evolving in the eco-system and improvements in surrounding technology, I'd assume even a SOTA scraper has a 1-3% error rate if we're being optimistic (broken links, new CAPTCHA security protocols, format drift, scraper error...). Someone has to triage those tickets 24/5, and someone needs to manage those who triage those tickets.
  5. Things change all the time: Reverse mergers, spin-offs, ADRs, etc. Each capital markets action spawns new tickers and legacy filing trees. Tracking parent/child relationships is non-trivial and re-mapping company - Wissen taxonomies requires some domain expertise.

Those real-time challenges come in addition to the following challenges of ontology, taxonomy, and schema management.

  1. Creating a comprehensive ontology of all financial filings and data for multiple geographies (done)
  2. Mapping a company level taxonomy onto our global taxonomy - rather difficult to do. Especially for the more niche document types that aren't regulatory, each company names them differently. This does require some domain expertise to get right, and also an LLM in the loop which can post scrape classify documents.
    1. Version the ontology itself, regulators may add new forms, companies may change their filing types, etc
    2. I've tried several different agents (deep research, gemini deep research, Claude deep research, o3, Manus) and built some incredibly good prompts (available upon request) and we can get roughly 70-75% there taxonomy mapping. Tough to get 100%
  3. Having validation checks on a company by company level for cadence and periodicity of document types. E.g. GIVN files a trading update for Q1 but files an earnings release for 9M, but only provides earnings presentations for H1 and FY earnings.
    1. Again I've got this to 70/75% there hill climbing on prompts using o3 on ChatGPT, but tough to get 100% with confidence

Before we get ahead of ourselves...

Any ambitious project will throw off many issues and problems when held against the scrutiny of theoretical assessment. We have however moved away from the theory and you personally have spent a few weeks on the scraping problem full time, and a few months with oversight.

Though there are new evolving technologies that make things possible in the data scraping that weren't before:

  • LLM-generated scraping code, and self healing scrapers:
    • Reworkd points an LLM at a target website, autogenerates extractor code in its in-house SDK (Haramble LOL), validates the output and re-deploys when the selector breaks, turning manual refactors into an autonomous loop
  • Reasoning enabled crawling:
    • Fire-1 by Firecrawl controls a headless browser, clicking through cookie banners, paginates tables and wait for React graphs to hydrate before extraction. Capabilities that traditional request-based crawlers missed

And I'm sure there are plenty others. In my early trials using Reworkd after hill climbing on the extraction ontology and descriptions, I've managed to produce a pretty damn good extraction of GIVN (again like 75% there).

There is scope for us to become early adopters of rapidly evolving technology, and leapfrog the data providers in our industry by providing a more complete data offering, with a better dev / user experience, at a lower cost. Let's not forget, even if we cast our time-line out till the end of 2026, the TAM for Public Equity Data >>> TAM for LLM based productivity enhancement tools.

Judgement

I believe getting scrapers up and running to provide a "decent" level of real-time coverage across 50 equities is something that is possible with our current set up. Where both you and I can spend 2 hours between us daily on data sourcing, checking that taxonomies are correctly mapped, manually overseeing the data validation checks, pinging the Syed 1 and Syed 2 with any gaps, and assuming 2% of companies require a schema remapping every month, I can spare 3 hours to conduct it personally.

To provide an SLA level data provision across even 5 sectors in the US (350+ equities) and then if we want a client like DE Shaw or Millennium we can see the minimum coverage requirement being at least 1000 equities globally, we suddenly are playing a completely different game.

Not to consider the fact that if we close a client like DE Shaw or Millennium, they can MORE than afford to pay for the data from S&P and Quartr and odds are they already have facilitator agreements in place. Meaning that we'd be doing a hugely up-hill battle to provide a platonic "data-completeness" when people don't even have the basic earnings presentations in place yet.

And as for the clients that can't afford to pay for S&P, well, they probably don't have exactingly high standards requiring sharp data completeness in the first place.

Opportunity Cost of every hour spent on data sourcing:

  1. Retrieval optimisation and building to production grade - there is soo much to be done on the retrieval side still to get us to the point where we can provide retrieval across 500 equities (which I'm taking as the lower bound of coverage for a DE SHAW size contract)
  2. Ontology Design and Mapping: Even just working with S&P's APIs for documents etc. a lot of alpha is in actually mapping the S&P taxonomy to one that the user and our agents can understand. E.g. they don't provide any actual comprehensive mapping between US and Europe, or even within Europe.
  3. Alternative Data Sourcing: We can start doing multi-modal data sourcing of videos, podcasts, etc. There's also other alt data points like government websites, etc. There's also ad - hoc scraping jobs people would happily pay for. These are all Greenfield opportunities with no competition and potentially huge upside for our users.
  4. Agentic Workflows: At the end of the day, my high conviction belief is that over the next 3 years getting retrieval right, and building agents that can conduct high quality financial research and analysis, is the HUGE opportunity in our space and NO ONE has got this right yet. We could be spending even more time here, which clips us the $300k consultancy contract which may end up birthing an incredible Palantir for Finance business model if we get it right.

Next Steps and Action Points

Choose one of the following 3 options:

  1. Wissen as a Full Data Provider (build global live feed)
    1. Up-front Cash: $$$ (in team + infra)
    2. Ongoing CapEx: High
    3. Speed to market: Slow (>9 months?)
    4. Differentiation: Low (S&P and Quartr and BBG and AlphaSense already provide most if not all of it and more)
    5. Key Risks: Scaling Pain, competitor like Quartr increases coverage and undercuts on price
    6. Potential Upside: Become a very high margin competitive threat to S&P, prime for acquisition
  2. Wissen as a Facilitator (S&P/Quartr)
    1. Up-front Cash: 0.5$ (S&P free for 6 months)
    2. Ongoing Capex: Low
    3. Speed to market: 4-8 weeks
    4. Differentiation: High - we focus on the Retrieval and Agents layer
    5. Key Risks: Vendor lock-in
    6. Potential Upside: Become the necessary infrastructure layer for the AI era of Finance, becoming a strategic asset in the industry ripe for acquisition. Further being able to provide the high ticket consultancy services to clients and focus on Greenfield opportunities in Agents and RLFT for specific industries and tasks.
  3. Hybrid
    1. Up-front Cash: $
    2. Ongoing Capex: Medium (cost of having a scraping team and managing them, albeit smaller than option 1)
    3. Speed to market: 8-12 weeks realistically
    4. Differentiation: Ultra-high - Niche data coverage + Retrieval and Agents Layer to drive home the edge
    5. Key Risks: We still need the scraper team and the headache of scaling the system to be real-time across potentially 500-1000 equities
    6. Potential Upside: Become an even more strategic acquisition target, and use prop data as a wedge to get people into the more lucrative retrieval and agentic offering we will have

And feel free to suggest one of your own, or critique and evaluate the ones I have put up.

But my general view, and I will write this up separately in another strategy document, is that our EDGE is in making sure we can get to market quickly with a retrieval offering that is better than others and agents and reports that are still better than Deep Research and alternatives - and close the giant whales. The space is evolving rapidly, there are only a few whales, and we shouldn't have any moment of distraction of data sourcing etc.

Now I do see a lot of benefits of becoming a data providing company aswell, and recent tech innovations have opened up some incredibly lucrative business opportunities of compiling and structuring datasets that just weren't possible before. My view here is I want to hire someone later down the road who manages and runs the whole process of sourcing, designing the entire systems infrastructure, and taking on the risk of hitting SLAs with the full time dedication it needs and maximally leveraging all the new AI adjacent technologies when it comes to scraping. And we can sell the data before scraping it effectively by leveraging relationships we already have from our Retrieval + Agents customers. This person can be paid a lot of money for creating even more in business value. It just isn't mission critical for us right now and I see a lot of potential volatility.