About SITW

Signals in the Wild (SITW) is a benchmark for demand anticipation from public signals. A model — or a human — is given only a target (company, segment, fiscal quarter, cutoff date T) and a point-in-time signal environment, and must mine the relevant signals itself, predict how segment revenue will actually come in versus consensus, cite the evidence it used, and calibrate its confidence.

What it measures

Financial analysis is a forecasting-from-evidence task: good analysts read demand from the outside world and translate it into a revenue call weeks before the company reports. SITW measures exactly that capability, end-to-end, on clean public labels.

  • Mine demand, selling, and exogenous signals — plus peer read-through and substitution.
  • Predict direction and magnitude vs. point-in-time consensus, with a volume / price / FX / M&A decomposition.
  • Cite real, contemporaneous, load-bearing signals.
  • Calibrate — express confidence and abstain when signals are insufficient.

Why revenue, not stock

Stock price is a noisy third-order effect. Revenue is the clean, audited, demand-driven line. We predict it and decompose the move by cause. The same signal→demand engine also powers go-to-market: account prioritization, buying intent, churn risk, and pipeline forecasting.

The causal ladder

The chain SITW tests — and where it deliberately stops.

What "blind", "feed", "C6", "CC-News" mean

Every table on this site labels results by information-access condition. Here's what each one actually gives the model — the full ladder (with ablations) is in the paper.

  • Signal-blind (C0): the model is given only the target — company, quarter, cutoff date — no search, no signals. Isolates prior knowledge / base-rate reasoning.
  • Signal-fed (C2): the model is handed the curated demand-driver signals directly, no mining required. Isolates reasoning quality once mining is solved.
  • Deep-mined (C4): a fixed, non-agentic pipeline — decompose into demand-driver families, issue multi-queries, rerank, summarize, ensemble (Halawi-style fan-out).
  • C6 (recent-SOTA agentic miner): a ReAct-interleaved, RL-agent-style miner with its own search budget that decides what to query next based on what it's already found — closest to "give it Google and a budget."
  • CC-News: a separate, local-only point-in-time news index (Common Crawl News). Deliberately not mixed with live web search — it's a parallel condition compared against the web-search conditions above, not a hybrid of the two.
  • Weekly / live track: a second axis, not a condition — any of the above can be run once at the cutoff, or repeated weekly in the runup to it, to measure how early a model locks in the correct call (lead-time).

A real human/analyst baseline (live Zacks sell-side consensus, 365 of the 371 live-track companies) is on the Live page — click a row to see it alongside each model's call. Not yet on this site: sell-side consensus as its own scored condition (it resolves too recently to score against yet), and an AIA-Forecaster-style supervisor/calibration layer on top of C6 — these exist in the paper's design but aren't run at scale yet.

Two tracks, one backbone

  • Live rolling track (flagship): freeze predictions before each report; resolve mechanically after — contamination is structurally impossible.
  • Frozen historical track: replay past quarters against a date-pinned corpus for fast iteration — run at S&P-500 scale (443 companies, 3 models) on this site.
  • Read-through track: late reporters get peer/customer read-through; first movers get none — a designed difficulty axis.

How it's scored

  • Skill vs. consensus and naive persistence
  • Direction / magnitude / timing as separate axes
  • Driver decomposition (volume vs. price vs. FX)
  • Mining recall, feed-utilization, open-web lift
  • Temporal-availability provenance (retrievable ≤ T)
  • Calibration (Brier / ECE) + selective prediction

Updates

Dated changelog of methodology and infrastructure changes. Full technical rationale for every decision lives in the code repo's docs/DECISIONS.md.

Authors & contact

Yaman Kumar Singla, Balaji Krishnamurthy

Contact: behavior-in-the-wild@googlegroups.com

Paper and code: see the project repository. This site presents S&P-500-scale results (443 companies, 3 models) on the frozen historical track, plus a standing live rolling track (371 companies with an open, not-yet-reported quarter) that resolves mechanically as each company actually reports.

For research and illustration only. Not investment advice.

How to cite

@misc{sitw2026,
  title  = {Signals in the Wild (SITW): A Benchmark for Anticipating Company Demand from Public Signals},
  author = {Singla, Yaman Kumar and Krishnamurthy, Balaji},
  year   = {2026},
  howpublished = {\url{https://behavior-in-the-wild.github.io/signals-in-the-wild/}},
  note   = {behavior-in-the-wild@googlegroups.com}
}