# LLM Inference Benchmarks: pilot in progress

**No data is published on this page.**

An earlier version, published 19 October 2026, showed Groq throughput and latency figures from one and two requests, with confidence intervals those requests cannot support. The dataset files behind it also held six more rows whose figures were typed into the generating script as sample values and labelled as measured. Its "80-node grid" claim was unsupported too: the run checked the health of 10 nodes and issued every request from one machine. The dataset downloads (`/data/benchmarks.*`, `/data/manifest.json`) and the JSON feed (`/api/v1/benchmarks/latest.json`) are offline and return `410 Gone`. Please do not rely on any figure from that version.

## Planned method (pilot)

- **Regions:** US West, US East and Germany, each measured from a server in that region.
- **Providers:** only services we call with our own paid API keys, under terms that allow publishing results. A provider whose terms require written consent to publish is left out until we hold that consent.
- **Metrics:** time to first token and streaming throughput. Request counts, run times and per-request records are published with every summary figure. End-to-end timings are reported as measured; we do not subtract handshake time to estimate a chip-only figure.
- **Cadence:** hourly, at two prompt lengths. Failed requests are counted and shown.

Questions: press@modevc.com. See also the [Inference Calculator](https://modevc.com/tools/llm-inference-calculator).