Benchmarks
A measurement contract before the numbers harden.
The benchmark plan separates what can be measured today from what must wait for the 22 nm test chip. Early numbers should be labeled as simulated, modeled, recorded, or projected until silicon produces measured results.
Honeycomb is still before silicon, so this page is not a performance-results page. It is the public measurement plan: the axes that should become results as the implementation matures.
What can start now
Most benchmark families can start before silicon because the project already has executable semantics, generated RTL, golden simulation tests, mesh RTL, OpenLane reports, and analytical models.
| Benchmark family | Can start now? | Current source | Publish label |
|---|---|---|---|
| Inter-token latency and jitter | Yes | RTL cycle counters and deterministic trace simulation | simulated |
| Batch and concurrency sensitivity | Yes | synthetic request traces over kernel and mesh schedules | simulated |
| Decode versus prefill utilization | Yes | analytical model plus GPU baseline constants | modeled |
| Mesh scaling | Yes | generated line/grid RTL, routing proofs, contention models | proved / simulated |
| Deterministic reproducibility | Yes | repeated golden RTL runs and executable-model comparison | recorded |
| Numeric accuracy | Yes | software emulation of Honeycomb arithmetic against FP16/BF16 baselines | modeled |
| Power and energy estimates | Yes | sky130 P&R reports, then VCD/SAIF activity-driven power | recorded estimate |
| 22 nm silicon PPA | No | fabricated test chip only | measured silicon |
The most important near-term shift is from vectorless power to activity-driven power. Once workload traces exist, the site can report energy per operation even before the 22 nm chip returns.
Latency suite
The latency suite should measure serving behavior, not just maximum throughput:
- time to first token;
- time per output token;
- p50, p90, p99, and max latency;
- latency versus batch size;
- latency versus concurrent users;
- long-context decode at several context lengths;
- maximum throughput while keeping p99 TPOT below a stated service target.
This is the suite where Honeycomb should be strongest. The expected shape is a flatter low-batch latency curve than GPU serving, with less jitter because schedules and movement are explicit.
Energy suite
The energy suite should be split by operation and workload:
- pJ/MAC;
- pJ/SRAM read and write;
- pJ/mesh hop;
- pJ/cycle while idle, running a resident kernel, and moving packets;
- joules/token for prefill and decode;
- tokens/sec/W at fixed latency SLO.
The first two passes have started for the compute cell: the sky130 vectorless estimate is now backed by an activity-driven measurement over the routed netlist (idle, control, and kmac traces), reported on the performance page. It already shows the important shape — energy is dominated by keeping resident SRAM clocked, not by the MAC — so pJ/cycle and the power decomposition are recorded, while pJ/mesh-hop waits on routing the mesh and the pure pJ/SRAM-read term is limited by a clock-pin-dominated macro library. The third pass replaces estimates with 22 nm silicon measurements.
Workload matrix
The public workload matrix should include small, medium, and large model classes, with model, context length, quantization, batch, and serving policy fixed in the methodology.
| Class | Purpose | Initial measurement path |
|---|---|---|
| small decoder model | fast iteration and trace validation | software harness plus RTL cycle model |
| Llama-class dense model | buyer-readable inference baseline | modeled Honeycomb placement plus GPU baseline |
| MoE or reasoning model | stress decode, routing, and long-context behavior | placement model and simulator first |
| synthetic resident kernel | isolate datapath, SRAM, and mesh costs | RTL and OpenROAD power |
MLPerf vocabulary is useful even before formal submission. The site should speak in terms of server, offline, and interactive serving modes where those terms map cleanly to the workload.
Accuracy suite
Performance is not credible without accuracy measurements for the numeric path:
- perplexity delta against FP16/BF16;
- downstream task accuracy delta;
- sensitivity to block size and accumulator width;
- overflow-bound coverage for the kernels used in the public results;
- bit-exact reproducibility across runs.
The proof work gives Honeycomb a stronger story than empirical quantization alone, but the website should still show empirical accuracy deltas next to the proof claims.
Scaling suite
Scaling should be measured at each boundary:
- single cell;
- small tile;
- line and grid mesh;
- larger simulated mesh;
- multi-chip projection after 22 nm measurements exist.
The metrics should include hop latency, bisection bandwidth, route contention, weak scaling, strong scaling, and the placement failure rate when cells are disabled as spares.
Methodology rules
Every published number should carry:
- silicon or estimate source: sky130 estimate, 22 nm measured, or production projection;
- model and parameter count;
- quantization and accumulator format;
- context length;
- batch and concurrency;
- clock, voltage, and corner where applicable;
- whether power is vectorless, activity-driven, or measured at the board.
One uncaveated number would make the careful numbers harder to trust. The benchmark pages should keep the label attached to the number until the 22 nm test chip lets us replace estimates with measurements.