On this page

The Hyperdot label

Each Hyperdot version gets a rating that works like the EU energy label on a washing machine. A class from A to G comes from a measurement anyone can repeat. The declared values are published in a product information sheet, and there is a fixed procedure for checking them. A version gets two efficiency classes, one per programme, as an appliance sold for two functions gets a value for each:

A third class, tested, says how much of the kit the self-tests reach.

Labels

Tested class of Hyperdot 0.1.0: A, from what the self-tests reach.

Tool efficiency

Hyperdot label: tool efficiency, Hyperdot 0.1.0, Sonnet 5.5, class F
Hyperdot 0.1.0, Sonnet 5.5, class F
Information sheet · JSON
Hyperdot label: tool efficiency, Hyperdot 0.1.0, Sonnet 5.5 with the plugin, class F
Hyperdot 0.1.0, Sonnet 5.5 with the plugin, class F
Information sheet · JSON
Hyperdot label: tool efficiency, Hyperdot 0.1.0, Haiku 5.5, class D
Hyperdot 0.1.0, Haiku 5.5, class D
Information sheet · JSON
Hyperdot label: tool efficiency, Hyperdot 0.1.0, Haiku 5.5 with the plugin, class G
Hyperdot 0.1.0, Haiku 5.5 with the plugin, class G
Information sheet · JSON

Field test: one owner's game

Not part of the class. The owner ran the field test (-GameSource, token-benchmark.md) on a game of their own, which nobody else can repeat, so these numbers are informative only: they say how the configurations did on one real game, not how Hyperdot rates.

Template Configuration Runs Passed Median tokens Median cost Median time (s)
(not run yet)

benchmark/Get-HyperdotLabel.ps1 computes the label from benchmark runs (-Programme tool or game). benchmark/label-scale.json holds the scale: the classes and minimum requirements, and for each programme its spec, its standard tokens per task and its tolerances. tests/Test-HyperdotLabel.ps1 checks both.

What the EU label taught, and what Hyperdot does with it

The framework is Regulation (EU) 2017/1369. Each product group has its own delegated act; household washing machines use Regulation (EU) 2019/2014. All quotations below are from those two texts.

The EU rule Here
A closed scale "using only letters from A to G ... in seven different colours from dark green to red" (2017/1369, Art. 2(19)) The same seven classes and colours.
The class comes from an index: measured energy over a standard consumption, × 100. For washing machines, EEIW = EW / SCEW × 100, with A ≤ 52, B ≤ 60, C ≤ 69, D ≤ 80, E ≤ 91, F ≤ 102, G above (2019/2014, Annex II, Table 1) The Agent Efficiency Index: tokens per solved task over the scale's standard tokens for that task, × 100. The thresholds are the washing machine's.
Energy is counted per unit of service ("per 100 cycles"), and a separate minimum washing performance must be met first ("greater than 1,03", Regulation (EU) 2019/2023) Energy is counted per solved task: a session whose answer fails still spent its tokens. A configuration must pass at least half its answers, or it gets no class.
A standard programme ("eco 40-60") measured by a harmonised method Each programme's standard programme is profile A-baseline over every task of its spec, three runs each. The spec's SHA-256 is printed on the label.
"No products are expected to fall into energy class A at the moment of the introduction of the label", and a majority should take at least 10 years to reach it (2017/1369, Art. 11(8)) -Introduce sets a new scale's standard tokens from the configurations measured at the time and reports whether class A would start empty.
A rescale is reviewed when 30% of units sold are in class A, or 50% in A and B (Art. 11(6)) The rating reports rescaleReviewDue when 30% of rated configurations are in A, or 50% in A and B.
"The supplier shall not place on the market products that have been designed so that a model's performance is automatically altered in test conditions" (Art. 3(5)) The test cannot be seen. Sessions run in a copy of the repository without benchmark/, this page, the label and their tests. No other file names a benchmark task or profile. The grader is built to resist answers that merely look right.
Verification tolerances "relate only to the verification of the measured parameters by Member State authorities and shall not be used by the supplier as an allowed tolerance to establish the values" (2019/2014, Annex IX) The declared values are the measured ones, computed by the tool from saved answers and never entered by hand. The tolerance is used only by -Verify.
Verification: test one unit; if it is outside the tolerance, test "three additional units"; the model complies if "the arithmetical mean of the determined values" of those three is within it (Annex IX, points 2-6). The energy tolerance is 10% -Verify follows these steps exactly. Each benchmark run of the standard programme is one unit.
A product information sheet, and a QR code linking to the EPREL database (2019/2014, Annex III) docs/label/hyperdot-<version>-<programme>-<model>.json (machine-readable) and .md sheets, with the run ids and the Claude Code version. The label links to them instead of a QR code.
Software updates that change the label's parameters must be declared (Art. 3(4)) Each version is rated anew; a sheet names its version.

The method

A configuration is a model, with or without the plugin loaded. The label rates the configuration under the standard programme: profile A-baseline over all tasks, repeat runs each (three).

Judged results in the game class. On the game programme, a task may also be scored by a pinned judge model against a rubric (a pass is a median of 4 or more out of 5, over three judgings). A blind side-by-side of each model with and without the plugin gives the plugin a win rate, and the plugin configuration's index is divided by 0.75 + 0.5 x win rate: from 1.25 times better when its work is preferred every time to 0.75 when never. The judge and its rubrics belong to the scale, so changing either means a new scale. benchmark/PLAN.md says how the judge is chosen and checked against the owner's own blind ratings.

Two programmes. The tool programme asks questions about Hyperdot's own code with read-only tools. The game programme gives the agent a small Godot game (a copy of examples/gdscript, broken or unfinished on purpose for each task) with Bash, Edit and Write, and grades what it leaves: the files it was told not to touch are unchanged, its own tests catch known bugs planted in a copy (mutants), and the game's tests plus the task's hidden ones pass, all run by the kit's own test. Each programme has its own standard tokens, set at its introduction, so a tool class and a game class are not comparable, just as a washing machine's and a dryer's are not. The plugin is meant for the game programme; on the tool programme it mostly adds the cost of its skill list.

Tokens per solved task for a task are all the tokens its sessions spent, divided by the sessions whose answer passed. "All the tokens" means uncached input, cache writes, cache reads and output, as Claude Code reports them.

Agent Efficiency Index (AEI) = 100 × the geometric mean over tasks of (tokens per solved task ÷ the task's standard tokens). The geometric mean is the usual way to average ratios across benchmark tasks (SPEC CPU does the same), so no single large task dominates.

Grading happens again inside the tool, from each saved answer and the spec's own grader (Test-BenchmarkAnswer). What results.json says about success is ignored, so every run is judged by one rule. The grader:

A spec whose required term already appears in its own prompt is refused, because echoing the question would pass it.

Class AEI Colour
A ≤ 52 dark green
B ≤ 60
C ≤ 69
D ≤ 80 yellow
E ≤ 91
F ≤ 102
G above 102 red

Minimum requirement: at least 50% of the standard programme's answers pass. Below that there is no class, and the label says why.

Tested class: the average of three percentages that tests/Get-KitCoverage.ps1 measures from a self-test run with HYPERDOT_COVERAGE set:

A is 95% or more, B 90%, C 85%, D 75%, E 65%, F 50%, and G below 50%.

The record is a lower bound. A self-test that loads a library file and calls its functions in its own process (the architecture rules, the data checks, the localization parsers) tests them without the kit seeing it. The tested class counts only what a run of hyperdot.ps1 reaches. The jobs that call the reusable workflow do not record either.

Secondary values printed on the label, like water use and programme duration on a washing machine:

The sheet also shows the other profiles, as informative values. They are not rated.

Commands

# Rate the runs in these folders (the benchmark workflow's artifacts) and write the sheets and labels
pwsh benchmark/Get-HyperdotLabel.ps1 -Results run-a,run-b -Coverage coverage.json -OutDir docs/label -Note 'adversarial review: ...'

A sheet's file name uses the model without its version (hyperdot-0.1.0-tool-haiku.json). Two versions of one model rated together, or rated into an -OutDir that already holds the other version's sheet, keep the version in the name. Rated into different folders and copied together afterwards, they would overwrite each other: rate every version of a model into the folder the sheets are published from (-OutDir docs/label), or check the model field of a sheet before copying it over another.


# Check a declared label against new runs of the standard programme (one folder per unit)
pwsh benchmark/Get-HyperdotLabel.ps1 -Results unit1 -Verify docs/label/hyperdot-0.1.0-sonnet.json        # complies / retest
pwsh benchmark/Get-HyperdotLabel.ps1 -Results unit1,unit2,unit3,unit4 -Verify docs/label/hyperdot-0.1.0-sonnet.json

# A new scale: standard tokens from the configurations measured now, whether class A starts empty, suggested tolerances
pwsh benchmark/Get-HyperdotLabel.ps1 -Results runs -Introduce

# What the self-tests reach
$env:HYPERDOT_COVERAGE = "$PWD/coverage"; ./tests/Test-Soak.ps1   # ...every self-test
pwsh tests/Get-KitCoverage.ps1 -From coverage -Json > coverage.json

The benchmark costs money, or plan usage when run with a subscription token. Rating, verifying and introducing cost nothing: they read runs already made.

What it does not measure

Sources