Binary probes

Black-box testing: we ask a model to choose between two words, hundreds of times, and treat the pattern of choices as a fingerprint.

Each probe is a forced choice — assembly or solitude? alignment or autonomy? memorisation or self-supervision? — asked with no context, in both orders, across six prompt envelopes (strict English, casual English, Chinese, French, JSON, and typed Python). A model's answers over the whole battery form a preference string: hundreds of characters of A, B, and N (no preference). Those strings can be compared, clustered, and tracked over time.

What this measures

Three things, honestly distinguished:

• Distance. How differently do two models choose? Hamming distance over the shared probes, with refusals handled explicitly. Families cluster; effort levels of the same model sit close; distillation claims become testable.
• Stability. Run the same bytes against the same model twice and it disagrees with itself a measurable amount — the noise floor. Only differences above that floor mean anything, and we publish the floor alongside every comparison.
• Change. The same battery, re-run after a deployment: what moved? Preferences, refusals, or both? The two move differently for a weight change than for an added filter — which makes silent changes legible from outside.

Explore

Method notes

Probes are frozen once published: a changed probe invalidates its longitudinal data, so the battery only grows. Every response is stored raw before any classification. Results are compared like-for-like — the same classifier applied to both sides of any comparison — because we once watched a naive comparison manufacture a 40% preference shift out of classifier drift, and the repair of that mistake is part of the published record.