When Python and Rust disagreed on a mailbox tag

How live cross-language interop tests exposed a subtle double-HKDF bug that frozen vectors alone did not catch — and what we learned building Phase 11.

interoprustpythonphase-11engineering

Phase 9 gave us frozen JSON test vectors and a standalone verifier: “does your implementation match the spec bytes?” Phase 11 asked a harder question: can two real clients, written in different languages, pair with each other and deliver a message through an actual relay?

That sounds straightforward until you try it.

Two codebases, one protocol

Yakr has a Python reference stack (where most of the protocol design evolved) and an independent Rust port. Both pass the same primitive vectors — Ed25519 invites, HKDF domains, mailbox-tag HMACs, hybrid ML-KEM transcripts.

Passing vectors in isolation is necessary but not sufficient. Each codebase has its own session layer, storage layout, CLI surface, and assumptions about when derivation happens. The spec defines the bytes on the wire; it does not prevent you from deriving those bytes twice.

The harness: not a shared library

We did not link Python and Rust in one process. The Phase 11 harness treats them like two independent implementers:

  1. One side creates an invite; the other accepts.
  2. Pairing artefacts move as base64 files on disk (invite.b64, request.b64, response.b64).
  3. Each side runs its own interop CLI against a real in-process mailbox relay.
  4. Tests cover both directions: Python inviter → Rust joiner and Rust inviter → Python joiner, then send, fetch, decrypt.

That mirrors how a third party would actually integrate: read the normative spec, implement pairing, exchange files or URLs, talk HTTP to a relay.

CI builds the Rust binary and gates merges on test_phase11_cross_lang.py. When it fails, you get a subprocess stderr trail — not a neat unit-test diff.

The bug: double HKDF on a mailbox secret

The first live cross-language send/fetch failed in a very specific way: encrypt and upload succeeded, but the other language polled the wrong mailbox tag. The relay had an opaque blob; the recipient looked in an empty slot.

Python and Rust both implement the same model:

  1. From the pairwise master_secret, derive a per-direction mailbox secret with HKDF (yakr/v0.1/mailbox-tag + direction string).
  2. From that mailbox secret, compute the mailbox tag with HMAC over direction|epoch.

Python’s MailboxTagDeriver stores the mailbox secret from step 1. Its derive() method only runs step 2.

Rust’s deriver also stored the mailbox secret — but derive() called derive_mailbox_tag(), which ran HKDF again, treating the already-derived secret as if it were still the master secret.

Same function names, same test vector file, same hex expected tag — when you called the low-level API from master_secret. The bug hid in the composition layer used by live sessions.

Frozen mailbox_tag.json did not catch it. Cross-language relay interop did.

The fix was to split “derive mailbox secret from master” from “HMAC tag from mailbox secret” — mailbox_tag_from_secret() — and ensure the deriver only does the second step.

Why vectors alone mislead

This is a pattern we now plan for:

Check What it proves
Frozen positive vector One function, one input → expected output
Negative vector Malformed wire data is rejected
Live cross-lang path Composition, storage, direction, epoch, and relay JSON agree end-to-end

A vector suite proves you can implement pages of the spec. Interop proves you can implement a conversation.

Hybrid post-quantum: the second gap

Classical X25519 interop was the first milestone. Yakr also advertises hybrid PQ pairing (X25519 + ML-KEM-768). Hybrid transcripts had their own vectors; Python and Rust both passed them.

An independent Phase 11 review still asked: have hybrid identities actually paired and messaged across languages in CI? Vector agreement is not the same as inviter/joiner roles, ML-KEM branches, reply traffic, and reloading a persisted contact store after restart.

We extended the harness with hybrid_pq=True in both directions, explicit --classical only where we want the legacy path, plus reply and restart cases. That closed the gap between “PQ bytes match” and “PQ clients interoperate.”

What we took away

  1. Two implementations are a feature, not duplication — they are differential testers for ambiguous spec wording and composition bugs.
  2. File-based artefact exchange in tests approximates real independent development better than importing each other’s libraries.
  3. Phase 11 belongs in CI — the mailbox-tag bug slipped through vector-only gates and appeared immediately under cross-lang send/fetch.
  4. Post-quantum needs live paths, not just KEM vectors — hybrid adds branches in invite, pairing, and session state that only show up in full workflows.

If you are implementing Yakr without importing our libraries, start with frozen test vectors and the interop checklist on GitHub. When you think you are done, try talking to the other reference stack. That is where the spec becomes real.

For the visual walkthrough of pairing and delivery, see our protocol guide.

← All posts