code-daemon-relation-v1

A 117M-parameter relation classifier. Mark two entities inside a passage and it answers, in one forward pass, how they relate — one of three relation types, or no relation.

It does a job usually handed to a large generative model — read a passage, extract typed relations between the things it mentions — as a single classification instead of token-by-token generation. That makes it cheap enough to sweep an entire corpus: ~2 900 pairs/sec on a laptop RTX 5060.

logits = session.run(None, {"input_ids": ids, "attention_mask": mask})[0]   # [B, 4]

Text is fed as an (empty query, marked passage) pair. Multilingual — the XLM-R backbone reads prose and code comments in many languages.

---

1. The four classes

Wrap each entity in [E1]…[/E1] and [E2]…[/E2] inside its natural context. Take the argmax; class 0 is an explicit abstain, and a softmax threshold drops the rest of the low-confidence tail.

idx	label	meaning
0	NO_RELATION	co-occur, unrelated — abstain
1	semantically_similar_to	same purpose or same goal, other mechanism
2	supersedes_or_conflicts	one supersedes, replaces or contradicts
3	depends_on	one requires or configures the other

The taxonomy is deliberately coarse. An earlier 8-way version split these into near-synonym pairs (semantically_similar_to vs shares_purpose_with, replaced_by vs contradicts, depends_on vs configured_by) and the distinctions were not reliably separable from context — the classifier spent its capacity on boundaries that downstream consumers then collapsed anyway. Merging them into three positives plus abstain is what the model is actually good at.

Decision rule as shipped: argmax != NO_RELATION and 1 - softmax[NO_RELATION] >= tau. Gating on the probability that any relation exists rather than on the winning class's own probability is more robust: when a real relation's mass spreads across two plausible classes, the per-class maximum sags while "something is here" stays high.

---

2. Architecture

• Warm-start — cross-encoder/mmarco-mMiniLMv2-L12-H384-v1.
• Encoder — XLM-RoBERTa, 12 layers / 384 hidden / 12 heads, FFN 1536. ~117M parameters, of which 96M is the multilingual embedding table.
• Vocabulary — 250 006 pieces = 250 002 XLM-R + 4 markers [E1] [/E1] [E2] [/E2] (ids 250002–250005).
• I/O — input_ids, attention_mask (no token_type_ids) → logits[batch, 4]. Sequence 256 on the shipped engines; 64 / 128 also provided.

Entity-marker pooling, not [CLS]

The classification head does not read the [CLS] vector. It mean-pools the hidden states at the entity-start markers — the [E1] and [E2] positions — concatenates the two, and passes that through a single linear layer.

This matters for a relation task. A [CLS] vector summarises the whole passage, so the head has to recover which two things the question is about from a global summary. Reading the marker positions instead gives the head both arguments directly and in order, so the relation is scored between the two entities rather than inferred from the sentence as a whole. Direction comes free: swap the markers and the input genuinely changes.

---

3. How it was made

Warm-started from a strong multilingual ranking cross-encoder, with its single ranking logit replaced by the 4-class marker-pooling head, then fine-tuned by sequence-level distillation on relation tuples, each grounded in the passage it was drawn from.

Two consequences are visible at inference and worth knowing. NO_RELATION is trained, not inferred — abstain is a class the model was shown explicitly, which is why the tau gate below behaves sensibly instead of firing on every co-occurrence. And the passage is expected to arrive windowed so that both markers survive truncation: a pair whose second entity falls outside the sequence budget is not a hard case, it is an unanswerable one.

---

4. Speed

Measured on one laptop: Intel Core Ultra 9 275HX / NVIDIA RTX 5060 Laptop.

All lanes at batch 16 × seq 256:

lane	pairs/s	per batch
TensorRT FP16, RTX 5060	2 942	5.44 ms
OpenVINO FP16, iGPU	111	143 ms
OpenVINO FP16, CPU	44	366 ms
ONNX Runtime FP32, CPU	42	380 ms

OpenVINO rows were measured on 2026.3; the IRs in this repository are built for 2026.4 (re-timed on the embedding model: within a few percent either way).

Per bucket, OpenVINO 2026.3 on the same laptop (pairs/s, solo):

bucket	batch × seq	CPU FP16	iGPU FP16
s	16 × 64	159	508
m	16 × 128	85	245
l	16 × 256	44	111

The integrated GPU is ~2.6× the CPU on every bucket, so on a host without a discrete card the iGPU lane is the one to route relation extraction to. Both devices at once give ~87 % of the sum of their solo rates — they share one memory controller.

The GPU lane is ~67× the CPU lane, which is the point: relation extraction over a corpus means tens of thousands of candidate pairs, and only the compiled-engine path makes that a background task rather than a batch job.

Three length buckets ship — seq 64 / 128 / 256 at batch 16. Attention is quadratic in sequence length, so routing short passages to a short engine is worth taking when your pairs vary in length. Padding is attention-masked, so a pair produces the same logits from any bucket that fits it.

At corpus scale

The table above is a per-batch micro-benchmark. Sweeping a real corpus batches far wider, which changes what the bottleneck is:

at corpus scale	measured
GPU inference, 267 pairs/call at seq 256	5.87 ms → ~45 000 pairs/s
End-to-end incl. tokenisation, 12 threads	~2 500 pairs/s
Relations written, 3 043 chunks	15 175 in 15.3 s → 994/s
Relations written, 4 975 chunks	31 696 in 45.0 s → 704/s

Once the engine is batched this wide the GPU is no longer the limit — tokenisation is, and it is worth giving it real thread count. Measured padding waste at these batch sizes is 18–20%, which is what the length buckets are there to keep down.

---

Inside a real index

Measured live in the UltraCode daemon over two repositories, TensorRT FP16, wide batches of ~263 pairs per call:

metric	value
pairs scored	2 162 / s sustained (210 071 in 97.2 s)
inference share of wall	90 %
padding wasted	33 %
relations created	1 011 – 1 345 / s

The live rate is below the 2 942 pairs/s micro-benchmark above because a third of every wide batch is padding. Relations created run lower still: most scored pairs fall under the acceptance threshold and are rejected, which is the point of the threshold.

---

5. Standalone use

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

LABELS = ["NO_RELATION", "semantically_similar_to", "supersedes_or_conflicts", "depends_on"]

tok  = AutoTokenizer.from_pretrained(".")        # includes the [E1]/[E2] marker tokens
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

def classify(marked_text, max_len=256, tau=0.7):
    enc = tok([""], [marked_text], padding="max_length", truncation=True,
              max_length=max_len, return_tensors="np", return_token_type_ids=False)
    logits = sess.run(None, {"input_ids":      enc["input_ids"].astype(np.int64),
                             "attention_mask": enc["attention_mask"].astype(np.int64)})[0][0]
    p = np.exp(logits - logits.max()); p /= p.sum()
    if 1.0 - p[0] < tau:                          # gate on "any relation at all"
        return "NO_RELATION", float(p[0])
    i = int(p.argmax())
    return (LABELS[i], float(p[i])) if i else ("NO_RELATION", float(p[0]))

classify("The [E1]FAISS[/E1] index was replaced by the [E2]native IVF[/E2] backend.")
# -> ('supersedes_or_conflicts', 0.7x)

Both entities must appear inside one passage, marked in place. The model reads context, so a bare pair of names with no surrounding text carries little signal.

---

6. Evaluation

Dev macro-F1 = 0.547 over the four classes, on a 691-row held-out split of the distillation set.

Read that as what it is. The classes are intrinsically imbalanced — a teacher describing documentation emits "similar" far more often than "depends on" — and the merged taxonomy still contains genuinely ambiguous boundaries that human annotators would also disagree on. The abstain class plus the tau gate exist because the useful operating point is high-precision edges, not maximum recall: for building a graph, the real test is spot-checking the edges it emits at your chosen threshold.

Against zero-shot NLI — the alternative to training a head

With no training data for these classes, the textbook approach is a zero-shot NLI model: state each class as a hypothesis and take the entailment score. MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 is the standard pick. Same 691 rows, same 4-way mapping, same machine.

What matches and what does not. There is no public model trained on these four merged classes, so the comparator matches on task and dev set but not on size (279 M against 117 M) or runtime (PyTorch against onnxruntime) — and zero-shot pays one forward pass per class by construction, which the speed figure below makes visible:

model	params	macro-F1	accuracy
this model	117 M	0.569	0.670
mDeBERTa-v3-xnli, zero-shot	279 M	0.194	0.295
always NO_RELATION	—	0.158	0.463

On the question the daemon actually asks — is there a relation at all — the gap is narrower on F1 and wider in kind:

model	has-rel F1	P	R
this model	0.810	0.793	0.828
mDeBERTa-v3-xnli	0.699	0.538	0.997

A recall of 0.997 at a precision of 0.538 is the whole story: asked whether two entities in the same paragraph are related, an NLI model says yes almost every time. For a graph builder that reads the score as an edge weight, that is not a usable signal.

Speed, same hardware, same 691 pairs: 351 pairs/s for this model (onnxruntime CUDA, FP32, batch 16 × seq 256) against 61 pairs/s for the zero-shot baseline (PyTorch CUDA). The 5.8× is structural, not an implementation detail — zero-shot needs one forward pass per class, so a 4-class taxonomy costs four passes per pair where this model costs one. The shipped TensorRT FP16 engine is faster again (2 942 pairs/s, above).

Measured 2026-09-11; harness and raw JSON ship in the UltraCode repo (models/_distill_shared/bench_vs_generic.py).

Suited to

• Turning prose or documentation into a typed concept graph.
• Any sweep where a large LLM per pair would be too slow or too expensive.
• Multilingual corpora, including code comments.

Not suited to

• Fine-grained relation ontologies — this is 3 positives plus abstain by design.
• Entity extraction: it classifies pairs you already found, it does not find them.
• Passages where the two entities are far apart — the marked window is 256 tokens.

---

7. What is in this repo

Compiled engines, named per runtime × OS × GPU arch, plus the ONNX for standalone use.

• TensorRT FP16 — code-daemon-relation-v1-{s,m,l}_{win_x64,linux_x64}_trt11.0_sm_{75,80,86,89,120}.engine, plus code-daemon-relation-v1-{s,m,l}_linux_x64_trt11.0_sm_90.engine (H100 / H200, Linux only) (buckets seq 64 / 128 / 256, batch 16).
• OpenVINO FP16 — code-daemon-relation-v1-{s,m,l}_ov2026.4_{cpu,igpu}_fp16_b16_s{64,128,256}.{xml,bin}.
• Tokenizer — tokenizer.json, sentencepiece.bpe.model, tokenizer_config.json (XLM-R SentencePiece with the four marker tokens added).
• ONNX — model.onnx (+ model.onnx.data), FP32, the build source for every engine above.
• Raw weights — model.safetensors + config.json, the same FP32 weights under transformers names: the encoder as roberta.*, the head as classifier.weight [4, 768] + classifier.bias. ⚠ The head is NOT the stock one — AutoModelForSequenceClassification would leave its own classifier.dense / classifier.out_proj randomly initialised and ignore ours. Load the encoder with AutoModel, mean-pool the last hidden state over the tokens equal to each marker id (config.json → ultracode.entity_marker_ids, 250002 and 250004), concatenate the two and apply the classifier. Done that way it matches the ONNX to 2e-6. The Apple (MLX) build is prepared from this pair: the UltraCode MLX runtime implements this entity-pool head, with the marker ids carried in the prepared file's header.

FP16 rather than INT8: this architecture's activation outliers make per-tensor INT8 calibration lossy, and FP16 costs nothing on any GPU that can run it.

---

Apple Neural Engine (Core ML)

coreml_ane/embed.mlpackage/ is a Core ML multifunction package that runs this relation cross-encoder on the Apple Neural Engine. It is a bundle, not a file — four entries (Manifest.json, shapes.json, Data/com.apple.CoreML/model.mlmodel, Data/com.apple.CoreML/weights/weight.bin) that must keep their relative paths.

One compiled function per shape, named b<batch>_s<seq>: b16 s64, b16 s128, b16 s256. Fixed shapes are not a simplification. ct.EnumeratedShapes converts and runs, and measures 233 emb/s against 2 873 on the same encoder, because the dynamic ops it injects push the graph off the Neural Engine. shapes.json lists what was compiled, so a caller can ask instead of assuming.

Measured on an M4 through the daemon's Core ML wrapper, batch 16: 1 909 rows/s at s64, 829 at s128, 275 at s256 — a 4.5x spread per row, which is why the daemon routes each window to the shortest function that covers it rather than padding the whole call to the longest.

Weights are fp16. The package is built from the MLX safetensors beside it, so a model is ANE-ready exactly when it is MLX-ready — there is no second set of source weights. Load it with MLComputeUnits.cpuAndNeuralEngine: plain .all lets Core ML place the graph on the GPU instead, which measured 809 emb/s against the ANE's 2 873.

A shape the package does not carry is not an error — the caller is expected to fall back to the MLX graph, which takes any shape. That is what makes the fixed-shape package safe to ship alongside model_gpu_mlx*/ rather than instead of it.

8. License & attribution

Released MIT.

Warm-start base: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 — mMARCO ← MS MARCO, whose terms are non-commercial research.

⚠️ The warm-start base derives from MS MARCO (non-commercial). Whether a fine-tuned model inherits dataset-use terms is legally unsettled — this is not legal advice. Retrain from a permissive base if strict compliance matters to you.

Warm-started from cross-encoder/mmarco-mMiniLMv2-L12-H384-v1. Backbone: XLM-RoBERTa. Used by the UltraCode code assistant, though nothing about the model is specific to it.