KnowLine-35B-A3B-Gen1

The first release of PelaAI's 35B-A3B System One decision model, trained in the same loop as the KnowLine-4B line.

Inference guide · 中文说明 · weights Apache-2.0 · smaller sibling: KnowLine-4B-Gen4

Highlights:

• Decision Index 0.3 public suite: 69.49 (self-run), 4.59 points above KnowLine-4B-Gen4 (64.90). On the board, the highest public score is 65.64 (RSI-Jev v6.1 VL 27B), followed by vLLM-SR d3 (27B) at 65.45, which has the highest full score (66.26); Perplexity Decider v1.1 (27B) scores 62.25 and Jev 1.13 scores 57.96; see Comparison.
• Stronger where the 4B was weakest: knowledge and reasoning rises from 44.6 (Gen4) to 52.5, close to Jev 1.13 (53.9); see Comparison.
• Calibrated probabilities: a temperature, fitted on held-out data, is folded into the weights. Computed with the board's method, ECE is about 0.04-0.05 (board median: 0.078) and the Brier score about 0.25-0.27, below the lowest Brier score on the board (0.28); see Calibration.
• Other public benchmarks: C-Eval 87.6 (Gen4: 78.7), JevBench / JevBench-hard 85.7 / 70.3 (86.1 / 72.1): slightly lower (the JevBench-hard difference is not significant); see Evaluation.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of KnowLine-4B only need to change the model name, because the interface is the same.

Base model	Qwen/Qwen3.5-35B-A3B (Apache-2.0): mixture of experts, 35B parameters in total, about 3B active per token
Training	LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights
Release format	bf16 weights (about 67 GB), with a readout temperature T = 1.20 folded into the final norm weight (see Calibration). The vision tower and MTP head are unchanged from the base model; the tokenizer is the same as in KnowLine-4B.
Serving	the same serve_knowline.sh as KnowLine-4B: SGLang converts the weights to FP8 at load (about 35 GB). We ran every evaluation with one server on one 96 GB H20; plan for at least 48 GB of GPU memory.
Languages	English, Simplified Chinese, Traditional Chinese

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-35B-A3B-Gen1 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same code as in KnowLine-4B-Gen4.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

How Gen1 was made

• Base model: Qwen/Qwen3.5-35B-A3B (Apache-2.0).
• Training: LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights. Training examples include decision tasks written and labelled by LLM teachers.
• Readout temperature: a single temperature T = 1.20, fitted on held-out data, is folded into the weights. It changes the size of the probabilities, but not which option ranks first.

The training data is not included in this repository.

Comparison

Scores on the Decision Index 0.3 public suite. The full 0.3 score also includes private tests that only the maintainers run (weighted 0.5 same-skill and 0.3 new-domain); our full score is not available yet.

model	size	DI 0.3 public	DI 0.3 full	source
KnowLine-35B-A3B-Gen1	35B-A3B	69.49	not yet scored	self-run, official kit
KnowLine-4B-Gen4	4B	64.90	not yet scored	self-run, official kit
RSI-Jev v6.1 VL 27B (effort auto)	27B	65.64	58.66	board
vLLM-SR d3	27B	65.45	66.26	board
Torchcast Decision 27B	27B	65.10	59.91	board
Perplexity Decider v1.1	27B	62.25	62.75	board
Clef	27B	61.71	53.08	board
Jev 1.13	(API)	57.96	60.11	board
Cloudflare clef-omni	30B-A3B	56.54	48.28	board
JPT-35B-A3B	35B-A3B	52.68	51.41	board
decisio 0.4.0	35B-A3B	48.43	47.72	board

Board rows as of the board's latest update (2026-10-13).

Public-suite scores by area (DI 0.3):

area	KnowLine-35B-A3B-Gen1	KnowLine-4B-Gen4	Perplexity Decider v1.1 (27B)	Jev 1.13
Knowledge & reasoning	52.5	44.6	52.0	53.9
Language	71.2	65.5	67.5	59.2
Retrieval & routing	75.2	72.5	61.3	55.4
Tools & agents	88.1	87.6	78.9	75.1
Arts & taste	63.4	59.2	47.0	39.1

Releases

The model is trained in a self-evolving loop, and new versions will follow.

model	date	DI 0.3 public	notes
KnowLine-35B-A3B-Gen1 (this model)	2026-10-12	69.49	first 35B-A3B release; board figures updated 2026-10-13
KnowLine-4B-Gen4	2026-10-10	64.90	4B, fourth release
KnowLine-4B-Gen3	2026-10-09	63.11	4B, third release
KnowLine-4B-Gen2	2026-10-08	62.54	4B, second release
KnowLine-4B-Gen1	2026-10-07	60.47	4B, first release

Evaluation

All results are self-run and have not been verified by a third party.

Other public benchmarks

suite	35B-A3B-Gen1	4B-Gen4
C-Eval (4 categories, macro)	87.6	78.7
JevBench / JevBench-hard	85.7 / 70.3	86.1 / 72.1

In a paired sign test against Gen4, the difference on JevBench-hard is not significant.

Web operation (Mind2Web official test splits)

split	35B-A3B-Gen1 element selection	35B-A3B-Gen1 operation (balanced)	4B-Gen4 element selection	4B-Gen4 operation (balanced)
test_task (Cross-Task)	94.4	97.4	92.9	97.0
test_website (Cross-Website)	92.4	97.5	91.9	95.7
test_domain (Cross-Domain)	93.1	98.0	91.7	97.2

• Slightly above Gen4 on all three splits; no request failed.
• So far, we use this only to explore the model for RPA-style automation and to check that it generalises to some degree.
• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

metric	35B-A3B-Gen1	4B-Gen4	board median (159 models)	Jev 1.13
ECE (lower is better)	0.04-0.05	0.04-0.05	0.078	0.074
Brier score (lower is better)	0.25-0.27	0.29-0.31	0.47	0.36
Confidence ≥95% but wrong	1.7%	1.7%	1.5%	2.1%
Mean confidence / accuracy	0.85 / 0.80	0.82 / 0.77		0.81 / 0.74

• Temperature in the weights: the final norm weight includes a readout temperature T = 1.20, fitted on held-out data, so every logit is divided by 1.20 before the softmax over the option labels. The top-ranked option does not change, so neither do the Decision Index scores; the probabilities are just less extreme.
• Why ranges: the board does not publish which 32 benchmarks it uses. We checked our computation against three board models with public results; our ECE was within about 0.015 of the board's values, so we report ranges over the plausible benchmark sets.
• Brier score: lower than every model on the board; the lowest there is 0.284 (Torchcast Decision 27B). Part of this comes from higher accuracy, not only from calibration.
• Still slightly overconfident: mean confidence is about 5 points above accuracy. Most of the gap comes from HLE, hard reasoning (MuSR, CRUXEval, GPQA Diamond), cfcolor and New Yorker caption matching.
• Recommendation: if your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Decontamination: all training data was checked with standard and strict decontamination against the Decision Index 0.3 public suite and the public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). 10,278 rows that nearly duplicated an evaluation item, repeated its question or claim, or contained its whole state were removed.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.
• Public suite: the Decision Index public suite is public; the board's 0.3 private tests may show a lower score.

Limitations

• Size and speed: it needs much more GPU memory than KnowLine-4B (at least 48 GB with FP8 at load), and each request takes longer: 51 ms median, one request at a time on one H20, against about 31 ms for KnowLine-4B (see INFERENCE.md).
• Knowledge-heavy reasoning: better than the 4B line but still the weakest area (52.5, DI 0.3); HLE remains at chance level.
• Math: answers are given without step-by-step reasoning.
• Prompt injection: the model can be steered by instructions planted in the input. Keep untrusted text clearly delimited and in data fields of the state, not in the instructions.
• Images: there was no image training; check image use on your own data before relying on it.
• Private tests: the 0.3 private tests may show a lower score than the public suite.

Citation

@misc{knowline35ba3bgen1,
  title  = {KnowLine-35B-A3B-Gen1: a 35B-A3B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).