SearchJev/SearchJev-27B

SearchJev 27B

SearchJev 27B is a typed decision engine ("System One" model) for the Decision Index. It reads a state and a set of typed questions (choice, noul yes/no, ordinal score) and returns, for every question, a probability for every supplied option in a single forward pass. It never generates text.

• Base model: Qwen/Qwen3.8-27B (Apache-2.0), bf16.
• This repository: a LoRA adapter (rank 128, all linear projections) that is merged into the base weights at load time, per-question-type temperatures (calibration.json), the label space (label_space.json) and the /v1/systemone server with the inference code (searchjev/).
• The adapter is the uniform merge of two adapters trained independently from the base model: ΔW = 0.5 · ΔW(stage 3) + 0.5 · ΔW(round 2), stacked along the rank (details in merge_info.json).
• Readout: the probability of each option's label token after the prompt, restricted to the displayed options, divided by a per-type temperature. One forward pass per question; the questions of a request share the encoded state.

Decision Index 0.3 public suite (self-run)

Public index	73.52 (raw index 79.71, breadth 72.97)
Completeness	"complete": true: 140,178 of 140,178 scored requests answered; none refused, none truncated
Areas (skill)	Knowledge & Reasoning 61.1 · Language Understanding 74.9 · Retrieval & Classification 76.6 · Tools & Automation 89.7 · Arts & Human Taste 66.3
Kit	apolinario/decision-index @ 25c6ec7, stock http engine, one request at a time

Latency of the server (NVIDIA H200, one request at a time, 300 requests of the public sample): median 94 ms, mean 312 ms, 80th percentile 413 ms, 95th percentile 864 ms.

Run it

pip install -r requirements.txt
hf download SearchJev/SearchJev-27B --local-dir SearchJev-27B && cd SearchJev-27B
python -m searchjev.server --model . --port 8000 --served-name searchjev-27b
# in the decision-index kit directory:
python -m decision_index pipeline --engine http --option base_url=http://127.0.0.1:8000 \
    --option model=searchjev-27b --out runs/searchjev-27b

Settings used for the submitted run (all defaults; written to server_settings.json): prompt layout plain, no read-twice, LoRA scale 1.0 (adapter merged into the weights), temperatures from calibration.json, batch size 64 questions within a request, prompt budget = the model's context (262,144 tokens). Nothing is ever truncated: a request that does not fit is answered with HTTP 422 "maximum context length" (unsupported); in the submitted run no request reached this limit. The server is single-threaded and answers one request at a time.

Training data

Public data only. Both adapters were trained with a cross-entropy loss over the option menu plus a Brier term, one epoch.

• SearchJev unified data, version 2: search-decision tasks and Chinese/English classification, retrieval and rating tasks.
• About 120 public datasets: natural-language inference, sentiment, stance, question answering, intent, toxicity, preference and rating datasets.
• Train splits of Decision Index benchmarks, rewritten into the request format (declared): the train/dev splits of the public sources of the board's benchmarks; never their test splits. Rows were deduplicated against every Decision Index public-suite request.
• 33 public datasets from new domains (legal, finance, health, science, customer operations, news, software security, ethics, tabular decisions), held out from our own new-domain validation.
• Program-generated gap-fill items with exactly computed answers: phishing-email judgments built from templates and public URL lists, code-output questions from executed public code, logic and tracking generators, long-story reasoning, science multiple choice from SciQ / OpenBookQA / QASC, API-call choice from the API-Bank training data.
• Round 2 adapter: 1.27M questions; stage 3 adapter: 5.24M questions (it continues an adapter first trained one epoch on a 5.83M-question mix of the same sources), including one extra copy of questions an earlier checkpoint found hard.

Disclosures:

• About 3% of the training rows (from the SearchJev unified data) carry labels that were produced or filtered by language models (label tiers llm_teacher, llm_generated, llm_vote, derived_llm_filtered). All gap-fill and new-domain rows added for this model are human-labelled or exactly computed.
• No outputs of other Decision Index or JevBench entrants were used in any form. No Decision Index test item was used for training. Inference settings (prompt layout, adapter strength, the merge weights) were chosen with a rule fixed in advance: 0.6 × the kit's 4,500-request public sample + 0.4 × our own held-out new-domain validation (12 sets built from public datasets never trained on). The private tests were never seen.
• The SearchJev unified data, version 2, is published at SearchJev/search_jev_unified_v2_release_20260928.

Licence

CC BY-NC 4.0 for the adapter, calibration files and code (some training sources carry non-commercial terms). The base model keeps its own licence.