mkirkby commited on
Commit
a5923ff
Β·
verified Β·
1 Parent(s): 3903da0

Match the elephant-1 and mammoth-1 model card; add the banner

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +32 -16
  3. assets/banner.png +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ assets/banner.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -16,16 +16,28 @@ tags:
16
  - baseten
17
  ---
18
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  # joey-1
20
 
21
  joey-1 is a 2B parameter **decision model** post-trained from [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) by Base Labs at Baseten.
22
  Provide the model a state (text, optionally with images) and a set of questions and it will return a calibrated probability for every option,
23
  answering all questions from **one shared-prefix prefill**. On tested benchmarks:
24
 
25
- - **63.3 on the public split of the Jev Decision Index**, ahead of every listed decision model of 2.5B parameters or fewer.
26
  - **62.9 on a replica of the Artificial Analysis Decision Index**.
27
 
28
- For tasks that need more accuracy, see the 9B sibling [elephant-1](https://huggingface.co/baselabs/elephant-1) and the 27B [mammoth-1](https://huggingface.co/baselabs/mammoth-1).
29
 
30
  ## πŸ—’οΈ Model Details
31
 
@@ -43,6 +55,11 @@ For tasks that need more accuracy, see the 9B sibling [elephant-1](https://huggi
43
 
44
  ## πŸƒ How to use
45
 
 
 
 
 
 
46
  ### Self-hosted
47
 
48
  `setup.sh` installs the pinned serving runtime (OneJev + `serve.patch`). It needs one CUDA GPU and the `pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel` image.
@@ -87,33 +104,32 @@ Requests longer than the trained length limits are refused, not truncated.
87
 
88
  ## πŸ“Š Performance
89
 
90
- ### Jev Decision Index, public split (our runs)
91
 
92
- | Model | Public score |
93
  |---|---:|
94
- | elephant-1 | 72.2 |
95
- | **joey-1** | **63.3** |
 
 
 
96
 
97
- | Area | joey-1 |
98
- |---|---:|
99
- | Knowledge & reasoning | 50.0 |
100
- | Language | 62.8 |
101
- | Retrieval & classification | 71.2 |
102
- | Tools & automation | 85.8 |
103
- | Arts & human taste | 54.8 |
104
 
105
  ### Artificial Analysis Decision Index (our replica)
106
 
107
  | Model | Score |
108
  |---|---:|
 
109
  | elephant-1 | 65.2 |
 
110
  | **joey-1** | **62.9** |
111
 
112
- ## πŸ”§ Training
113
 
114
- - **Data**: 6.21M decision examples (Mix 8), the same mix as elephant-1.
115
  - **Objective**: cross-entropy plus a Brier calibration term (Ξ» = 1), with a trained "I don't know" option.
116
- - **Recipe**: 1 epoch (43,390 steps), peak LR 1e-5, AdamW (no weight decay), 3% warm-up, cosine decay, bf16.
117
  - **Hardware**: 2 nodes Γ— 8 H100 (16 GPUs).
118
 
119
  ## Citation
 
16
  - baseten
17
  ---
18
 
19
+ <div align="center">
20
+ <img
21
+ src="https://huggingface.co/baselabs/joey-1/resolve/main/assets/banner.png"
22
+ alt="Baseten joey-1 decision model"
23
+ style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;"
24
+ />
25
+ <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;">
26
+ <a href="https://www.baseten.co/library/joey-1/"><strong>Try on Baseten</strong></a> β€’
27
+ <a href="https://labs.baseten.co/articles/mammoth-1-and-elephant-1-frontier-decision-models"><strong>Technical report</strong></a>
28
+ </div>
29
+ </div>
30
+
31
  # joey-1
32
 
33
  joey-1 is a 2B parameter **decision model** post-trained from [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) by Base Labs at Baseten.
34
  Provide the model a state (text, optionally with images) and a set of questions and it will return a calibrated probability for every option,
35
  answering all questions from **one shared-prefix prefill**. On tested benchmarks:
36
 
37
+ - **Best model under 4B on the Jev Decision Index 0.4 public split**: 63.30, #37 of 230 models.
38
  - **62.9 on a replica of the Artificial Analysis Decision Index**.
39
 
40
+ For tasks that need broader general knowledge, see the 9B sibling [elephant-1](https://huggingface.co/baselabs/elephant-1) and the 27B [mammoth-1](https://huggingface.co/baselabs/mammoth-1).
41
 
42
  ## πŸ—’οΈ Model Details
43
 
 
55
 
56
  ## πŸƒ How to use
57
 
58
+ ### Baseten Model APIs
59
+
60
+ joey-1 is available on [Baseten Model APIs](https://www.baseten.co/library/joey-1/) at $0.01 per 1M input tokens, with zero data retention on by default. The endpoint accepts text and image inputs.
61
+
62
+
63
  ### Self-hosted
64
 
65
  `setup.sh` installs the pinned serving runtime (OneJev + `serve.patch`). It needs one CUDA GPU and the `pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel` image.
 
104
 
105
  ## πŸ“Š Performance
106
 
107
+ ### Jev Decision Index 0.4 (public split)
108
 
109
+ | Model | Decision Index |
110
  |---|---:|
111
+ | elephant-1 | 72.24 |
112
+ | **joey-1** | **63.30** |
113
+ | Cloudflare Clef | 61.71 |
114
+ | Jev 1.14 | 60.70 |
115
+ | Perplexity Decider Flash v1.2 | 60.06 |
116
 
117
+ On the public split, joey-1 trails elephant-1 in every area and beats Jev 1.14 in four of five (language, tools & automation, retrieval & classification, arts & human taste).
 
 
 
 
 
 
118
 
119
  ### Artificial Analysis Decision Index (our replica)
120
 
121
  | Model | Score |
122
  |---|---:|
123
+ | mammoth-1 | 65.9 |
124
  | elephant-1 | 65.2 |
125
+ | d1-ultra | 65 |
126
  | **joey-1** | **62.9** |
127
 
128
+ ## πŸ”§ Training (all details in the technical report)
129
 
130
+ - **Data**: 6.21M decision examples (Mix 8). 58.3% core decision skills, 16.1% safety and abstention, 16.1% judging and expert domains, 9.5% public decision data.
131
  - **Objective**: cross-entropy plus a Brier calibration term (Ξ» = 1), with a trained "I don't know" option.
132
+ - **Recipe**: 1 epoch (43,390 steps), peak LR 1e-5, AdamW (Ξ² 0.9/0.95, no weight decay), 3% warm-up, cosine decay, bf16.
133
  - **Hardware**: 2 nodes Γ— 8 H100 (16 GPUs).
134
 
135
  ## Citation
assets/banner.png ADDED

Git LFS Details

  • SHA256: 6201739a376b851ae1c2afc3b31f7c40489b315dbe502615268b4c3c04e34752
  • Pointer size: 132 Bytes
  • Size of remote file: 1.47 MB