apande-TT/nemotron-3-5-lightning-30b-a3b-bf16-tt

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — on Tenstorrent

A Tenstorrent deployment of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — the same 30B checkpoint, running on Tenstorrent p300x2 and served through an OpenAI-compatible API with a single command. The Validated on Tenstorrent section reports figures measured on Tenstorrent hardware; the sections that follow describe the base weights and are attributed to NVIDIA.

Validated on Tenstorrent

Measured with tt serve on Tenstorrent p300x2 (mesh 1×4) with greedy decoding, each user sending its own prompt. Each row can be reproduced with the script in Reproduce the speed below.

Batch (users)	ISL	OSL	TTFT	TPOT	tok/s/user	tok/s total
1	128	128	~0.13 s	~25 ms	~40	~40
32	128	128	~1.5 s	~25 ms	~40	~1,270

TTFT: time to first token. TPOT: time per output token. tok/s/user = 1 / TPOT; tok/s total = batch × tok/s/user.

The package serves up to 32 users concurrently, each with its own Mamba state and attention cache, and a context of 2048 tokens (prompt + output) per user. Decode tokens are sampled on the chip. In a correctness check, 32 different questions (the capitals of 32 countries) submitted at once each received the correct answer — the same answer as when asked individually under greedy decoding.

Run on Tenstorrent

uv tool install tenstorrent          # once — installs the Tenstorrent CLI, `tt`
tt model pull apande-TT/nemotron-3-5-lightning-30b-a3b-bf16-tt
tt serve apande-TT/nemotron-3-5-lightning-30b-a3b-bf16-tt        # OpenAI-compatible API on http://localhost:20000/v1

On first use, tt asks once whether to enable anonymous usage telemetry; either answer works. tt serve notes that this is an unverified community bundle and asks for confirmation — answer y, or pass --yes. In an interactive terminal it streams the boot log and returns once the server is ready: about 7 minutes on the first boot (kernels are compiled once) and about 5 minutes thereafter. When run from a script, it returns as soon as the container starts; the server is ready once the following command succeeds:

until curl -sf http://localhost:20000/v1/models >/dev/null; do sleep 10; done; echo ready

If port 20000 is in use, the server binds the next free port; tt model ps shows the port in use.

The server accepts requests from any OpenAI client (pip install openai), with the base-model id as the model name. Reasoning is enabled by default and can be disabled per request with "chat_template_kwargs": {"enable_thinking": false}, as in this example:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:20000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
    messages=[{"role": "user", "content": "What is the capital of Japan?"}],
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

Stopping and restarting. After the server is stopped, the board must be reset before the next tt serve; without a reset, the boot fails within about 2 minutes with Timed out while waiting for active ethernet core ... Try resetting the board:

tt model stop apande-TT/nemotron-3-5-lightning-30b-a3b-bf16-tt
tt device reset -y                    # about 45 s; then run `tt serve` again

Reproduce the speed

The script below prints one row of the table above. Saved as bench.py, python bench.py 1 produces the single-user row and python bench.py 32 the 32-user row:

import json, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

URL, MODEL = "http://localhost:20000", "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
USERS, ISL, OSL = int(sys.argv[1]), 128, 128
TOPICS = ["science", "music", "the ocean", "cities", "farming", "mathematics", "medicine", "trains"]

def prompt_ids(u):  # each user sends its own 128-token prompt, as in real traffic
    text = f"Story {u}. The history of {TOPICS[u % len(TOPICS)]} is a long story of people, ideas and work. " * 60
    req = urllib.request.Request(f"{URL}/tokenize", json.dumps({"model": MODEL, "prompt": text}).encode(),
                                 {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req))["tokens"][:ISL]

prompts = [prompt_ids(u) for u in range(USERS)]
client = OpenAI(base_url=f"{URL}/v1", api_key="EMPTY", timeout=3600)

def one_user(u):
    stamps = []
    stream = client.completions.create(model=MODEL, prompt=prompts[u], max_tokens=OSL, temperature=0, stream=True,
                                       extra_body={"ignore_eos": True})
    for chunk in stream:
        if chunk.choices and chunk.choices[0].text is not None:
            stamps.append(time.time())
    return stamps

t0 = time.time()
with ThreadPoolExecutor(USERS) as ex:
    runs = list(ex.map(one_user, range(USERS)))
ttft = sum(r[0] - t0 for r in runs) / USERS
tpot = sum((r[-1] - r[0]) / (len(r) - 1) for r in runs) / USERS
print("| Batch (users) | ISL | OSL | TTFT | TPOT | tok/s/user | tok/s total |")
print(f"| {USERS} | {ISL} | {OSL} | {ttft:.2f} s | {1000 * tpot:.1f} ms | {1 / tpot:.1f} | {USERS / tpot:.0f} |")

Model Summary

Total Parameters	30B (3B active)
Architecture	MoE — Mamba-2 + MoE + Attention hybrid
Precision	BF16 checkpoint; on the chips, the experts are stored in 4-bit (bfloat4_b), the Mamba, shared-expert and LM-head weights in 8-bit (bfloat8_b), and attention in BF16
Context Length	2048 tokens per user in this package (base weights: up to 1M on GPU)
Concurrent users	Up to 32 in this package
Tenstorrent Hardware	p300x2 (Blackhole), mesh (1, 4)
Supported Languages (base model)	English (+ code), Spanish, French, German, Italian, Japanese
License	Packaging: Apache-2.0 · Base weights: OpenMDW-1.1
Base model	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Status	Community Tenstorrent deployment

Model Overview

Model developer (base model): NVIDIA Corporation. Tenstorrent deployment: community port.

What is Nemotron? NVIDIA Nemotron™ is a family of open models (open weights, data and recipes). Nemotron 3.5 Lightning is a hybrid Mixture-of-Experts model with interleaved Mamba-2 + MoE layers and select Attention layers, 3B active / 30B total, released alongside speculative-decoding drafters and Multi-Token Prediction (MTP). (Base-model description, from the upstream card.)

Description

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is the full-precision (BF16) reference release of Nemotron 3.5 Lightning. This repository packages that checkpoint for Tenstorrent: the weights are downloaded unchanged from the upstream repository and converted at load time (experts to 4-bit, most other weights to 8-bit block floating point), and the model runs on device through the Tenstorrent vLLM generator behind an OpenAI-compatible API. The tokenizer, chat template and license are inherited unchanged.

Base‑model benchmarks (NVIDIA, measured on GPU)

Accuracy of the base weights as published by NVIDIA (NeMo Gym / NeMo Evaluator, on GPU); the upstream card contains the full suite and plots. These scores have not been measured on Tenstorrent. NVIDIA ran them with reasoning enabled and a 262,144-token context, whereas this package serves 2048 tokens per user and stores the experts in 4-bit, so scores on Tenstorrent can differ.

Task	Nemotron-3.5-Lightning-30B-A3B-BF16 (NVIDIA GPU)
MMLU Pro	81.94
GPQA Diamond (no tools)	75.44
HLE (text-only, no tools)	11.72
SciCode	32.60
SWE-bench Verified	51.56
Terminal-Bench 2.1	24.58
IFBench (loose)	71.88
AA-LCR (long context)	52.00

Base‑weight accuracy, measured by NVIDIA on GPU.

Model Architecture

• Architecture Type: Mixture-of-Experts hybrid (Mamba-2 + MoE + Attention)
• Network: Nemotron-3-Lightning + Multi-Token Prediction (MTP)
• Parameters: 30B total / 3B active
• Precision: BF16

(Base-model facts. On Tenstorrent, the full 52-layer graph runs on the p300x2 mesh, tensor-parallel across its 4 chips; the MTP layers are not used.)

Training, Datasets, and Methodology

The training methodology (pre-training on 20T+ tokens, MTP continued pre-training, SFT, and RL via NeMo RL/Gym) and the complete pre- and post-training dataset tables are NVIDIA's and unchanged; this package does not retrain or modify the weights. The upstream card's Training Methodology and dataset sections contain the full tables. No training was performed on Tenstorrent.

Input / Output

• Input: Text (1-D token sequences). Output: Text. Context window in this package: 2048 tokens per user (prompt + output); the base weights support up to 1M tokens in GPU deployments.

Software Integration

• Runtime engine (Tenstorrent): TTNN / tt-metal with the Tenstorrent vLLM plugin
• Supported hardware (this package): Tenstorrent p300x2 (Blackhole)
• Base-model supported hardware (upstream): NVIDIA Ampere (A100), Hopper (H100/H200), Blackwell (GB200/B200) — GPU; not applicable to this package

Base‑model GPU deployment (reference)

For GPU deployments of the base weights, NVIDIA serves the model with vLLM (--mamba-backend flashinfer, --reasoning-parser nemotron_v3, --tool-call-parser qwen3_coder) or SGLang, with optional DSpark / DFlash / MTP speculative decoding and up to 1M context on multi‑GPU systems. On Tenstorrent, the model is served with tt serve as described above. Full GPU recipes are in the upstream card.

Inference — Test Hardware

• This package: Tenstorrent p300x2 (Blackhole), mesh (1, 4).
• Base model (upstream): NVIDIA Hopper (1–8× H100/H200), Blackwell (GB200). (GPU — not this package.)

Build from source (public)

All components needed to rebuild this package are public:

• Model code — the code/ folder in this repository is the complete model package contained in the image. The bring-up is in the tt-metal fork apande-TT/tt-metal, branch nemotron-3-5-lightning-bringup, commit 2dc606c7c; the 32-user serving layer on top of it consists of code/models/demos/nvidia_nemotron_3_5_lightning_30b_a3b_bf16/tt/serve.py, tt/sampler.py, tt/kernels/, the serving adapter in vllm_bundle/, and three files in _stubs/.
• Serving stack — tenstorrent/vllm at 7c99bd3b8, with its in-tree plugin plugins/vllm-tt-plugin at the same commit.

The weights are fetched from nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 into the local Hugging Face cache and are not included in the image.

Provenance

code/ in this repository is byte-identical to the model code inside the published image:

Component	Built from
tt-metal	2dc606c7c2402701ba0411677aea9ce437ac54b5 (branch nemotron-3-5-lightning-bringup), plus the serving layer in code/
vLLM	7c99bd3b8 — tenstorrent/vllm, public dev
vllm-tt-plugin	in-tree plugins/vllm-tt-plugin of tenstorrent/vllm 7c99bd3b8
code/ digest	b9d19613d4a56065 (sha256, first 16 hex digits)

Ethical Considerations & License

Ethical-use, bias, privacy, safety and explainability guidance for the base model is NVIDIA's; see the Model Card++ subcards on the upstream card. Use of the weights is governed by the OpenMDW-1.1 license; the packaging is licensed under Apache-2.0.

Feedback

Questions and issue reports are welcome in the community discussions.