Instructions to use SORA-Lab/TriCosmos-2.5-predict with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Cosmos
How to use SORA-Lab/TriCosmos-2.5-predict with Cosmos:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
TriCosmos-2.5-predict
A driving world model that forecasts RGB, thermal, and LiDAR depth together. Given 5 observed frames (0.5 s at 10 Hz), it forecasts the next 20 frames (2 s) of all three sensors, in real adverse weather: fog, rain, snow, and night.
- Base model: Cosmos-Predict2.5-2B (
base/post-trained), post-trained with LoRA - Training data: DSERT-RoLL (real driving data with synchronized RGB, thermal, and LiDAR)
- Key idea: the three sensors are shown to the model as one video, a 2Γ2 grid of sensor tiles
Paper under review. Full training and evaluation code will be released with the paper.
Contents
- Repository files
- Quick start
- Input format
- Reading the output
- Model and training details
- Results
- Limitations
- License and citation
Repository files
| File | Description |
|---|---|
tricosmos_ema_bf16.pt |
Model weights (EMA, bf16, 3.9 GB): the base model with the trained LoRA weights, training step 500. This is the checkpoint behind all reported results. |
TriCosmos_inference.ipynb |
Google Colab notebook: install, download, forecast, and a side-by-side comparison video. |
examples/light_snow_example.mp4 |
Test window from the DSERT-RoLL validation set (Light Snow, night): 5 observed + 20 real future frames. |
examples/light_snow_example.json |
The exact caption used for that window. |
config/experiment.json |
Model options used in training. Needed to rebuild the model. |
config/export.json |
Checkpoint name and training step. |
config/train_tri_lora.yaml |
Full training recipe. |
Quick start
Requirements
An NVIDIA GPU with at least 48 GB of memory and bf16 support. The 2B video model runs together with Cosmos-Predict2.5's large text encoder and video tokenizer at 1280Γ704.
GPU Memory Works? NVIDIA RTX PRO 6000 Blackwell (Colab runtime G4) 96 GB β tested NVIDIA H100 80 GB β tested NVIDIA A100 80 GB 80 GB β used for the paper's results Colab A100 (40 GB), L4 (22.5 GB), T4 (15 GB) < 48 GB β not enough memory A Hugging Face account that has accepted the license of nvidia/Cosmos-Predict2.5-2B. The base model's tokenizer and text encoder are loaded from NVIDIA's gated repositories.
Option 1: Google Colab (easiest)
- Download
TriCosmos_inference.ipynband open it in Colab (File β Upload notebook). - Select Runtime β Change runtime type β G4 (RTX PRO 6000, 96 GB). Colab's A100 and L4 do not have enough memory.
- Run all cells from top to bottom and paste your Hugging Face token when asked.
The notebook checks the GPU first, then forecasts the Light Snow example with TriCosmos and with the zero-shot base model, then shows a side-by-side video: input | real future | TriCosmos | Cosmos-Predict2.5 zero-shot.
Option 2: Local machine
# 1. Cosmos code, exact version used for this model
git clone https://github.com/nvidia-cosmos/cosmos-predict2.5
cd cosmos-predict2.5
git checkout a2c298b0a3df3778b973fe65e9e58877b292d8a7
uv sync --extra=cu128 --python 3.10
uv pip install --python .venv/bin/python "imageio[pyav]" av
# 2. This model
hf auth login
hf download SORA-Lab/TriCosmos-2.5-predict --local-dir ../tricosmos
# 3. Forecast the example window
.venv/bin/python ../tricosmos/tricosmos_infer.py \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_tricosmos
# Optional: the zero-shot base model on the same input
.venv/bin/python ../tricosmos/tricosmos_infer.py --mode zeroshot \
--model-dir ../tricosmos \
--input ../tricosmos/examples/light_snow_example.mp4 \
--prompt-json ../tricosmos/examples/light_snow_example.json \
--out ../forecast_zeroshot
Each output folder holds forecast_grid.mp4 and rgb.mp4, thermal.mp4, lidar.mp4 at 1280Γ704.
Settings used in the paper (the script defaults): 35 sampling steps, guidance 7, seed 0. One forecast takes about 9 minutes on an RTX PRO 6000, model loading included.
Input format
The model expects a video of 1280Γ704 frames at 10 Hz, each made of four 640Γ352 tiles:
| left | right | |
|---|---|---|
| top | RGB camera | thermal camera |
| bottom | LiDAR depth | black (unused) |
Only the first 5 frames are used as input.
| Tile | How it is made |
|---|---|
| RGB | Left RGB camera, 1280Γ704, area-averaged to 640Γ352. |
| Thermal | Left thermal camera, 8-bit gray copied to 3 channels, 1280Γ704, area-averaged to 640Γ352. Shows the thermal camera's own view (not warped to the RGB image). |
| LiDAR depth | Depth video aligned with the left RGB camera: 3 accumulated, ego-motion-compensated sweeps of the long-range LiDAR, projected into the camera, densified with a 5Γ5 elliptical dilation and a 5Γ5 morphological closing, then max-pooled over 2Γ2 blocks (the nearest depth wins). |
| Black | All zeros. Fills the fourth slot so the grid keeps the model's native size. |
LiDAR depth encoding. Depth d in meters (up to 80 m) is stored as an 8-bit code, copied to 3 channels:
q = round(1 + 254 * (1 - d / 80)) q = 0 means no LiDAR return
Near objects are bright, far objects dark, and empty pixels black.
Caption. The text prompt names the 2Γ2 layout, then the weather and the lighting, in this format (from the Light Snow example):
driving scene shown as a 2x2 grid: front RGB camera top left, thermal camera top right, lidar depth map bottom left. front camera driving video, light snow, low light
For other data, change only the weather and lighting at the end.
Reading the output
The output has 25 frames: frames 0β4 reproduce the input, and frames 5β24 are the forecast (0.1 s to 2.0 s ahead).
- Split each frame into its tiles: RGB
[:352, :640], thermal[:352, 640:], LiDAR[352:, :640]. - Upsample each tile to 1280Γ704: bilinear for RGB and thermal, nearest neighbor for LiDAR.
- Decode LiDAR depth in meters:
depth_m = np.where(q > 0, 80.0 * (1 - (q - 1) / 254.0), np.nan)
Model and training details
| Setting | Value |
|---|---|
| Base model | Cosmos-Predict2.5-2B, base/post-trained (video-to-world) |
| Cosmos code | cosmos-predict2.5, commit a2c298b0a3df3778b973fe65e9e58877b292d8a7 |
| LoRA | rank 16, alpha 32, on q_proj, k_proj, v_proj, output_proj, mlp.layer1, mlp.layer2 of every transformer block |
| Frozen | base weights, text encoder, video tokenizer |
| Conditioning | 2 latent frames (= 5 observed frames) |
| Optimizer | learning rate 1e-4, weight decay 1e-3 |
| Schedule | 50 linear warm-up steps, then linear decay to 10% |
| Training length | 500 steps, bf16, 93-frame clips |
| Weather balancing | clips sampled with weights β n_w^(-1/2), where n_w is the number of training clips of the clip's weather condition |
| Training data | 35 DSERT-RoLL training sequences: Clear 5, Fog 23, Heavy Rain 1, Heavy Snow 5, Light Rain 1. Light Snow was not in training. |
Results
DSERT-RoLL official validation set: 58 sequences, six weather conditions, 300 windows. 2 s horizon, all models scored at tile resolution (640Γ352 per sensor). Camera metrics use windows where the scene is visible; LiDAR is scored in meters against the measured LiDAR points.
| Method | RGB PSNR β | RGB LPIPS β | Thermal PSNR β | Thermal LPIPS β | LiDAR MAE (m) β | LiDAR Ξ΄1 β |
|---|---|---|---|---|---|---|
| Persistence (repeat last frame) | 19.26 | 0.351 | 22.90 | 0.259 | 5.71 | 0.777 |
| Single-sensor LoRA models | 21.79 | 0.284 | 24.86 | 0.243 | 6.81 | 0.752 |
| TriCosmos (this model) | 21.71 | 0.304 | 24.61 | 0.270 | 4.68 | 0.794 |
- LiDAR: forecasting the three sensors jointly lowers the LiDAR error by 2.1 m compared with a LiDAR-only model (4.68 m vs. 6.81 m). TriCosmos is the only model whose LiDAR forecasts beat persistence in mean error.
- Cameras: the RGB and thermal forecasts stay close to those of the single-sensor models, with a small loss in LPIPS.
Limitations
- Near range: for LiDAR at less than 10 m, repeating the last frame is still more accurate than the forecast.
- Resolution: each sensor is seen at half resolution (640Γ352) inside the grid.
- Heavy Snow: only two validation sequences, so results in this condition are not reliable.
- Thermal registration: the thermal tile is not registered to the RGB image.
- Speed: about 9 minutes per forecast on an RTX PRO 6000 (model loading included); not for real-time use.
- Memory: needs a GPU with at least 48 GB.
- No action input: the model is not conditioned on driving actions.
- Intended use: research on world models and sensor forecasting. Not for safety-critical decisions.
License and citation
License. The weights are derived from Cosmos-Predict2.5-2B and are distributed under the NVIDIA Open Model License. The model was trained on DSERT-RoLL; the example window comes from its validation set. See the dataset's license for any use of the data.
Acknowledgments. Built on NVIDIA Cosmos-Predict2.5 and the DSERT-RoLL dataset (Cho et al., CVPR 2026).
Citation. The paper is under review. A citation will be added after publication.
license: apache-2.0
- Downloads last month
- 47
Model tree for SORA-Lab/TriCosmos-2.5-predict
Base model
nvidia/Cosmos-Predict2.5-2B