tiny-h2o-lightning-4b-for-testing

This is a modified, damaged copy of h2oai/h2o-lightning-4b, made only to test software. Its output is gibberish. Do not use it for anything else.

It keeps the architecture, tokenizer, chat template, serve_config.json (prompt, labels, temperatures) and MTP layer of h2o-lightning-4b, so it runs the same code paths as the real model in llama.cpp (h2o-lightning decision type: labels read at the token after the Answer: prefill, labels and temperatures from serve_config.json, image input), but it is small enough for CI.

GGUF version: ggml-org-dev/tiny-h2o-lightning-4b-for-testing-GGUF

Changes from the original model

Sliced from the safetensors with slice_qwen35_decision.py (python3 slice_qwen35_decision.py h2oai/h2o-lightning-4b <out dir>), which follows the slice-model skill of llama.cpp:

  • Text model: 4 of 32 layers (0-3: 3 linear attention + 1 full attention), hidden size 2560 -> 32, FFN 9216 -> 64, heads 16 -> 2, KV heads 4 -> 1 (head_dim 256 kept), linear attention key heads 16 -> 2, value heads 32 -> 4 (head dims 128 kept). The MTP layer is narrowed the same way
  • Vision encoder: 2 of 24 blocks, hidden size 1024 -> 128, heads 16 -> 2 (head_dim 64 kept), FFN 4096 -> 256, output 2560 -> 32
  • h2o_lightning_shim.py is kept unchanged only because convert_hf_to_gguf.py detects the checkpoint by it
  • The other Python code, tests, label_rows.lightning (two-way mode), tokenizer/ copy, vllm_patches/, engine/ and assets of the original repo are not included

License

Derived from h2o-lightning-4b by H2O.ai (itself built on Qwen/Qwen3.5-4B), under the Apache License 2.0.

Downloads last month
12
Safetensors
Model size
9.36M params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-h2o-lightning-4b-for-testing

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(10)
this model
Quantizations
1 model