modilify/Modilify-Mk2-preview

Modilify Mk2 Preview · 26B-A5B

Refine a whole canvas. Think in latent space. Carry memory forward.

Modilify Mk2 is a 26B-A5B multimodal block-diffusion model that brings parallel token refinement, latent deliberation, and persistent trajectory memory into one generation loop. A rolling 256-token canvas gives the model room to revise upcoming text together. A dedicated latent Transformer turns the history of those revisions into context for the next step. As output advances, a learned memory writer carries information across canvas boundaries.

The central idea is simple: give each answer a workspace, a memory, and a variable compute budget. Mk2 can refine its internal state over multiple passes before committing text, and release multiple tokens together when its confidence-and-entropy policy permits. Deliberation happens in continuous hidden states, without requiring every internal update to become a visible reasoning token.

Built on DiffusionGemma, Mk2 combines sparse expert routing with a dual-timescale latent architecture. Text, images, and sampled video frames feed the same generation path.

What makes Mk2 different

Architecture choice	What it enables
Parallel block diffusion	Revise a 256-token canvas jointly and commit a variable-length prefix, allowing multiple output tokens per denoising pass.
Trajectory-aware latent deliberation	Condition the next revision on how hidden states have evolved, including their changes, acceleration, and residuals relative to the latest state.
Memory at two timescales	Rebuild working state every pass while retaining persistent slots across rolling-window shifts within a generation.
Memory inside the decoder	Feed working and persistent memory directly into full-attention decoder layers so both can influence token refinement.
Adaptive commitment	Use confidence and entropy to decide how much text to release, with explicit budgets for continued refinement and forced progress.
Sparse multimodal foundation	Select 8 of 128 experts per token and bring text, image, and video context into a shared decoder.

Inside the generation loop

A canvas built for revision

Mk2 maintains a rolling canvas of 256 candidate tokens. Each denoising pass updates the candidates using the encoded prompt, the current canvas, and latent context. Attention lets positions within the canvas inform one another before the output prefix is finalized. After a commit, the canvas shifts forward and opens space for new candidates.

This creates two useful degrees of freedom: how many times to refine and how many tokens to release. A pass can commit a longer prefix when the policy permits, or spend additional computation refining an uncertain frontier.

Deliberation that reads its own trajectory

A 4-layer latent Transformer, 2,816 dimensions wide, builds working context before each decoder denoising pass. It reads the current canvas alongside three complementary sources of state:

• Per-token trajectory history: 16 recent frames represented through four views—hidden state, first difference, second difference, and residual from the latest state—projected to rank 1,024. These give the processor access to the direction and stability of recent revisions. History follows surviving canvas tokens; new positions start empty.
• Denoising tape: 16 pooled probes per frame summarize canvas activity in a time-indexed ring. The tape preserves recent step-level context as token positions move through the window.
• Persistent memory: 256 slots of 2,816 dimensions carry learned summaries across commits within the generation.

The resulting working context enters the decoder through its self-conditioning bridge and working-memory bus. Full-attention layers also read a separate persistent-memory bus. The refinement history becomes an input to the next refinement.

Fast working state, lasting commit memory

Working state is recomputed on every denoising pass. Persistent slots update only when tokens commit, using a Transformer writer with a separate gate for each slot. That writer draws on the committed region's trajectory, working state, and final decoder representation.

This separates rapid revision from memory consolidation: the canvas can keep changing while persistent memory stays stable between commits. When the window advances, the memory slots remain available to later tokens.

Compute that follows the commit frontier

Mk2 combines proposal confidence with an excess-entropy penalty and selects the longest prefix whose cumulative failure score stays below the configured budget. A tighter budget requires stronger evidence before normal commitment; a looser budget admits longer prefixes for the same scores. Stagnation handling and a pondering watchdog bound continued refinement.

Temperature and commit budget are exposed directly at inference time, making the generation policy adjustable per request. These scores govern commitment; they are not calibrated guarantees of factual correctness.

One generation path for text, images, and video

The Gemma 4 vision tower supplies visual features through the DiffusionGemma multimodal encoder. Text, image, and sampled video-frame inputs are encoded into the prefix KV cache that conditions the decoder. The rolling text canvas then uses the same latent deliberation and memory loop across all three input modalities.

Model Summary

Architecture	Mixture-of-Experts block diffusion + dual-timescale latent Transformer
Model Size	26B-A5B
Vision Encoder	Gemma 4 Vision
Layers	30
Number of Experts	128
Selected Experts per Token	8
Vocabulary Size	262,144
Configured Context Length	262,144 tokens
Sliding Window	1024
Canvas Length	256
Latent Width	2,816
Latent Transformer	4 layers, 16 attention heads
Persistent Memory	256 slots × 2,816 dimensions
Memory Writer	2-layer commit-sequence Transformer + per-slot gated writer
Trajectory History	16 frames, 4 views, rank 1,024
Denoise Tape	16 probes per frame
Modality	Text, Image, Video
Preview checkpoint	Training step 900
Adaptation tokens	~12.4 million

Preview release

This first public preview includes merged BF16 weights, inference code, processor, and tokenizer. The text backbone and latent stack are exported from training step 900, after approximately 12.4 million adaptation tokens. The Gemma 4 vision tower is restored from DiffusionGemma.

The release makes the architecture available for hands-on exploration and evaluation. Comprehensive benchmark results are not included; measured speed, reasoning quality, and multimodal reliability remain to be established for specific workloads.

Getting Started

Transformers 5.14.1 is the minimum supported version.

pip install -U transformers torch accelerate

Text generation

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "modilify/Modilify-Mk2-preview"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain why the sky is blue."}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=256,
    denoise_temperature=0.8,
    commit_failure_budget=0.2,
)
new_tokens = output.sequences[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(new_tokens, skip_special_tokens=False)[0])

Image input

from PIL import Image

image = Image.open("example.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Describe the image and identify uncertainty."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Video-frame input

The processor represents video as a sampled sequence of frames. The following example uses PyAV to decode a short local clip and samples at most 32 RGB frames.

import av
from PIL import Image

container = av.open("short_clip.mp4")
decoded = [Image.fromarray(frame.to_rgb().to_ndarray()) for frame in container.decode(video=0)]
stride = max(1, len(decoded) // 32)
frames = decoded[::stride][:32]

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": frames},
        {"type": "text", "text": "Summarize the main visual events in order."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)

Thinking mode

Latent deliberation runs inside the generation loop regardless of the chat template's thinking flag. The flag controls the prompt's request for a textual thought channel:

• enable_thinking=True inserts a system turn that contains <|think|> and still ends the prompt at <|turn>model.
• enable_thinking=False does not inject an empty thought channel. The prompt ends at <|turn>model.

The model may still open <|channel>thought on its own when thinking is disabled. The template flag does not guarantee suppression of generated thought-channel text. Applications should handle that channel explicitly before displaying an answer.

Configurable inference

The two primary knobs are sampling temperature and the prefix failure budget. Both default to the values used in Mk2 training (0.8 and 0.2) and can be changed per call or on the config object.

Parameter	Default	Meaning
denoise_temperature	0.8	Sampling temperature for every canvas step
commit_failure_budget	0.2	Cumulative prefix risk limit for normal commits
jump_failure_budget	2.0	Cumulative risk limit for forced jumps
jump_on_no_progress_after	12	Stagnation steps before a forced jump
max_ponder_steps	64	Watchdog multiplier per requested token
min_trajectory_progress	0.005	Minimum fused-risk improvement counted as progress
canvas_length	256	Rolling diffusion canvas length
repetition_penalty	1.0	Transformers-style repetition penalty
turn_end_token_id	106	Gemma turn terminator

Call-site override:

output = model.generate(
    **inputs,
    max_new_tokens=256,
    denoise_temperature=0.4,
    commit_failure_budget=0.05,
)

Load-time override:

from transformers import AutoConfig

config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
config.denoise_temperature = 0.4
config.commit_failure_budget = 0.05
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    config=config,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

A tighter commit budget allows fewer tokens for the same confidence-and-entropy scores; a looser budget allows more. Temperature changes the sampling distribution and also affects those scores, so its effect on throughput depends on the prompt and generation trajectory. Measure latency and output quality together when tuning these controls.

Generation supports left-padded batches with independent stopping. Batch prompts of similar lengths together for the best throughput. Streaming and caller-supplied KV caches remain limited to batch size 1.

Evaluation status, limitations, and risks

This preview does not include a complete accuracy, robustness, calibration, fairness, or safety evaluation. Architectural features describe how Mk2 generates; they do not establish benchmark superiority or fitness for a particular deployment. The configured context limit is not a validated long-context quality result.

The model can hallucinate facts, citations, visual details, or temporal relationships; reproduce bias, unsafe content, personal information, or copyrighted material; and consume substantial time and memory during long iterative generation. Confidence-based commits control compute. They do not certify that a prefix is true. Visual performance can degrade with poor resolution, motion, occlusion, unusual aspect ratios, or domain shift.

Evaluate the exact deployment on representative, adversarial, and out-of-distribution inputs. Use layered safeguards, monitoring, incident response, and qualified human review. Never delegate autonomous high-risk medical, legal, financial, employment, housing, education, critical-infrastructure, or safety decisions to the model.

License

Released under the Modilify Open Model License 1.0, subject to its responsible-use and derivative-impact terms. Upstream rights, attribution, Apache-2.0 text, and the impact-statement template are retained in NOTICE.md.

Citation

@software{modilify_mk2_preview_2026,
  title = {Modilify Mk2 Preview: 26B-A5B},
  author = {Modilify},
  year = {2026},
  note = {A multimodal dual-timescale latent-deliberation derivative of DiffusionGemma}
}

Also cite the upstream DiffusionGemma release as requested by Google DeepMind.