A Low-Cost Decision Model on CPU: Optimizing DiffusionGemma with vLLM - Part I

1. Introduction

Serving decision models in production requires balancing latency, throughput, decision quality, and infrastructure costs. At League, we investigated whether we could improve this balance by optimizing CPU inference for DiffusionGemma as a decision model using vLLM. Our work focused on the execution path behind each decision: vLLM processed every decision in two engine steps, and we changed it so that eligible decisions complete in one.

We evaluated this optimization with DiffusionGemma 26B-A4B in BF16 on two Intel CPU configurations: a 24-vCPU C4 node and a 44-vCPU C3 node. On C4, the optimization reduced latency by 18.8% at concurrency 1, improved throughput by 5.5-15% across concurrency levels 2 to 20, and reduced cost per decision by 5-13%. Choosing C4 also lowered the monthly cost of a continuously running replica by 44% compared with C3 and 79% compared with the A100 deployment, reducing the cost of keeping capacity available between requests.

We also evaluated decision behavior. JevBench accuracy changes were not statistically significant on either machine, but individual decisions changed. Our experiments also showed that moving the same build from C3 to C4 changed more decisions than applying the optimization did. Performance and cost are only part of the evaluation: we also need to understand how implementation and hardware changes affect the decisions a model produces.

In Part I of this blog series, we explain the execution-path optimization and compare its performance and cost across C3 and C4. In Part II, we check whether our CPU optimization still makes the same decisions, and why some of them may change.

Every change described here went upstream to vLLM: the fixes that made structured reads run on CPU are merged, and the optimization is currently under review.

2. Background: DiffusionGemma and CPU inference

Before diving into the optimization, let’s look at the model and serving setup that motivated this work.

2.1 Why DiffusionGemma?

After TypeSafe AI announced Jev, we saw its potential and started building Jev-like solutions for our agents at League. Jev is a decision model: it reads an input, such as a conversation, and answers structured questions with a probability for every option instead of generating free text. Because the options are known in advance, the answers are quick to produce and straightforward to check and audit, both of which matter for AI in healthcare.

The big question was which decision model to run underneath. Jev itself is a hosted service, and working in healthcare gave us two requirements. First, we had to serve the model in our own environment, which meant open weights under a permissive license. Second, the model also had to answer with low latency, because our agents make decisions while a member is waiting for a reply.

Figure 1. Comparison between autoregressive models and DiffusionGemma. (a) An autoregressive model runs one forward pass per output token. (b) DiffusionGemma denoises a block of positions in parallel, for up to 48 steps, and commits the whole block at once. The real canvas holds 256 positions by default. Adapted from vLLM’s blog post.

The second requirement works against autoregressive models. They write an answer one token at a time, each token needs its own forward pass, and every question adds more tokens. After a couple of days of research, we had our answer: DiffusionGemma, which Google released earlier this year under the Apache 2.0 license. Instead of generating tokens left to right, DiffusionGemma refines a canvas, a block of positions that it fills in parallel through iterative denoising (Figure 1). For a decision, the canvas holds the answer template with one open slot per question. A single denoising step predicts every slot at once, and the probabilities over each question’s allowed options are the decision.

2.2 Why CPU inference?

DiffusionGemma checked all the boxes for our decision model. The catch was the hardware required to run the solutions: DiffusionGemma’s size left us little choice. The model was too big for a cheap GPU: in BF16, its weights take 48.1 GiB, more than two L4 GPUs hold together at 24 GB each. That left us with an 80 GB A100, at $4,754 a month for one always-on replica on Vertex AI. Our earlier two-L4 deployment ran FP8 weights with a 1,024-token context, a quarter of the 4,096 tokens we wanted.

For some of the operations we run at League, however, our traffic makes GPU bills hard to justify. A service that makes occasional decisions about conversations sees bursts of requests with long quiet periods in between. The replica keeps running through those gaps, and at low traffic its monthly price matters more than its peak throughput.

Capacity was the other problem. GPU quota and availability decided what we could deploy, and when. That’s why, at League, we wanted to investigate whether we could serve our decision-model workload efficiently on CPUs without compromising the requirements of these applications.

CPUs offered a different trade-off. A 24-vCPU C4 node holds the BF16 weights in memory and costs about a fifth of an A100 ($996 a month as of October 2026 in the us-east4 regions). Its Intel Xeon processor includes AMX, which runs BF16 matrix multiplications in hardware. The same is true of the C3 machines we started with. Matrix multiplications make up most of a transformer’s compute, and AMX was the main reason we chose these two machine series. The question was whether a CPU could also meet the rest of our requirements:

• Keep the model in BF16 with a 4,096-token context.
• Answer a single request in under 1 second at the 95th percentile.
• Keep its decisions stable when the build or the machine changes.

So far, we had the decision model and the potential hardware. The next piece of the puzzle was the inference engine.

2.3 Why vLLM?

At League, vLLM is our main inference engine for our self-hosted models. For this project, it offered three key advantages.

The first was a single serving stack for both GPUs and CPUs. vLLM’s CPU backend serves the same model through the same OpenAI-compatible API as its GPU backend. Fortunately for us, v0.32.0 includes AMX kernels for the attention and Mixture of Experts (MoE) layers. Moving off the A100 meant changing the machine type and the serving image, not the serving engine.

The second was native support for DiffusionGemma and its structured reads, which we describe below.

The third was the open-source nature of vLLM, an engine that we could inspect, modify and contribute to whenever we found opportunities for improvement. This turned out to be essential: before we could optimize DiffusionGemma’s execution on CPU, we first had to make structured reads work on CPU.

2.4 Making structured reads run on CPU

Since v0.31.0, vLLM supports structured reads for DiffusionGemma. Without them, the model would generate free text over many denoising steps, and the answer would have to be parsed out of whatever layout it chose. With structured reads, a request can seed the canvas with the answer template, pin the template positions, stop after one denoising step and restrict the output to the allowed options (Figure 2). The response carries a probability for every option of every question.

Although critical, structured reads were initially tested only on GPU. Our first CPU runs, built from main, revealed three problems, and we fixed each one upstream:

• Narrow canvases were rejected. A read can ask for a canvas narrower than the served one, sized to its answer template, and a narrower canvas is faster. vLLM allowed that only with async scheduling, but the CPU backend always schedules synchronously. vllm-project/vllm#59107 added a synchronous diffusion scheduler that trims each request’s canvas to its own width.
• The engine failed at startup. A change on main made DiffusionGemma’s per-request causal mask int32, while the CPU attention kernel still read it as bool. vllm-project/vllm#59992 made the CPU kernel read int32. The same PR compiled the sampling step again on every platform.
• Async outputs were overwritten. With async scheduling turned on for testing, CPU snapshots of sampled tokens shared buffers that the next step reused, and every read came back corrupted. The default synchronous path was not affected. vllm-project/vllm#59829 makes each snapshot own its memory.

All three fixes are included in the vLLM v0.32.0 release, which we use as the baseline for the CPU optimization. These fixes did more than unblock our CPU runs. They also laid the groundwork for the optimization in the next section, which builds on the synchronous scheduler from vllm-project/vllm#59107 and on the causal mask that vllm-project/vllm#59992 fixed in the CPU attention kernel.

Figure 2. A structured read. The request seeds the canvas with a fixed answer template that leaves one open slot per question. A single denoising step fills every open slot in parallel, restricted to the allowed options. The decision server then normalizes each question’s probabilities over its own options. Questions and options are illustrative.

3. The optimization: Executing eligible reads in one engine step

vLLM runs structured reads with DiffusionGemma along a specific execution path, and that path is what our optimization changes. Before following that path, let’s make a read concrete and check an example to show what the model receives and what it returns.

Consider a healthcare agent at League that classifies user conversations. In this illustrative example, a user writes:

I’ve had a headache for 3 days. Is that normal?

The agent needs two answers about this conversation: what it is about (billing, symptoms or scheduling) and how urgent it is (routine, soon or urgent). Instead of asking DiffusionGemma to write those answers as free text, the agent sends the conversation with an answer template.

The template has one open slot per question, and each question has its own options:

Prompt (rubric)  
Topic:    A = billing   B = symptoms   C = scheduling  
Urgency:  A = routine   B = soon       C = urgent

Canvas (answer template)  
Topic:    [ ]  
Urgency:  [ ]

Read output  
Topic:    A 0.01   B 0.97   C 0.02  
Urgency:  A 0.12   B 0.79   C 0.09

The model scores the allowed options at every open slot in a single denoising step (Figure 2). The decision server then normalizes the scores over each slot’s own options. Here, the read says the conversation is about symptoms and needs a follow-up soon.

A read is therefore one inference request that processes the prompt and scores the options in a fixed template. It writes no free text. The read in this example is what we call an eligible read: a structured read that runs a single denoising step and does not request prompt logprobs. Only eligible reads can take the optimized path, and every read in our workload is one.

3.1 Understanding the original execution path

In vLLM v0.32.0, our baseline, and in v0.31.0 before it, every eligible read is executed across two engine steps: first it processes and caches the prompt, then it scores the open slots. An engine step is one iteration of vLLM’s engine loop: the scheduler chooses which tokens to run, and the model runs one forward pass over them.

The question that follows is: why does one read take two engine steps? That’s the subtle part. The reason lies in how vLLM runs DiffusionGemma. DiffusionGemma uses a single Gemma 4 backbone in two modes. In encoder mode, it processes the prompt with causal attention and writes the prompt’s keys and values to the KV cache. In decoder mode, it denoises the canvas with bidirectional attention, reading the prompt from the KV cache. For an eligible read, vLLM runs the two modes in separate engine steps (Figure 3a):

1. Engine step 1, encoder mode. The prompt, or its final chunk for long prompts, runs through the model and fills the KV cache. In our example, the prompt holds the user’s conversation and the two questions. Once the prompt is complete, the model builds the canvas from the request’s answer template and returns it to the engine as the request’s draft tokens. The read produces no output in this step.
2. Engine step 2, decoder mode. The scheduler picks up the draft tokens and runs the canvas through one denoising step. The logprobs of the allowed options at every slot are returned, here for Topic and Urgency, and the request finishes.

The answer to our question of why one read takes two engine steps is in how the canvas reaches the model. vLLM serves the decoder mode through the data path it uses for speculative decoding: the canvas positions travel as draft tokens, scheduled after the prompt finishes prefilling.

In speculative decoding, this order is required: the draft tokens depend on what the model produced in the previous step. An eligible read has no such dependency. Its canvas is the answer template, which arrives with the request. The split comes from the data path, not from a dependency in the read itself.

Figure 3. How vLLM executes an eligible read. (a) In v0.32.0, the prompt runs in encoder mode in one engine step, and the canvas runs in decoder mode in the next. (b) With single_pass_reads, the canvas joins the final prompt chunk, and the read finishes in one engine step. Each engine step is a full forward pass through the model.

Free-form generation runs many denoising steps, and one extra engine step barely matters there. An eligible read, with its single denoising step, pays it in full. That second step processes only the canvas positions, tens of tokens against hundreds or thousands in the prompt, but it is still a full forward pass through all 30 layers of the model. This additional step was the overhead we set out to remove.

3.2 Executing eligible reads in one engine step

Once we knew why an eligible read took two engine steps, the next question followed naturally: if the canvas does not depend on anything the first step produces, why not run it in the same step as the prompt?

That is what we changed. An eligible read now runs its final prompt chunk and its canvas in the same engine step (Figure 3b). For the read in our example, the prompt and the Topic and Urgency slots go through the model in a single forward pass. We proposed this change upstream in vllm-project/vllm#61480 as an opt-in flag, single_pass_reads, off by default.

The change has three parts:

1. The canvas is ready when the request arrives. An eligible read’s canvas is its answer template, and it does not depend on the prompt. With the flag on, vLLM writes it into the request’s draft tokens at admission, instead of at the end of the prompt.

2. The scheduler appends the canvas to the final prompt chunk. This happens in the synchronous diffusion scheduler that vllm-project/vllm#59107 added. On the step that finishes a read’s prompt, the scheduler adds the read’s canvas, at its own width, to the tokens it schedules. Simplified from vllm-project/vllm#61480:

def _final_prefill_spec_tokens(self, request, num_computed_tokens,
                               num_new_tokens, token_budget):
    finishes_prompt = num_computed_tokens + num_new_tokens == request.num_tokens
    if not (self.single_pass_reads
            and is_single_pass_read(request.sampling_params)
            and finishes_prompt):
        return []
    width = diffusion_canvas_width(request, self.num_spec_tokens)
    if (num_new_tokens + width > token_budget
            or request.num_tokens + width > self.max_model_len):
        return []
    return [-1] * width

An empty list keeps the original path. Otherwise, the read’s canvas positions join the step as placeholders.

3. The model keeps each part’s attention. In the fused step, the model splits the read into two attention rows. The prompt row stays causal and never sees the canvas. The canvas row stays bidirectional and sees the whole prompt. The CPU attention kernel handles both through the per-request causal mask that vllm-project/vllm#59992 made it read correctly.

In request terms, an eligible read sets diffusion_read_only and diffusion_max_steps=1, the default in vLLM’s structured decision example. Not every eligible read can take the fused path, though. The scheduler fuses a read only when the current step finishes its prompt, when the final prompt chunk plus the canvas fits in the step’s token budget, and when the full prompt plus the canvas fits within max_model_len. Otherwise, the read keeps the two-step path from Figure 3a, unchanged. vLLM also accepts the flag only on the CPU backend, and rejects it at startup with pipeline parallelism, KV connectors or KV offloading.

Turning the flag on does not change the computation a read performs. The prompt is still causal, the canvas still sees the whole prompt, and the read still runs one denoising step. What changes is how the work is batched. In BF16, the same computation in a different batch shape can round differently, and outputs are close but not bit-identical. With the flag off, reads take the original path: in our JevBench runs, the flag-off build changed none of the 231 decisions. Section 6 gives a first look at what the flag-on differences mean for decisions, and Part 2 covers them in full.

The build we evaluated also includes two smaller changes to how Model Runner V2 computes prompt logprobs, vllm-project/vllm#61693 and vllm-project/vllm#61679. Our reads do not request prompt logprobs, and Section 4 shows that the speedup comes from single_pass_reads alone.

The key to CPU optimization was changing how vLLM executes eligible reads.

3.3 Why the optimization improves performance

One neat advantage of reducing an eligible read to a single engine step is that the claim is easy to verify: run the same reads with single_pass_reads off and on, and count the engine steps. For the reads, we used JevBench, a public benchmark for Jev-class decision models. Each of its items gives the model a piece of state and a bounded set of options, and expects a probability for every option.

We ran its 231 public items serially, one read at a time, on a c3-standard-44 VM. There, unlike on our Vertex AI endpoints, we could count engine steps directly. The baseline and our CPU optimization build with the flag off both took 462 engine steps for the 231 reads. With the flag on, they took 231: one forward pass per read instead of two. We also ran the same items four at a time. The flag then cut the engine steps from 174 to 116, a third rather than half, because concurrent reads already shared some of their steps.

The time each removed step took is measurable too. In the A/B test we ran for vllm-project/vllm#61480 on the same VM, every single read saved between 0.12 and 0.17 s. That was 21 to 25% of a read with a 188-token prompt, and 12 to 13% with a 1,250-token prompt. The prompt grew almost sevenfold, but the saving barely moved. The time between steps was also small: bursts of four fused reads spent all but 4 to 9 ms of their wall time inside engine steps. The saving therefore comes from the forward pass that no longer runs, not from the engine’s bookkeeping between steps.

Why that forward pass costs so much is our inference, and it ties back to why we chose C3 and C4. Both use Intel Xeon processors with AMX, which runs the BF16 matrix multiplications at the heart of a transformer in hardware, and vLLM v0.32.0 uses it in its attention and MoE kernels. AMX makes the arithmetic of a small step cheap. What it does not shrink is the cost of reading the model’s weights from memory, which every forward pass pays however few tokens it processes. A CPU has far less memory bandwidth than a GPU, and for a step of only tens of canvas tokens, those reads likely dominate. Each canvas token also picks 8 of the 128 experts in every MoE layer, and even a small canvas can touch a large share of them. In the fused step, the canvas rides along on a forward pass that reads the weights anyway.

The explanation does make two predictions. The gain should be largest for a single request, where nothing else shares the removed step. It should also shrink as concurrency rises, because more of a read’s canvas steps share their forward pass with other requests’ work, as the concurrent JevBench run already suggests. Section 4 tests both, on C4 and C3.

4. Benchmarking performance on C4 and C3

Section 3 left us with two predictions: the gain should be largest for a single request, and it should shrink as more requests share each engine step. To test them, we compared two builds:

• The baseline: vLLM v0.32.0, our baseline, which runs every eligible read in two engine steps.
• Our CPU optimization: vLLM v0.32.0 plus our three pull requests, with single_pass_reads on.

We ran both builds on the two CPU machines from Section 2.2:

• C4 (c4-standard-24, 24 vCPUs): our primary configuration, since it is the machine we want to deploy.
• C3 (c3-highcpu-44, 44 vCPUs): nearly twice the vCPUs, to check that the result holds on a larger machine. It also runs with our CPU optimization build with the flag off, which isolates the effect of single_pass_reads.

Each build ran on its own Vertex AI dedicated endpoint in the us-east4 region, with one replica. Before sending any request, we checked that every endpoint served the expected image, machine type and diffusion configuration.

The workload is 24 synthetic conversations, asked through four prompt formats: an original and a compact version of our own format, and two Jev-style formats, one with five choices and one with conditional questions. Each format makes 90 decisions about the conversations. Every read is constrained to its allowed options, with one sample, one denoising step and seed 42. Reads use canvases of 32 tokens, or 16 for the compact format, narrower than the served canvas thanks to vllm-project/vllm#59107. All of the more than 90,000 requests completed successfully.

4.1 Latency at concurrency 1

When we compared the two builds, the optimization build cut per-request latency by 18.8% on C4, with a 95% confidence interval of 18.5% to 19.0%, and by 21.2% on C3 (20.9% to 21.5%). In practice, a typical decision on C4 now comes back in about 540 ms instead of 660 ms. The gain held for every prompt format and in every pass, between 17.3% and 20.6% per pass on C4 and between 21.0% and 21.4% on C3 (Figure 4a, Table 1).

This matters for the requirement we set in Section 2.2: a single request in under 1 second at the 95th percentile. The baseline missed it for the format with conditional questions, at 1,059 ms on C4 and 1,130 ms on C3. With the optimization, every format met it on both machines, with a p95 between 645 and 813 ms on C4 and between 700 and 889 ms on C3. Across all formats on C4, requests slower than 1 second fell from 80 of 4,320 to 9 (Figure 4b, Table 1).

Figure 4. Per-request latency at concurrency 1. (a) Change relative to the baseline, with 95% confidence intervals. (b) Latency of every request, with whiskers at the 5th and 95th percentiles and the 1-second target dashed.*

Table 1. Latency results. Baseline vs CPU optimizarion comparison on C3 and C4 machines.

4.2 Throughput under concurrency

Throughput is the number of successful decisions per second at each concurrency level, averaged over the four passes. Under load, our CPU optimization build handled more decisions per second at every concurrency level, for every prompt format, on both machines (Figure 5). Averaged over the four formats, the gain on C4 was 15% at concurrency 2 and between 5.5% and 9% from concurrency 5 to 20. On C3, it was 18% at concurrency 2 and between 7% and 12% from concurrency 5 to 20.

The gain is largest at low concurrency and smaller as the machine approaches saturation. This is the second prediction from Section 3.3, and the latency gain shrinks the same way. On C4, per-request latency fell by 18.8% at concurrency 1, while the median latency was 13% to 14% lower at concurrency 2 and 4.5% to 7.3% lower at concurrency 20.

Beyond concurrency 15, throughput stops growing on both machines. The hardware matters here too. C4 handled slightly more decisions per second at concurrency 2 and 3, while C3, with nearly twice the vCPUs, pulled ahead from concurrency 5. Higher concurrency also increases the time each request waits: at concurrency 20 on C4, the median latency was 3.7 to 4.6 seconds with the optimization, against 3.9 to 4.8 seconds for the baseline.

Figure 5. Median latency against throughput for each prompt format, as concurrency rises from 2 (left) to 20 (right).

4.3 Attribution and run-to-run variation

Two controls tell us how far to trust these numbers. First, each run measured the baseline twice, on the same endpoint. The repeat differed from the first measurement by -0.2% on both machines (95% confidence interval: -0.3% to -0.2% on C4, -0.4% to +0.1% on C3), and by at most 1.8% in any single pass. Our CPU optimization build’s gain was larger than that difference in every pass, which was the condition we set in advance for reporting a speedup.

Second, the C3 run included the optimization build with single_pass_reads off. That build carries the two prompt-logprob changes, vllm-project/vllm#61693 and vllm-project/vllm#61679, but runs reads on the original two-step path. Its latency was 0.5% higher than the baseline (+0.2% to +0.7%). The two prompt-logprob changes do not explain the speedup, the single_pass_reads flag does.

Taken together, the results show that the optimization improved inference performance on both machines. C4 is particularly interesting because it achieved these gains on a machine with a substantially lower monthly cost than C3. However, performance alone does not establish which configuration offers the best deployment trade-off. To answer that question, we also need to consider cost.

5. Serving cost

Cost has two sides for a service like ours. The first is the price of keeping one replica running, which Vertex AI bills around the clock, whether requests arrive or not. The second is the cost of each decision when the replica is busy.

At low traffic, the first side dominates. With one always-on replica, the cost per 1,000 decisions is the monthly price divided by the decisions served that month (Figure 6a). At 100,000 decisions a month, that is $9.96 per 1,000 on C4, $16.74 on two L4s, $17.93 on C3 and $47.54 on the A100. The same C4 replica, fully busy, would cost $0.078 per 1,000 decisions. Even at a million decisions a month, idle time makes each decision about 13 times more expensive than at full load. For bursty traffic that fits on one replica, the cheapest machine to keep running is the cheapest machine to use, and that is C4.

Under load, the optimization lowers the cost of each decision, because the same machine completes more decisions per hour (Figure 6b). On C4, the cost per 1,000 decisions fell by 5% to 13% across concurrency levels 2 to 20, and on C3 by 7% to 15%. At concurrency 10, our CPU optimization on C4 cost $0.088 per 1,000 decisions, 39% less than the optimization build on C3.

The picture changes when traffic keeps a replica busy. Under sustained load, our earlier GPU runs cost less per decision than either CPU: at concurrency 10, $0.027 per 1,000 decisions on two L4s and $0.046 on the A100, against $0.088 on C4 with our CPU optimization. Two L4s were cheaper at every concurrency level we measured, and the A100 from concurrency 3 upward. We left these runs out of Figure 6b because they are not like-for-like: two L4s can only hold the model in FP8, and both GPU runs used an earlier vLLM nightly, a 1,024-token context and unconstrained reads. When traffic grows beyond what one C4 replica can serve, a GPU is likely the better choice. Until then, the monthly price decides, and C4 costs a fifth of the A100 that BF16 needs.

Figure 6. Serving cost. (a) One always-on replica: the monthly price divided by the decisions served, with the CPU optimization C4 build at full load dashed. (b) Cost per 1,000 decisions on CPU under load.

6. Faster and cheaper, but are the decisions the same?

For a decision model, speed and cost are only part of the evaluation. A faster build is only useful if it keeps making the decisions the application expects.

We checked this on the 231 public JevBench items. With single_pass_reads off, our CPU optimization build changed none of the 231 decisions. With the flag on, the change in accuracy was not statistically significant on either machine, but individual decisions did change: 2 of 231 on C4 and 4 on C3. That would be easy to blame on the optimization, until we compared the two machines. Moving the same vLLM baseline from C3 to C4, with no optimization at all, changed 10.

So where do these changes come from, and which ones should worry us? In Part 2, we take a closer look at decision agreement: why aggregate accuracy can hide individual changes, how batching, hardware and numerical precision each move decisions, what happened when we repeated the reads in FP32, and how we decided whether the CPU optimization build was ready to deploy.

Try it out!

single_pass_reads is available in vllm-project/vllm#61480 on a CPU build of vLLM. With that build, the serving settings from this post are:

vllm serve google/diffusiongemma-26B-A4B-it \
    --dtype bfloat16 --max-model-len 4096 \
    --max-num-seqs 16 --max-num-batched-tokens 8192 \
    --max-logprobs 32 --enable-prefix-caching \
    --diffusion-config '{"canvas_length": 64, "single_pass_reads": true}'

Structured reads come from vLLM’s example decision server, which sends one-step reads by default.

Acknowledgements

Thanks to all the vLLM reviewers that helped us to land our changes!

The three CPU fixes are merged (vllm-project/vllm#59107, vllm-project/vllm#59992, vllm-project/vllm#59829), and the optimization is under review (vllm-project/vllm#61480, vllm-project/vllm#61679, vllm-project/vllm#61693).

Next, in Part 2: how hardware, batching and numerical precision change a decision model’s answers, and how we measured it.