Choosing an Agentic Model for a 2× DGX Spark Cluster (September 2026)

Summary. In September 2026 there are three models on Hugging Face you would want to put on a pair of DGX Sparks behind Claude Code: DeepSeek-V4.1-Flash at 552B, GLM-5.3-Flash at 320B and Qwen3.8-Flash-Next at 125B. The first leads every benchmark but does not fit in 243 GiB even at 4 bits. The second takes 198 GB and 15 minutes to load. The third fits on a single Spark — apart from the 2 GB it is short. Below: why on this cluster the choice is made by memory arithmetic rather than a benchmark table, why 6 billion active parameters do not give you 90 tok/s, and why for Claude Code what decides it is not tok/s but a single vLLM flag that the fastest config turns off.

A month and a half ago I deployed deepseek-ai/DeepSeek-V4-Flash-0731 tensor-parallel across 2 nodes. The measured speed came out around 57 tok/s, and for a while that was the best option available. Then 3 new models landed within 3 weeks, each of which looks better on paper than what I am running, and the question of what to run came up again. The method is the same as last time: do the math before downloading, not after.

First, the rig itself, because all the arithmetic below starts from its numbers.

The rig: two Sparks with unified memory

Inside each Spark is an NVIDIA GB10 Grace Blackwell superchip: 20 Arm v9.2-A cores (ten Cortex-X925 plus ten Cortex-A725), a Blackwell GPU with compute capability 12.1, and a thermal envelope of about 140 W.

The defining property of this hardware is not the teraflops but the unified memory. The 128 GB of LPDDR5x-9400 in each Spark is system memory and video memory at once; there is no separate VRAM at all. So the familiar config of “keep 88 GB in VRAM, offload the rest to the host” does not work here: there is nowhere to offload to, it is the same pool, and such “offloading” just takes memory away from the model itself. Of the 128 GB, 121.6 GiB is usable, 243 GiB for the pair, and everything in this post is counted against that number.

The nodes are connected directly, with no switch: ConnectX-7, a QSFP112 cable port to port. My measurements on this link, taken in July: about 17 Gbit/s with a single TCP stream, about 107 Gbit/s with 8 parallel streams, and about 19 GB/s for NCCL over RoCE when both of the card’s PCIe interfaces are in use. The ConnectX-7 is attached to the system through two separate PCIe Gen5 x4 interfaces and shows up as two adapters; through just one of them, bandwidth tops out at 13.9 GB/s. There is also ordinary gigabit Ethernet between the boxes — 943 Mbit/s in iperf3 — used only for management. The weights of large models live on the nodes themselves.

The model is wired into Claude Code through free-claude-code, a gateway that accepts requests in Anthropic API format and translates them to a local vLLM. Claude Code talks to the gateway, and in place of every Claude model — Opus, Sonnet, Haiku, Fable — there is one local one. While I was comparing candidates, it turned out the gateway had been switched off for several days without my noticing: Claude Code had been running through the cloud the whole time.

The numbers for the current stack that I rely on below: the DeepSeek-V4-Flash weights take 155 GiB, split evenly across the two nodes, with the vLLM API server and scheduler on the first node. The model gives 57.54 ± 5.61 tok/s single-stream (median across runs — run-to-run spread on this cluster is about 20%, which is why the number comes with a sigma rather than as an exact figure), 1600–2000 tok/s prefill, a KV pool of about 699 thousand tokens with a 262,144 window, and a cold load of about 5 minutes.

The calculator beats the benchmark

DeepSeek-V4.1-Flash came out on September 10. According to its developer it beats my current stack on every line, and my first reflex was to download it. My second was to open config.json.

The model has 552 billion parameters in the transformer and another 196 billion in Engram — an n-gram table that is read on every token. At 4 bits the transformer alone weighs about 276 GB. A pair of DGX Sparks has 2 × 121.6 GiB = 243 GiB between them, and that is before the KV cache. The model does not fit at any quantization at which it still deserves to be called a model: it needs 3 or 4 Sparks, and I have two (if you have a four-Spark cluster, I have come across such setups online).

At this point the usual thought is “what about NVFP4?” It occurred to me too, and it is wrong: DeepSeek’s experts are already stored in MXFP4, 4 bits per weight. NVFP4 is a different scaling scheme at the same 4 bits, not a halving.

So the best model of the month is out in 5 minutes without a single byte downloaded. On to the other candidates.

Why on two Sparks memory decides, not the interconnect

Once more than one node is involved, the worry is that the bottleneck will be the inter-node link, which carries an all-reduce collective on every token. But look closer: with tensor parallelism on an MoE model, what crosses the link on each token is the hidden-state vector — a few megabytes at most — while the interface moves 18–19 GB/s. The link is loaded to a fraction of a percent.

In practice everything comes down to memory.

First, capacity. Tensor parallelism is a way to run one model across several GPUs in which every weight matrix is cut into pieces, and each node stores and computes only its own piece; the partial results are then combined across nodes by that same all-reduce. So everything in the weights has to fit in 243 GiB, split in half between the nodes. For GLM’s 198 GB (≈184 GiB) that is 92 GiB of weights per node out of 121.6 available. At gpu-memory-utilization 0.85, that leaves on the order of 11 GiB per node for KV cache and activations — which is why the config pins the KV cache size with --kv-cache-memory 6 GiB instead of leaving it to automatic sizing.

Second, memory bandwidth. Decoding a token means reading all the active weights from memory. GB10’s rated bandwidth is 273 GB/s (the vendor’s figure; I have not measured it). GLM has 18B active parameters; at 4 bits that is about 9 GB per token, split across two nodes — 4.5 GB each. Divide: a ceiling of about 60 tok/s single-stream without speculation. It is a rough estimate, but it — not the benchmarks — is what tells you this model will not go faster than 60 tok/s, and that any “100 tok/s” on someone’s model card should be read as the result of speculative decoding, not of the model itself.

By the same arithmetic, Qwen3.8-Flash-Next with 6B active parameters should be 3 times faster. It is not, and that is the second half of this post.

GLM-5.3-Flash: 198 GB and a 15-minute load

GLM-5.3-Flash came out on August 26: 320B parameters, 18B active, MIT licence, image input, a million tokens of context natively. z.ai has a table where, inside Claude Code, it scores 29.0 on Code Bench against 29.5 for Opus 4.8. It is the developer’s own number, but one detail in it matters more than the number: the benchmark was run inside Claude Code 2.1.207 — exactly the scenario I need the cluster for.

There are two NVFP4 builds of it on HF, and the first trap is picking the wrong one. NVIDIA’s ModelOpt build corrupts tool-call tokens: instead of an argument you get a mangled string, and Claude Code silently loses the tool_call (vLLM #54150). The working build is RedHatAI/GLM-5.3-Flash-NVFP4 in compressed-tensors W4A4 format: 10 shards of 20 GB plus 7.6 GB of MTP head, about 198 GB in total.

tonyd2wild’s config for a pair of DGX Sparks is complete end to end: a custom vLLM image for sm121, a 2.3 GB DFlash2 draft model for speculative decoding with k = 7, a patch to the sparse-attention indexer that otherwise fails in top-k on SM121, --block-size 2304, fp8 KV cache, --enforce-eager. Per its README: 46.9 tok/s single-stream on coding tasks, 56 aggregate at 5 concurrent requests, a KV pool of 581–678 thousand tokens, and a cold load of about 15 minutes. Against my estimated ceiling of 60 that is 78% — with speculation, which is plausible.

A couple of important lines from the README:

• Swap must be enabled, with swappiness=0, or loading 198 GB through system memory does not survive the peak. I have swap disabled on both nodes, and that is no accident. With swap off, any attempt to go past 128 GB ends immediately in an explicit CUDA OOM. With swap on, the machine sinks into paging for hours: technically alive, unable to do any work.
• Before starting, echo 3 > drop_caches, otherwise the page cache left by the previous model holds memory the loader needs. A ritual that looks like superstition until you watch an OOM at gigabyte 190.

Qwen3.8-Flash-Next: 133 GB and a Spark it does not quite fit on

Qwen3.8-Flash-Next is a preview of Qwen’s fourth generation: 125B parameters, 6B active, plus 51B in the PLE n-gram table and 4B in the MTP head; 262K context natively, a million via YaRN, image input. The quantized build nvidia/Qwen3.8-Flash-Next-NVFP4 weighs about 133 GB, and NVIDIA’s model card says NVFP4 is indistinguishable from FP8 in quality here.

The first thing that jumps out: 133 GB is almost exactly one Spark. Almost: 121.6 GiB is 130.5 GB, and the model is 2.5 GB short. A TP1 config on a single Spark exists anyway — because the PLE table is loaded in parts rather than held in memory whole — and gives 43.9 tok/s against 53.7 on two. It is the only one of the three candidates that can run on one node at all: no inter-node link, no two-node launch, no all-reduce on every token. Both of my Sparks will go to the model regardless, so I would run TP2 — but the fact that a single-box option exists is interesting in itself.

Second, the TP2 numbers. The same author gets 53.7 tok/s median single-stream on two Sparks, 97.9 aggregate across 6 streams, and a KV pool of 1.97 million tokens. That last number is 3 times GLM’s, and it follows directly from 6B active parameters: the model itself is light, so the memory goes to the cache.

Third, one line in the config that for an agentic workload matters more than every number above: --no-enable-prefix-caching. Prefix caching for Qwen3.8-Flash-Next is force-disabled in vLLM, because with it on the model returns wrong answers (vLLM #54173).

6 billion active parameters do not give you 90 tok/s

Back to the speed ceiling estimated from memory bandwidth — the one that gave about 60 tok/s for GLM. GLM has 18B active parameters, Qwen 6B. If GLM’s ceiling is about 60 tok/s, Qwen’s should be about 180. The config delivers 53.7. That is 30% of the ceiling against 78% for GLM, and the gap is too large to write off as noise.

I have two explanations, and both are inferred, not measured:

• The PLE table. 51 billion parameters that do not count as “active” but are read on every token. Exactly how many bytes depends on the n-gram and on how the gather is performed; the README says nothing about it.
• CUDA kernel launch overhead. With 6B active parameters the matrix math itself is so short that the share taken by kernel launches, all-reduce and the scheduler stops being negligible. tonyd2wild’s Qwen config shows the author fighting exactly this: CUDA graph capture, which removes the kernel-launch cost, is enabled for decode only, and speculative decoding through the MTP head is set up to predict 3 tokens per model pass instead of 1 — paying the overhead once per 3 tokens.

The practical takeaway: “active parameters” is a poor predictor of speed once there are fewer than 10 billion of them. Between 18B and 6B active there is almost no difference in speed: 47 versus 54 tok/s. The difference is elsewhere — in the size of the KV pool, and in the fact that one of the two models fits on a single Spark.

Where the estimate diverges from measurement — and why I am publishing it as is

The bandwidth-based ceiling is a model with two assumptions:

• 273 GB/s is a rated figure; real bandwidth under vLLM is lower, and by how much I have not measured.
• I assume only the active weights are read per token; in reality the KV cache is read too, plus the sparse-attention indexer for GLM and the PLE for Qwen.

Both corrections pull the ceiling down, so GLM’s 78% is most likely closer to 90% of the real ceiling, and Qwen’s 30% closer to 40%.

I am leaving the estimate in this form because it did its job: it ruled out DeepSeek-V4.1 and explained why two models with a threefold difference in active parameters have the same speed. I am not going to stretch it into a “measurement” — the measurement comes in the next post, once one of the two models is running on this pair.

Summary table of what is known today. Italics are my numbers; plain text is other people’s.

The consequence: for Claude Code, prefix cache decides, not tok/s

Claude Code is not a chat. Every agent request sends the server the same system prompt with descriptions of ~150 tools, the whole conversation history, and at the end the new turn. For the server this means 95% of the prompt matches the previous request, and the entire point of prefix caching is not to compute it again.

I have my own numbers on this, taken in August from the gateway logs on the current stack. Requests where more than 80% of the prompt hit the prefix cache got their first token in 4.3 s. Requests with a hit rate below 20%, at the same prompt size, waited 73.2 s. A 17× difference with the same model, the same hardware and the same length — and it is explained entirely by whether the prefix was recomputed.

Now set --no-enable-prefix-caching in the Qwen config against that. At 1600–2000 tok/s prefill (my figure for DeepSeek; Qwen with 6B active parameters should prefill faster, but the order of magnitude is the same), on every agent turn with 100 thousand tokens of context the model will recompute those 100 thousand from scratch. That is a minute of waiting before every reply, and no 53.7 tok/s of decode makes up for it: the agent generates ~200 tokens per turn and reads 100 thousand.

So my candidate ranking came out in an order a benchmark table would never have produced. GLM-5.3-Flash goes first, because it works exactly like the current stack and has already been compared against Opus inside Claude Code. Qwen3.8-Flash-Next goes second, with one price of admission: if its prefix caching can be turned on without corrupting answers, it wins 3× on pool size and frees up the second node. If not, it stays a chat model rather than an agent model, however fast it is.

Takeaways

1. The first filter for a model on two DGX Sparks is not a benchmark but the total weight size against 243 GiB. A model that fails this filter is not worth reading about: DeepSeek-V4.1-Flash at 552B is ruled out in 5 minutes.

2. NVFP4 does not shrink a model whose experts are already MXFP4. If the source model card says “FP4”, the 4-bit derivative will be the same size.

3. The decode ceiling follows from memory bandwidth and active parameters, and it is worth computing before reading anyone else’s thousands of tok/s: for 18B active on GB10 it is about 60 single-stream; anything above that is speculative decoding.

4. Below 10B active parameters, their count stops predicting speed: Qwen’s 6B and GLM’s 18B give 54 and 47 tok/s. Look for what else is read per token — PLE, indexers, KV.

5. For an agentic workload, look not at tok/s but at prefix caching: on my gateway the difference between a hit and a miss is 4.3 versus 73.2 s to first token. A config with prefix caching disabled is not suitable for Claude Code, however fast the model is.

6. A config that requires swap changes the memory policy of the whole machine, not one line in a config file. I keep swap off for an honest OOM instead of silent paging, and turning it on is a decision made before installation, not after the first hang at gigabyte 190.

7. Before choosing a model for a service, check that the service is on.

Sources: RedHatAI/GLM-5.3-Flash-NVFP4, nvidia/Qwen3.8-Flash-Next-NVFP4, tonyd2wild’s configs for GLM-5.3-Flash on two Sparks and Qwen3.8-Flash-Next, vLLM issues #54150 and #54173, z.ai blog on GLM-5.3-Flash.