125 Billion Parameters on Two GPUs of 8 GB and 6 GB: Qwen3.8-Flash-Next with Strata

In October I tried Strata for the first time, didn’t get good results and uninstalled it. I wrote about it in my article on the expert cache. Now I have installed it again to run Qwen3.8-Flash-Next with 8 GB VRAM on one card and 6 GB on another.

One detail made me doubt: the example machines the project publishes start at cards with 12 GB of VRAM (an RTX 3060, an RTX 5070). I have an 8 GB RTX 2080 SUPER and a 6 GB RTX 2060, two different generations, 14 GB between the two. Can you run a 125-billion-parameter model on a dual GPU setup with so little VRAM? Spoiler: yes, at about 34 tokens per second.

Qwen3.8-Flash-Next is still, today, the best thing I have seen for running locally, and it had been nagging me. This time with a clear goal: not a demo, but the engine of a real agent inside my own application.

This article covers three things: how Strata works on my machine and which settings made the difference, how I integrated it into Agente Jiménez with a new agent, Adolfo-Strata, and what it can do, with a real example: I asked it to program a 2D video game in HTML and I checked the result.

This article follows up on my expert cache test, where I explain what an MoE model is and why a huge model can fit on a small GPU. The hardware and the way of measuring are those of my first article on autonomous coding. I don’t repeat that context: if you need it, start there.

Results summary

Question	Measured answer
Does Qwen3.8-Flash-Next work on an RTX 2080 SUPER (8 GB) + an RTX 2060 (6 GB)?	Yes. Between 33 and 34 tokens per second decoding, with the final configuration
Which setting gave almost all the improvement?	STRATA_ROUTE_PRIOR=1.5: from +21% on prose to +63% decoding after a long prompt
Does it have a cost?	Yes: it slipped Chinese characters into the middle of Spanish words. I fixed it by banning those tokens in every request, with no speed loss
And MTP, speculative decoding?	I got it to start, but it is no faster and it takes the engine down with a long prompt. Dropped
Does it work as the engine of an agent?	Yes. Adolfo-Strata runs commands and searches the web. I asked it for a 2D video game and it meets the 11 requirements of the brief
And with Strata’s official benchmark?	More modest: 27.5 tok/s in chat, 18 tok/s as an agent and 36.5 tok/s reading a 16K prompt, which takes about 7 minutes to produce the first token
Where does the effort show?	When reading long prompts. The confirmation turn of the game took almost 7 minutes

Strata 0.2.9 · Qwen3.8-Flash-Next Gyro-S · RTX 2080 SUPER + RTX 2060 · Ryzen 7 5800X · 32 GB of RAM · two runs per test, one for the long prompt.

Qwen3.8-Flash-Next with 8 GB VRAM? And with two GPUs of 8 and 6 GB?

With two GPUs of 8 GB + 6 GB, yes, as long as you have RAM and a fast NVMe: I use 32 GB of RAM and a disk at ~2.5 GB/s. Strata splits the model’s layers between the two cards, even if they are different generations (an RTX 2080 SUPER and an RTX 2060), and leaves the remaining experts in RAM and on disk. With a single 8 GB card I did not measure it in this test, so I’m not promising anything.

The price: reading long prompts is slow. Everything is measured below, including with the project’s official benchmark.

Before we start: four simple ideas

• Strata. A local server that runs huge MoE models on small cards. It splits the model’s layers between the GPUs and leaves the experts that don’t fit in RAM and on the NVMe, fetching them when needed. It speaks the OpenAI API, so any program that can talk to OpenAI can use it.
• Gyro-S. The heavily compressed version of Qwen3.8-Flash-Next that Strata uses. It weighs 58.5 GB, about 2.6 bits per weight on average. It is Strata’s own format: regular llama.cpp can’t load it.
• MTP (multi-token prediction). The model proposes several next words at once and checks whether it got them right. If it did, it moves forward several at a time. It is a speed technique.
• Token ban (logit_bias). An API option to tell the model “this word, don’t even think about it”. Further down I use it for a very specific purpose.

My machine: two GPUs of 8 GB and 6 GB (RTX 2080 SUPER + RTX 2060)

Part	What I have
Graphics cards	RTX 2080 SUPER (8 GB, it also drives the desktop and uses ~1.5 GB) + RTX 2060 (6 GB, on a PCIe x4 slot)
CPU	Ryzen 7 5800X. It has AVX2 but no AVX-512
Memory and disk	32 GB of RAM · NVMe at about 2.5 GB/s
System	Garuda Linux
Strata	App 0.2.9 and engine 0.1.55 (CUDA 13), installed in isolation with its own data directory
Model	Qwen3.8-Flash-Next Gyro-S: 58.5 GB, plus the expert pack and the MTP layer. About 90 GB on disk in total
Context	32,768 tokens

Strata detects that my RAM isn’t enough to keep everything in memory and switches on its “low-RAM mode”: the experts are read from the NVMe through the system page cache. That is why free RAM matters so much, as you will see in the results.

The model, in numbers

To know what I was working with, I read the header of the model file and added up the size of every tensor. These are not official Qwen figures, they are what comes out of my file.

Part	Parameters
Experts: 48 layers of 512, of which it uses 10 per token	120.8 B
Per-layer embedding table (only a few rows are looked up per token)	51.2 B
Attention, SSM layers, shared expert, input and output	4.9 B
Total	176.9 B

Without that embedding table, which is not multiplied per token, about 126 B remain. That is the “125B” figure I quoted in the previous article. About 6.6 B work on each token, which explains why it can go so fast.

How I tuned it

I started from the first configuration that started for me and kept comparing. This is what was left:

Setting	What it does	Effect on my machine
STRATA_ROUTE_PRIOR=1.5	Prefers the experts that are already on the GPU. It has been off by default since 0.2.8 because it caused repeated text in long sessions	Almost all of the speed gain
8-bit KV (int8)	Stores the conversation memory with less precision so it takes up less space	Frees VRAM and leaves room for experts
--gpus 0,1	Splits the layers between the two cards. I tried the “helper” role for one of them and Gyro doesn’t support it: Strata ended up loading everything on the 2080	Really uses both GPUs
--vram-reserve-mib 400	Leaves a little VRAM free for the desktop	Avoids running out of memory with the screen in use
--mtp off	Turns off multi-token prediction	See below: it doesn’t pay off

I also ran Strata’s automatic tuner (strata tune auto). It concluded there was nothing faster than my configuration.

Results

Test	Before	After	Change
Spanish prose	28.3 tok/s	34.2 tok/s	+21%
Python code	26.1 tok/s	33.7 tok/s	+29%
Decode after a 3,200-token prompt	20.3 tok/s	33.1 tok/s	+63%
Reading that prompt, cold	51 tok/s	63 tok/s	+23%

With the page cache already warm, that same prompt reads at 106-165 tok/s. “Before” is the first configuration that started for me (q4_0 KV, no MTP); “after” is the final one.

In plain words: what costs time is not writing, it’s reading. Decoding runs at about 34 tok/s, a comfortable speed. Reading a long prompt depends on the experts already being in RAM, and with 32 GB there is little margin. The official benchmark confirms it: 16K of context takes about 7 minutes to produce the first token.

The price of going faster: Chinese characters

With STRATA_ROUTE_PRIOR switched on, a Chinese word appeared in the middle of a Spanish sentence: “sea吸入ado”. It came out the same in both runs of the prose test (at temperature 0 the result repeats). Without the option, in the reference measurements, it did not appear.

I didn’t want to give up the improvement, so I fixed it on the client side. In every request I send a logit_bias of −100 for the 66,394 Chinese, Japanese and Korean tokens of the model’s vocabulary (I took them from the pack’s own vocab.json). The model simply can’t pick them. It costs no measurable speed.

MTP: I got it working and I dropped it

I was curious about MTP. With 8 GB, the draft layer (~825 MiB) doesn’t fit on the 2080 next to the desktop, so it had to be moved to the 2060. This is what it took:

• --pipeline-windows 2, so that the draft layer can be moved.
• The variable STRATA_MTP_STAGE0=1. The release notes say that with pipeline windows the layer goes to the first card, but on my machine, with NVIDIA and two GPUs, that didn’t happen. The engine explains it in its log: you have to ask for it with that variable.
• Setting the layer split by hand (--layer-split 7), limiting the expert cache (--expert-cache 150) and reducing the read block (--prefill 512).

It started and accepted half of the drafts. But it was not faster: 23-26 tok/s, the same or worse than without it. And with a 3,200-token prompt, the engine crashed for lack of VRAM. When the experts come from RAM or disk, the limit is fetching them, not the number of passes. It matches what other users with 6 GB cards reported in the project’s issues #474 and #916.

Strata’s official benchmark

The figures above are my own measurements. I wanted to check them against the project’s official quick benchmark (tools/strata_bench.py), which uses fresh prompts on every run, so the cache doesn’t help. These are medians of 3 runs:

Test	Median	Range
Chat, 3 short questions (100 tokens): decode	27.5 tok/s	22.7 – 28.6
Chat: first token	1.05 s	0.93 – 1.26
4K prompt: read	51.4 tok/s	35.7 – 58.6
4K prompt: first token	78 s	69 – 113
16K prompt: read	36.5 tok/s	32.7 – 46.9
16K prompt: first token	440 s (7.3 min)	343 – 492
Agent, 6-turn session with tools: decode	18 tok/s	11 – 26
Agent: first token per turn	85 s	50 – 151
Agent: conversation cache hits	71%	70.7 – 70.8

They are lower than my measurements (34 tok/s on prose against 27.5 in chat). I haven’t investigated this in depth. The tests are not the same: the official one generates only 100 tokens per question, with fresh prompts every time, and mine 400 tokens. And the range is wide: the first chat run was the slowest and the following ones climbed to 28.6, with the experts increasingly present in RAM.

What it does make clear is where the limit is: reading. With 16K of context, the first token takes 6 to 8 minutes. As an agent, with a session that grows turn after turn, each turn takes more than a minute to start answering.

One warning: the benchmark’s automatic report detected only one GPU (the RTX 2060) in its machine card, so that card is not reliable. The measurement was made with both cards, like the rest of the article.

How I integrated it into Agente Jiménez

Agente Jiménez is my application of local agents. Each agent has its own model, its own tools and its own way of starting, and doesn’t touch the others. I added a new one, Adolfo-Strata, which talks to Strata.

I wanted a direct flow, like my agent Adolfo-Assistant, without OpenCode in between. In the previous article, OpenCode’s first request already took 34,005 tokens, and with a model whose weakest point is reading, I’d rather send it little.

• My own tool loop. Adolfo-Strata can run shell commands and search the web. It asks for confirmation before anything delicate, like the rest of my agents.
• A “Start Strata” / “Stop Strata” button in the top bar, with its status (stopped, starting, ready). No agent loads models on its own: only the button does.
• Memory management. Before starting, if LM Studio is open with a model loaded, it unloads it to free the VRAM. If it’s closed, it doesn’t touch it. Agente Jiménez’s resource guard recognises the Strata process.
• The Chinese-token ban I just described, sent with every request.
• Locked model selector: this agent only uses the model Strata has loaded, which is what is shown.

Starting Strata from scratch takes about 13-15 seconds with the model already on disk. A simple request that runs one command took about 27 seconds end to end.

Three traps that showed up in use

The first two attempts at the video game test (next section) failed. What I found deserves its own section:

• Empty answer. The model wrote the whole file inside the tool call, but with malformed escape sequences in the JSON. My code couldn’t read it and discarded it, so nothing happened. Now, for this agent, it tries to repair the escapes. If there is still no command, it asks the model to repeat it. And the final answer is never empty.
• The safety filter asked for confirmation for no reason. My command filter saw the word “su” inside the game’s HTML and took it for a command. Now it ignores the content of heredocs that only write a file. If that content is piped to bash or sh, or written to /etc, it still asks for confirmation.
• A short output limit. A whole game takes thousands of tokens. I raised this agent’s per-turn maximum.

The test: have it program a video game

I wanted a task that could be checked for real, not by impression. I chose a classic space shooter in a single HTML file, for several reasons:

• Everybody knows how it should look, so it can be judged by eye with no explanation.
• It is self-contained: no libraries or images. Everything is drawn with canvas shapes.
• It forces you to coordinate different things at once: game loop, collisions, state, score, levels and persistence.
• It can be automated: it is opened in a headless browser, played with the keyboard and the JavaScript errors are logged.

The brief had 11 specific requirements: an 800×600 canvas; a start screen with “INVASORES”; a ship that moves with the arrow keys and shoots; a 5×11 formation of invaders with three designs and different points; invaders that fire from the bottom row; four destructible shields; a scoreboard with a high score saved in localStorage and three lives; a faster next level; game over; pause with the P key; and all the texts in Spanish from Spain. That is why the game’s on-screen texts are in Spanish.

Try it yourself

This is the file exactly as Adolfo-Strata wrote it, without changing a line. Click inside the game to give it focus and press Enter to start. Move with the arrow keys or with A and D, shoot with the space bar and pause with P. You need a keyboard.

Here the high score only lasts until you close the page. You know what awaits you: it’s hard.

What it did

What was checked	Result
Generated file	A single HTML file of about 370 lines (10.8 KB), with CSS and JavaScript inside, created by Adolfo-Strata with a single command
The 11 requirements	Met. In the automatic check: score (it reached 150), saved high score, pause, move to level 2, restart after defeat and Spanish texts
JavaScript errors	None
Generation time	239 seconds, about 4 minutes
Confirmation turn	407 seconds: Strata had to re-read about 10,000 tokens of the conversation

What it didn’t do well

• It is far too hard. In my automatic check all three lives were lost in about 23 seconds.
• The shields have an odd shape, with the blocks in disorder. They work, but they don’t look like shields.
• The model tested nothing of what it wrote: it only checked that the file existed and counted its lines, as the brief asked. I did the real tests myself.

In plain words: you ask a model compressed to about 2.6 bits, on 14 GB of GPU, for a whole game and it writes it first time, with no errors and meeting everything you asked. It doesn’t hand it over balanced, but it works.

Is it worth it?

If you have…	My recommendation
Two small GPUs (8 + 6 GB), 32 GB of RAM and a fast NVMe	Yes, worth trying: decoding at about 34 tok/s with a model this size is a good speed
Tasks with huge context or long conversations	Careful: reading the prompt is the slow part. With 16K of context the first token takes about 7 minutes in the official benchmark. The more free RAM, the better
In a hurry	A 26-35B MoE that fits entirely, like Cyber-Tiel, is faster. Which model writes better is another question: I haven’t compared them on quality
Want to tinker	Try STRATA_ROUTE_PRIOR=1.5, but watch for strange characters, or ban the tokens you don’t want to see
A GPU under 12 GB and a wish for MTP	Work out first whether the draft layer fits. On mine it isn’t faster

What I did not measure

• The speeds come from few runs (two per test, one for the long prompt). Take them as indicative.
• The official benchmark I did run is the quick one (strata_bench.py), not the standard one (strata bench run). The results are not comparable with the project’s standard suite.
• The video game is a single task, generated once, and I judged its quality myself, with an automatic check and a few games by hand.
• Gyro-S is heavily compressed: it does well in Spanish and simple code, but it is not the original model.
• I haven’t compared Flash-Next’s quality with Cyber-Tiel’s on the same tasks.
• Strata and Gyro-S are not my material: they are another project’s tool and these figures are only my measurements.

What I take away

With Strata 0.2.9, Qwen3.8-Flash-Next stops being a demo and becomes something you can use as the engine of an agent, with a model of about 125 billion parameters (177 billion if you count the embedding table) split across two modest GPUs.

It is not a fast reader, nor a perfect programmer: it writes you a game first time, but far too hard. Still, it is there, it works and it can be tuned.

If you want to understand the pieces better, what an MoE is and why llama.cpp’s expert cache didn’t work for me, read the previous test.

If it helped, buy me a coffee

These tests take hours of machine time and tens of gigabytes of models that I download, measure and delete. The only thing I can’t generate locally is coffee.

If this article saved you from testing it yourself, you can buy me one.

Keep exploring local AI

• Expert cache in llama.cpp, the test where I uninstalled Strata and which this article follows up on.
• Autonomous coding with local AI: is 8 GB of VRAM enough to program with 30B models?
• Inside local AI: what really lies behind an artificial intelligence agent.
• What is the best local AI model? 13 models across 1,080 real-world tests.
• Local AI music generation on 8 GB, another test of how far an 8 GB card goes.

Frequently asked questions

Can I run Qwen3.8-Flash-Next with an 8 GB GPU and a 6 GB one?

Yes, with Strata 0.2.9, the Gyro-S version, 32 GB of RAM and an NVMe. I get about 34 tok/s decoding. The slow part is reading long prompts.

Can I use dual GPUs with different cards, like an RTX 2080 SUPER and an RTX 2060?

Yes. With --gpus 0,1 Strata splits the layers between the two, even with different VRAM (8 GB and 6 GB) and a different generation. The 2060 sits on a PCIe x4 slot and that was not a problem. What it didn’t allow was giving one of them the “helper” role: with Gyro-S it ended up loading everything on the 2080.

How much RAM and disk does Qwen3.8-Flash-Next need on Strata?

In my case, 32 GB of RAM and about 90 GB of disk (the 58.5 GB model, the expert pack and the MTP layer). The more free RAM, the faster it reads prompts: with the cache already warm I went from 63 to 106-165 tok/s.

Does it work with a single 8 GB GPU?

I did not measure it in this test. The project’s examples start at 12 GB cards; I did it with two GPUs (8 GB + 6 GB).

What is STRATA_ROUTE_PRIOR?

A variable that makes Strata prefer the experts that are already on the GPU. It has been off by default since 0.2.8 because of a risk of repeated text in long sessions. You switch it on with STRATA_ROUTE_PRIOR=1.5 in the "env" block of the model’s configuration file.

Is MTP worth it with 8 GB of VRAM?

On my machine, no. You have to move the draft layer to the second GPU with a variable the engine only mentions in its log; it starts, accepts half of the drafts, is no faster, and crashes with long prompts.

Why doesn’t the agent use OpenCode?

I wanted a direct flow, like my other agents. My main reason is that OpenCode sends a very large prompt and reading is what this model finds hardest.

Why ban the Chinese tokens instead of not using the option?

Because the speed improvement was almost all of the gain and the ban costs nothing measurable. It is a client-side fix: it doesn’t change Strata.