Local AI Test Results
Real-world speed measurements from our two test workstations: a dual Radeon RX 7900 XTX build and an RTX 3080 Ti + RTX 3080 build. Every model runs on our own hardware, and the launchers, settings and raw results are published on GitHub.Real-world speeds from our AMD and NVIDIA test workstations.
Real-World Test Results: AMD Radeon
Results from the current build of our AMD workstation, from its latest runs: short tool-call checks on November 9, 2026, in the final card layout, with prefill calculated from time to first token. The launchers and configuration are published on GitHub.A real-world example of our testing results.
Two Radeon GPUs, 48 GB of VRAM
An Intel Core i5-13600KF on an ASUS B760M board with 64 GB of DDR5 and two AMD Radeon RX 7900 XTX cards, on Ubuntu. Both cards sit behind a Broadcom PCIe switch at x8 each, and direct GPU-to-GPU transfers through the switch are turned on (about 13.8 GB/s each way when tested). Every model is split across both cards.Core i5 with two Radeon RX 7900 XTX cards behind a PCIe switch on Ubuntu, each model split across both.
Five Engines, One API
SGLang and vLLM on ROCm with tensor parallelism across both cards, llama.cpp on HIP for mixture-of-experts models, and Strata and Maya on HIP for models streamed from the SSD, each pinned to a tested version. One engine runs at a time behind an authenticated OpenAI-compatible gateway that starts or switches to whichever model a request names, so the same chat and agent tools reach every model.SGLang, vLLM, llama.cpp, Strata and Maya across both cards, one at a time behind one authenticated API that loads the requested model.
Empero at 173 Tokens/s, 256K Context
Empero, a mixture-of-experts model kept entirely in GPU memory, runs at 173 tokens per second on the current build with a 256K context. Nine model routes are qualified on this build: Qwen3.8-27B on SGLang and vLLM, Nemotron Bolt and Ripple-65B in GPU memory, and Qwen3.8-Flash-Next, DeepSeek V4.1 Flash and GLM-5.3-Flash streamed from the SSD by Strata and Maya.Empero at 173 tokens per second with a 256K context on the current build, and 9 model routes qualified.
| Model / quant | Engine | Context | Prefill t/s | Time to first token | Decode t/s |
|---|---|---|---|---|---|
| VRAM-only models | |||||
| Qwen3.8-27B | |||||
| INT4Tool calls | SGLang TP2 + DFlash2 | 160K | 1,736input ÷ time to first token | 3.86 s | 211.2 |
| INT4Tool calls | vLLM TP2 | 160K | 1,766input ÷ time to first token | 3.64 s | 155.1 |
| llama.cpp | |||||
| Empero Qwen3.8-35B-A3B IQ4_XSTool calls | llama.cpp HIP + MTP3 | 256K | 2,893input ÷ time to first token | 2.22 s | 173.3 |
| vLLM | |||||
| Nemotron 3.5 Bolt 74B AutoRound W4A16Tool calls | vLLM TP2 + Triton int4 MoE | 256K | 1,963input ÷ time to first token | 3.55 s | 40.1 |
| Ripple-65B Preview GPTQ W4A16Tool calls | vLLM TP2 | 64K | 764input ÷ time to first token | 8.18 s | 28.0 |
| MoE models with SSD streaming | |||||
| Strata | |||||
| Qwen3.8-Flash-Next IQ3_XXSTool calls | Strata HIP + MTP | 256K | 740input ÷ time to first token | 8.38 s | 86.3 |
| DeepSeek V4.1 Flash 1017 | |||||
| UD-IQ1_STool calls | Maya HIP | 32K | 35input ÷ time to first token | 182.22 s | 6.8 |
| UD-IQ1_STool calls | Strata HIP | 32K | 111input ÷ time to first token | 57.21 s | 5.6 |
| Maya | |||||
| GLM-5.3-Flash IQ2_XXSTool calls | Maya HIP | 32K | 57input ÷ time to first token | 105.75 s | 5.2 |
Quality ranking
| Model / quant | Rank | Score (95% interval) | Code (20) | Instructions (32) | Knowledge (60) | Math (40) | Reasoning (32) | Tools (32) |
|---|---|---|---|---|---|---|---|---|
| Qwen3.8-Flash-Next IQ3_XXSStrata (RDNA3 fork) | 1 | 78.2%72.3–83.2% | 12 | 30 | 48 | 33 | 23 | 23 |
| DeepSeek V4.1 Flash 1017 UD-IQ1_SMaya | 2 | 77.8%71.8–82.8% | 12 | 28 | 51 | 29 | 22 | 26 |
| Qwen3.8-27B INT4SGLang | 3 | 75.5%69.3–80.7% | 8 | 31 | 47 | 32 | 18 | 27 |
| Empero Qwen3.8-35B-A3B IQ4_XSllama.cpp | 4 | 69.9%63.5–75.6% | 11 | 25 | 47 | 24 | 16 | 28 |
| Nemotron 3.5 Bolt 74B AutoRound W4A16vLLM | 5 | 66.2%59.7–72.2% | 7 | 23 | 46 | 30 | 15 | 22 |
| GLM-5.3-Flash IQ2_XXSMaya | 6 | 63.0%56.3–69.1% | 10 | 25 | 43 | 11 | 21 | 26 |
| Ripple-65B Preview GPTQ W4A16vLLM | 7 | 62.0%55.4–68.2% | 5 | 17 | 42 | 32 | 14 | 24 |
| Test configuration | |
|---|---|
| Processor | Intel Core i5-13600KF |
| Motherboard | ASUS TX GAMING B760M WIFI |
| Memory | 64 GB DDR5 (61.6 GiB usable) |
| Graphics | 2× AMD Radeon RX 7900 XTX, 24 GB each (48 GB in total) |
| GPU links | Both cards behind a Broadcom PEX88096 PCIe switch at x8 each (Gen4 x16 uplink); direct GPU-to-GPU transfers are on, about 13.8 GB/s each way when tested |
| Power | Two power supplies: one for the PCIe switch board and both cards, one for the rest of the system |
| Model storage | NVMe and SATA SSDs |
| Operating system | Ubuntu, Linux kernel 7.0 |
| GPU software | ROCm 7.2.3 (HIP) |
| Results measured on | Qwen3.8-Flash-Next: November 18, 2026, on the Strata RDNA3 fork with no fixed expert RAM budget. All other rows: the current build in its November 7 card layout, November 9, 2026, one short tool-call request each, with direct GPU-to-GPU transfers on (GLM-5.3-Flash with a longer watchdog interval while it loads) |
How we test
- One model is loaded at a time, split across both cards. Model loading and warm-up runs are excluded from every figure.
- Every row uses the latest run on the current build. Where the latest run did not measure a value, the cell shows the most recent earlier current-build result and is labelled with its date.
- Prefill and time to first token (November 9): one real tool-call request of about 5,900 to 7,000 input tokens, with 25 tool definitions, through the normal API. The client records only time to first token (from sending the request to the first content, including tool-call parsing) and the total request time, so prefill is calculated as input ÷ time to first token.
- Decode (November 9): the tool-call reply’s output tokens divided by the time from the first token to the end of the request. The reply is a tool call of about 54 to 171 tokens, so it is a short burst rather than a sustained rate.
- Context is the window the installed profile is configured for.
- Checks passed: Tool calls means the model completed a tool round trip and a follow-up answer through the normal API.
- Since November 9, 2026, engine updates are checked every day and installed by hand instead of by a nightly updater: each new version is taken unless it really breaks something (a failed tool-call check, a crash, under half the current speed, or more than three times the time to first token).
NVIDIA Hardware & Results
Current hardware · Ubuntu: RTX 3080 Ti + RTX 3080 · 22 GB VRAM
Our NVIDIA test workstation. All results below were measured on November 5 and 6, 2026, on its current build with 96 GB of RAM.Our NVIDIA test workstation.
Two GeForce Cards, 22 GB of VRAM
An Intel Core i9-13900KS on an MSI Z690 board with 96 GB of DDR5, on Ubuntu 26.04 with NVIDIA driver 615. The RTX 3080 Ti 12 GB and RTX 3080 10 GB each have their own PCIe 4.0 x8 link to the CPU, with no direct GPU-to-GPU path, and the processor’s integrated graphics drives the display. Most profiles split the model across both cards.Core i9, 96 GB of RAM, and an RTX 3080 Ti and RTX 3080 on Ubuntu.
Five Engines, One API
HyperQwen on vLLM and SGLang for dense Qwen3.8-27B, kept entirely in GPU memory; and llama.cpp, Strata and Maya for mixture-of-experts models that stream their experts from system memory and the SSD. A model router behind an authenticated OpenAI-compatible API loads one model at a time.HyperQwen, SGLang, llama.cpp, Strata and Maya, one model at a time behind one authenticated API.
Up to 84 Tokens/s on 22 GB
Strata streams the IQ3_XXS build of Qwen3.8-Flash-Next from the SSD at 84 tokens per second with a 128K context. Nemotron Bolt, in NVIDIA’s native 2-bit build with part of it in system memory, runs at 68 tokens per second with a 160K context, and HyperQwen runs the 4-bit Qwen3.8-27B entirely in VRAM at 63 tokens per second with an 84K context.Up to 84 tokens per second streaming from the SSD, 68 for Bolt and 63 entirely in VRAM.
| Model / quant | Engine | Context | Prefill t/s | Time to first token | Decode t/s |
|---|---|---|---|---|---|
| VRAM-only models | |||||
| Qwen3.8-27B | |||||
| AutoRound INT4Tool calls | HyperQwen vLLM TP2 | 84K | 2,094.6 | 2.80 s | 62.7 |
| AutoRound INT4Tool calls | SGLang TP2 | 56K | 2,130.1 | 2.90 s | 60.12 |
| Models partly in system memory | |||||
| llama.cpp | |||||
| Nemotron 3.5 Bolt 74B native IQ2Tool calls | llama.cpp CUDA | 160K | 673.8 | 9.60 s | 67.8 |
| MoE models with SSD streaming | |||||
| Qwen3.8-Flash-Next | |||||
| IQ3_XXSTool calls | Strata CUDA | 128K | 1,058.0 | 5.40 s | 83.7 |
| UD-IQ4_XS-v31Tool calls | Strata CUDA | 128K | 759.9 | 7.63 s | 43.7 |
| Maya | |||||
| GLM-5.3-Flash IQ2_XXSTool calls | Maya CUDA | 64K | 550.9 | 10.06 s | 14.2 |
| DeepSeek V4.1 Flash 1017 | |||||
| UD-IQ1_STool calls | Strata CUDA | 32K | 87.1 | 65.59 s | 12.00 |
| UD-IQ1_STool calls | Maya CUDA | 32K | 209.4 | 28.84 s | 8.3 |
Quality ranking
| Model / quant | Rank | Score (95% interval) | MMLU-Pro | AIME 2025 | LiveCodeBench | BFCL single-turn | BFCL multi-turn |
|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash 1017 UD-IQ1_SStrata | 1 | 73.0%67.3–78.4% | 86.7% | 76.7% | 55.0% | 90.0% | 56.7% |
| Qwen3.8-27B AutoRound INT4SGLang | 2 | 68.1%62.3–73.9% | 85.7% | 66.7% | 50.0% | 91.7% | 46.7% |
| Qwen3.8-Flash-Next UD-IQ4_XS-v31Strata | 3 | 65.5%60.2–70.8% | 87.6% | 80.0% | 58.3% | 78.3% | 23.3% |
| Qwen3.8-Flash-Next IQ3_XXSStrata | 4 | 62.0%56.5–67.5% | 85.2% | 73.3% | 58.3% | 73.3% | 20.0% |
| GLM-5.3-Flash IQ2_XXSMaya | 5 | 57.2%52.1–62.3% | 86.2% | 6.7% | 55.0% | 91.7% | 46.7% |
| Nemotron 3.5 Bolt 74B native IQ2llama.cpp | 6 | 53.9%48.7–59.4% | 76.2% | 50.0% | 33.3% | 90.0% | 20.0% |
| Test configuration | |
|---|---|
| Processor | Intel Core i9-13900KS, with integrated graphics for the display |
| Motherboard | MSI MPG Z690 Carbon WiFi |
| Memory | 96 GB DDR5 (about 91.8 GiB usable) |
| Graphics | NVIDIA GeForce RTX 3080 Ti 12 GB + RTX 3080 10 GB (22 GB) |
| GPU links | Each card on its own PCIe 4.0 x8 link to the CPU; no direct GPU-to-GPU path, so the cards exchange data through system memory |
| Model storage | NVMe system drive for the streamed models; Samsung 870 QVO SATA SSD for Qwen3.8-27B, Nemotron Bolt and part of DeepSeek |
| Operating system | Ubuntu 26.04, Linux kernel 7.0 |
| GPU software | NVIDIA driver 615.71.09, at stock power limits |
| Results measured on | HyperQwen: November 9, 2026, after updating to vLLM 0.33.0. Qwen3.8-Flash-Next: November 16, after updating to Strata 0.2.15. Qwen3.8-Flash-Next UD-IQ4_XS-v31: November 18, first request after a fresh load, same Strata engine. GLM and DeepSeek on Maya: November 18, after updating to Maya 1.1.67. Nemotron Bolt: November 13, after updating to llama.cpp 1.0.0, with its prompt cache in RAM. SGLang and DeepSeek on Strata: November 5, in a fresh rerun of every model. All on the current build with 96 GB of RAM |
How we test
- One model is loaded at a time, split across both cards. Model loading is excluded, and the operating system’s file cache is not flushed.
- Each row is one real tool-call request of about 5,400 to 6,400 input tokens and a few dozen output tokens through the normal API, followed by an exact follow-up answer; we keep the fastest successful run.
- Prefill and decode are the engine’s own counters; time to first token runs from sending the request to the first content, including tool-call parsing.
- Context is the window the installed profile runs with.
- Checks passed: Tool calls means the model completed a tool round trip and an exact follow-up answer through the normal API.
- Since November 9, 2026, engine updates are checked every day and installed by hand instead of by a nightly updater: each new build is staged separately, checked with the same short tool-call test before and after the switch, and kept unless it really breaks (a failed check, under half the current speed, or more than three times the time to first token).
Image and Video Generation
Image and video diffusion models measured in ComfyUI on our AMD and NVIDIA workstations: Qwen-Image, FLUX 3 Image Klein, MiniMax H3, Wan 2.2 and LTX 2.3.Image and video models measured in ComfyUI.
Ready to get started?
Tell us about your environment, we’ll design a deployment that keeps your data yours.