Skip to content
Theme
OnDevice Tools

GadgetsSmartphones

Three used Nvidia V100 GPUs reach 100 tokens per second in a 1,042 dollar homelab AI rig

A homelab builder has run the Qwen 3.6 35B model on three second-hand Nvidia Tesla V100 graphics cards at about 100 tokens per second, spending 1,042 dollars on the core components. The setup outperforms a more expensive Mac Studio at long context lengths, according to Tom's Hardware.

Illustration of three used Nvidia V100 graphics cards in a homelab AI build

Key takeaways

  • Three used Nvidia V100 GPUs ran Qwen 3.6 35B at about 100 tokens per second in a 1,042 dollar build.
  • At 128K context the V100s reached 72.7 tokens per second, staying above the 70 tokens per second realtime threshold.
  • The setup beat the M5 Max Mac Studio by 19 percent at the longest context length tested.
  • The V100s used about half the energy of P100s for the same work.
In this article
  1. What the builder achieved
  2. Benchmark results across context sizes
  3. How it compares with other hardware
  4. Energy use and future plans

What the builder achieved

A homelab builder ran Qwen 3.6 35B on three used Nvidia Tesla V100 GPUs, reaching about 100 tokens per second in a custom setup costing 1,042 dollars. Tokens per second measures how quickly a language model generates text, with 70 tokens per second treated as the threshold for realtime interaction. The builder used only one of the two CPUs in the price total and left out RAM, storage, the case and cooling.

Benchmark results across context sizes

Illustration: Alibaba Cloud logo, performance, AI, NVIDIA logo and the labels Homelab, NVIDIA V100

The test involved ten runs of 512 generated tokens per test, with context pre-filled at 4K, 32K, 64K or 128K. At Q8, or 8-bit quantization, Qwen 3.6 35B reached 100.8 tokens per second at 4K context, falling to 72.7 at 128K. The smaller Qwen 3.6 27B model managed 24.3 tokens per second at 4K and just 14.2 at 128K. Qwen 3.6 35B is a mixture-of-experts model, meaning it activates only about 3 billion of its 35 billion parameters for each token.

How it compares with other hardware

At Q8, P100 GPUs reached 40.2 to 55.2 tokens per second across the same context range with Qwen 3.6 35B. The M5 Max Mac Studio, tested with 40 GPU cores and 128 gigabytes of memory, reached 43.9 tokens per second with a roughly 195K-token prompt on the same model. The Mac was slightly faster at 4K and 32K context, while the three V100s pulled ahead at 64K and 128K, by 19 percent at the longest context. The Nvidia DGX Spark reached 45.8 tokens per second in a separate benchmark, less than half the V100s' 4K figure.

Energy use and future plans

The V100 setup used about 0.02 kilowatt-hours for a 128K run of the 35B model, while P100s used twice that amount and two or three K80s used as much as 0.35 to 0.45 kilowatt-hours. These figures cover the GPUs only and are not measured from the wall. The builder plans to compare inference backends and experiment with speculative decoding to improve performance. A standalone cooler for the cards is nearly finished, and beta testers will be recruited from a mailing list, though no date has been set.

Sources

  1. Three 'e-waste' Nvidia V100s hit 100 tokens per second on Qwen 3.6 35B in a $1,042 homelab rigTom's Hardware

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.