Skip to content
Theme
OnDevice Tools

GadgetsLaptops & PCs

Homelabber runs 2.8-trillion-parameter Kimi K3 at 3.11 tokens per second with two RTX 3090s

A Reddit builder's Optane-based server now generates text from Moonshot AI's Kimi K3 at 3.11 tokens per second after adding two RTX 3090s and moving 414 GB of the model into a RAM drive, a 41% speed boost.

Illustration of a home server with two graphics cards and Optane memory modules running a large AI model

Key takeaways

  • A Reddit homelabber's Optane PMem 200 server now runs Moonshot AI's 2.8-trillion-parameter Kimi K3 at 3.11 tokens per second for generation and 16.8 tokens per second for prompt evaluation.
  • Adding two RTX 3090s and moving 414 GB of the model into a tmpfs RAM drive raised generation speed by 41% and prompt evaluation by 26%.
  • The graphics cards handle components used on every token, while the routed experts stay in Optane and DRAM on the CPU side.
  • Cooling remains a challenge, with the two 12 W blower fans for the Optane modules still required despite the upgrades.
In this article
  1. What changed in the second-generation build
  2. How the RAM drive lifted speed by 41%
  3. Why two 24 GB cards can serve a huge model
  4. Cooling remains the hardest problem
  5. What could come next

What changed in the second-generation build

A homelabber has upgraded an unusual AI server built around Intel's second-generation Optane persistent memory, according to Tom's Hardware. The machine now runs Moonshot AI's Kimi K3, a model with 2.8 trillion parameters, at 3.11 tokens per second for text generation. The update, shared in a follow-up post on Reddit, adds two RTX 3090 graphics cards and moves a large slice of the model into system memory.

The builder also reports 16.8 tokens per second for prompt evaluation, the stage where the model reads incoming text.

How the RAM drive lifted speed by 41%

The key change is a tmpfs RAM drive, a Linux file system that keeps files in memory rather than on a disk. Tom's Hardware reports that 45 of the model's 94 shards, or 414 GB out of 865 GB, now sit in this drive, with the remainder on the Optane memory. Because DRAM is faster than Optane, reading that portion of the model no longer bottlenecks the system. The result was a 41% gain in generation speed and a 26% gain in prompt evaluation.

This approach trades away some of the original design's economy. The first build used Optane to avoid buying large amounts of system memory, and not everyone has hundreds of gigabytes of RAM to spare. The memory in a RAM drive is still volatile, so its contents vanish on power loss, even though it behaves like a disk.

Why two 24 GB cards can serve a huge model

Illustration: AI, crypto, NVIDIA logo, privacy and the labels Kimi K3, Moonshot AI, Optane

Two RTX 3090s offer 48 GB of video memory in total, only about 5.5% of the 865 GB model file. The cards do not hold the model. Instead, they run the parts used on every token: the attention mechanism, which decides how each token weighs the surrounding context, the dense feed-forward layer that is not split into experts, the shared experts, and the cache of the current conversation.

Kimi K3 is a mixture-of-experts model, meaning its weights are divided into many expert blocks with only a few active at a time. The model has 896 experts, of which 16 are chosen per token along with 2 shared ones, giving 104 billion active parameters out of the 2.8 trillion total. The routed experts, which change from token to token, stay on the CPU side in Optane and the RAM drive. The Q2_K quantised file used here compresses the model to 865 GB, well below the roughly 1.56 TB of Moonshot's own 4-bit release.

Cooling remains the hardest problem

Heat is still a serious concern. Tom's Hardware reports that the faster throughput has not removed the original thermal warning, so the two 12 W blower fans cooling the Optane memory modules remain in place. The persistent memory runs very hot under load.

Fitting the graphics cards also demanded physical changes. The builder had to cut away some of the case metal and move the power supply to make room for the cards and their cooling, on top of two GPUs in the 350 W class.

What could come next

The builder expects further gains. Planned work includes customised quantisation, where compression is chosen per tensor or layer, or based on where the data is stored. Tom's Hardware notes this could resemble techniques used by lawyer Nick Mermiges, and might reduce bandwidth and capacity pressure on the slower memory tiers.

Speed alone will not decide the next steps. Output quality has to be weighed alongside tokens per second, since heavier compression can affect how well the model performs. The 3.11 tokens per second figure is modest, but it falls within the 2 to 4 tokens per second the builder predicted in the original thread, and it shows that very large models can run on consumer hardware with careful tiering of memory.

Sources

  1. Homelabber’s second-gen Optane build uses two RTX 3090s to run 2.8-trillion-parameter Kimi AI model at 3.11 tokens per secondTom's Hardware

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.