Lawyer's eight-V100 server runs most of DeepSeek V4.1 Flash at 4-bit precision

What changed in the build

A trial lawyer in Columbia, South Carolina has upgraded the private AI server he uses for his law practice, according to Tom's Hardware. The machine, built around eight used Nvidia V100 GPUs, now runs about 80% of its DeepSeek V4.1 Flash workload at 4-bit NVFP4 precision instead of 2-bit. He also increased the system memory from 256GB to 1TB and added an Nvidia RTX 3090.

The lawyer was identified as Nick Mermiges, founder of PLI Labs and proprietarylegal.ai, after he commented on the original coverage of his build. His consultancy, PLI, now offers private, on-premises large language model serving as a service.

How the precision upgrade works

DeepSeek V4.1 Flash is a mixture-of-experts model, meaning its work is shared among many specialist sub-networks. It has 384 routed experts in each of 40 layers, or 15,360 in total. At the official 4-bit precision, which the project's README says matches Nvidia's published weights at 4.25 bits per weight, those experts would need roughly 269 GiB of memory. Eight 32GB V100 cards offer only 256 GiB in total.

The workaround keeps every expert on the cards at 2-bit and upgrades the most frequently used ones to NVFP4 until the cards are full. That came to 5,949 experts, or about 38.7% of the total. In single-request testing, between 80% and 86% of decode routes landed on the upgraded experts, with the figure predicted at 78% to 85% for the two-request setup he now runs.

Quality and speed trade-offs

The higher precision improves fidelity rather than accuracy, which stayed about the same. Measured against Nvidia's own version of the model, agreement on the next predicted token rose from 92.2% to 95.1%, and KL divergence, a measure of how far the model's probability outputs sit from the reference, fell from 0.196 to 0.079 nats. On 513 questions from a standard multiple-choice knowledge test, differing answers dropped from 79 to 8.

Speed is the price. Code output fell from 160–215 to 150–180 tokens per second, and legal writing is 9% to 23% slower depending on prompt length. Benchmarks still showed 86 to 106 tokens per second for legal text and 78 to 86 for prose, above his original target of about 80. Running two requests at once lifts total throughput to 159.0 tokens per second against 118.9 for one, and the first token arrives sooner at every prompt length tested.

Each extra request needs a slot that holds its own state on the cards, leaving room for fewer upgraded experts: 6,215, 5,949 and 5,663 for one, two and three slots. Each added slot may push one or two percentage points of calls back to 2-bit experts.

Hardware fixes and remaining limits

The memory upgrade uses eight 128GB modules at 2,933 MT/s, replacing eight 32GB modules that had to run at DDR4-2666 because of ECC errors. The V100s remain capped at 1,522 MHz.

Crashes that affected the earlier build were traced to the power supply after he systematically ruled out PCIe slots, cards and cables, concluding it was probably one bad PSU. Adding fans brought no temperature change, showing airflow was not the issue. The V100s themselves are on borrowed time, as R580 is the last Nvidia driver branch to support them.

Why it matters

The project shows what determined individuals can achieve with older hardware. Mermiges built the software largely with AI assistance, using Claude Code to write custom Volta kernels and run benchmarks under his direction, and he said he was surprised at how well one model handled the kernel work. His stated goals of handling more than one request at a time and offloading more of his workload have now been met.

Privacy remains the driving motive. He reaffirmed that the documents never leave the office, which is why he runs the model locally with a legal harness wrapped around it. He ported DeepSeek V4.1 Flash successfully after earlier attempts with local models failed, though he says it is not as capable as frontier models.