Single RTX 5090 Runs Full DeepSeek V4 Flash, Now Open Source, Token Freedom!

A joint team from UC Berkeley, MIT, and other institutions has open-sourced FreeToken, an edge inference system specifically designed for local deployment of MoE large models on consumer hardware through full-stack co-design. The paper, GitHub repository, and official website are all live, offering Windows/Linux desktop apps and CLI installation methods.

The performance figures are striking: the laptop RTX 4060 runs Qwen3.6-35B at a decode speed of 39.3 token/s, surpassing the median speed of 33 token/s in Codex production traces; a single RTX 5090 can run the 284B-parameter DeepSeek-V4-Flash. Traditional CPU-GPU offloading is extremely slow due to PCIe bandwidth bottlenecks, but FreeToken achieves interactive-level practical speeds.

For long prompts (4-16k context), FreeToken reduces time-to-first-token (TTFT) by 42-58%; for agent tool calls and chain-of-thought iterations, subsequent multi-turn interactions see TTFT reductions of 65-80%.

FreeToken's core innovations include full-layer double-buffered prefilling, bandwidth-adaptive hybrid scheduling, state reuse for agents, and elastic VRAM hot scaling. Double buffering ensures that while the GPU computes layer l, the weights for layer l+1 are already prefetched via PCIe, eliminating I/O wait times.

Bandwidth-adaptive scheduling probes PCIe bandwidth and CPU compute power in real time, dynamically calculating the optimal offload ratio q*. Missed experts are partially transferred to the GPU and partially computed on the CPU in parallel, allowing the system to converge to the theoretical maximum throughput of the hardware.

The state reuse mechanism sets lightweight checkpoints at special token boundaries, so context edits only resume from the nearest anchor, avoiding redundant computation. Elastic VRAM supports seamless shrinking of the LRU cache when VRAM is preempted, offloading missed experts to the CPU to ensure uninterrupted service.

Taking DeepSeek-V4-Flash as an example, its expert pool is about 140GB, requiring 32GB of VRAM (RTX 5090) and 192GB of system memory for stable operation. The system was tested on PCIe 3.0 x8, PCIe 4.0 x16, PCIe 5.0 x16, and different CPUs. Under heavy background I/O tasks, it automatically increases the CPU offload ratio, and when the bus is idle, it increases the GPU load ratio.

Stress tests simulate background 3D rendering or gaming causing a sudden VRAM drop of 4-8GB. Traditional solutions trigger CUDA OOM crashes, but FreeToken seamlessly shrinks the GPU-resident LRU cache with zero downtime overhead, offloading more missed experts to CPU computation, ensuring smooth inference degradation.

This shows that running large models no longer requires the entire model to fit in VRAM; with large DDR5 memory, high PCIe bandwidth, and a single consumer GPU, daily usable experiences are achievable.

Current open-weight models solve who can get the models, but not who can run them. Kimi-K3, GLM-5.2, and DeepSeek-V4-Flash approach the capabilities of the top closed-source models, but they typically require data centers. Steam data shows that about 72% of computers have Nvidia GPUs, with the most common being the laptop RTX 4060. FreeToken aims to leverage this hardware.

MoE models like DeepSeek-V4-Flash have 284B total parameters, 43 layers, and 256 routed experts per layer with 6 selected, activating only 13B per token. The active parameters can fit into the 32GB VRAM of an RTX 5090. However, prefilling destroys sparsity: the union of experts for long prompts covers nearly all experts, requiring near-complete expert pool transfers for each prefill.

FreeToken redefines the coordinated scheduling of heterogeneous edge hardware, treating CPU, system memory, PCIe bus, and GPU as a unified elastic computing platform, greatly reducing the cost of locally deploying high-intelligence models.

FreeToken's hardware requirements are still high; for example, DeepSeek-V4-Flash requires 192GB of system memory, which not all consumer users can meet. The system is optimized for MoE models, and its effectiveness on dense models is unknown.

The performance data in the paper is based on specific hardware configurations, and actual results may vary by environment. The stress tests simulate sudden VRAM drops, but other factors in real-world usage may affect performance.

The article ends with a mention of 'just missing the legendary Qwen3.8-A3B,' hinting that future smaller models may further lower the barrier, but specific details are not disclosed.