Operating one of the best massive language fashions domestically typically requires a pc with a considerable quantity of RAM and a strong GPU, particularly when coping with fashions containing tons of of billions of parameters. In case you aren’t involved about velocity, you may get by and not using a GPU — a CPU can do something a GPU can, simply with an entire lot much less parallelism. That leaves reminiscence because the non-negotiable issue. Or is it? Not in line with Jim Bruges, who simply demonstrated a technique that enables a 744-billion-parameter mannequin to run on a system with 16 GB of RAM.
Bruges pulled this feat off utilizing an Arduino VENTUNO Q single-board laptop, an NVMe SSD, and an open supply inference engine referred to as Colibri. As configured, the setup can run two completely different massive language fashions domestically: Qwen3.6-35B-A3B and GLM-5.3, which incorporates a whopping 744 billion parameters. Neither requires an web connection as soon as the fashions and software program have been downloaded, and all inference takes place straight on the board.
Getting round reminiscence limitations
The key lies within the mixture-of-experts structure utilized by each fashions. Relatively than activating each parameter for every generated token, these fashions use solely a subset of their obtainable professional networks at any given time. This reduces the quantity of computation required, however the mannequin weights nonetheless must be accessible someplace.
Colibri solves the reminiscence downside by storing the weights on an SSD and loading the mandatory consultants into RAM as they’re wanted. It additionally maintains a cache of lately used consultants to cut back storage accesses. Because of this, the pc does not must have sufficient reminiscence to carry the complete mannequin directly.
Bruges used a 1 TB Essential NVMe SSD linked to the VENTUNO Q. The quantized Qwen mannequin requires roughly 23 GB of storage, whereas GLM occupies an infinite 419 GB. The software program runs below Ubuntu 24.04 and performs inference utilizing 4 of the board’s Cortex-A78C CPU cores. Neither the GPU nor the NPU is used within the remaining configuration.
It really works, however do not count on an excessive amount of
Unsurprisingly, transferring mannequin weights between an SSD and RAM comes with a really massive efficiency penalty. With a 96-slot professional cache, Qwen generated roughly 0.74 tokens per second. Adjusting the CPU frequency settings improved that determine barely to 0.774 tokens per second.
GLM was significantly slower, managing simply 0.03 tokens per second. In a single check, Bruges requested it to call three planets. The mannequin accurately responded with “Earth, Mars, Venus,” but it surely took roughly seven minutes to supply that reply. Clearly, this is not a sensible alternative for a high-end AI workstation.
This venture is not going to make anybody abandon their GPU-equipped workstation, but it surely does reveal a viable various to loading complete fashions into RAM. For functions the place response instances aren’t essential, or for these merely fascinated about experimenting with huge language fashions on modest {hardware}, Bruges’ method opens up some attention-grabbing potentialities.