How slow is 16GB of memory running 744 billion models! 3 words in 7 minutes

On November 10th, Fast Technology announced that what would happen if a top AI model with 744 billion parameters ran on an Arduino development board?

Recently, developer Jim Bruges completed a seemingly impossible challenge: he successfully ran the 419GB GLM-5.3 large model on a single-board computer with only 16GB of memory.

The hardware used in this challenge is the Arduino Ventuno Q, although it is called Arduino, but it is equipped with a Qualcomm Dragonwing IQ8 processor, has 8 cores, and is equipped with 16GB LPDDR5 memory and 1TB NVMe SSD.

To force 16GB of memory to eat 26 times its own capacity of model data, the core secret is the MoE architecture of GLM-5.3. The hybrid expert model uses a routing mechanism to activate only a small number of expert weights at each inference, without having to call all 744 billion parameters.

Together with the open source tool Colibri, the system can store temporarily unused weights in a solid state drive and only replace them in real time if needed.

The test results were successful, but the efficiency was impressive. In a simple test of three planets, the system took about 7 minutes to spit out the words Earth, Mars, and Venus.

Data monitoring shows that the platform is only generating 0.03 tokens/s, which means that it may take nearly an hour to generate a 100-word response.

In terms of performance release, the Hexagon NPU on Qualcomm chips has a hashrate of 40 tops, but due to the imperfect back-end support of the software, the full test is carried out by the CPU. Even with the use of quantization technology to compress data accuracy, the read and write speed of SSDs has become the biggest shortcoming that restricts inference efficiency.

Simply put, this is more like a muscle demonstration of the geek community, demonstrating the possibility of edge devices running oversized models with SSD displacement technology.

Although the small model of 0.77 Token/s performs well, for the 744B, the performance of the current single-board computer is far from the practical threshold.

Source: MyDrivers (Fast Technology). Content is presented with source attribution. Read the original report.