Overview
Mergen is a distributed language-model inference runtime for old phones and tablets. It splits a GGUF model into consecutive groups of transformer layers and assigns those groups to workers, allowing several devices to participate in a single generation pipeline.
I built it around the constraints of this hardware: limited memory, mixed device capabilities, and communication over Wi-Fi. The coordinator inspects a model, plans layer placement within memory limits, builds the required shards, and transfers them to workers. Intermediate activations then travel through the pipeline as the workers execute their assigned layers.
The implementation uses C++20 and GGML's CPU backend. Transfers can resume after interruption, and cached shards are checked before reuse. Worker sessions keep generation state isolated, while diagnostic commands expose hardware information, connectivity, and deployment status.
A physical deployment has partitioned TinyLlama-1.1B Q4_K_M across an Alpine Linux GM8 and a postmarketOS Lenovo phone, transferred the shards, and generated a token over the real Wi-Fi path. That is an end-to-end hardware milestone, rather than a claim about sustained throughput or interactive usability.
Mergen is still an experimental runtime. Real-model Android validation, broader compatibility, and sustained performance measurements remain unfinished. The project explores what distributed inference on recovered hardware can do, with results documented separately from planned capabilities.
Technologies: C++20, GGML, GGUF, Linux, Android, CMake
Project Tags: