Overview
Mergen (v1.0.0) is a distributed language-model inference runtime in C++20 and GGML built for old phones and tablets. It partitions a GGUF model into consecutive transformer layer shards, assigns each shard to a phone worker according to its available RAM and measured throughput, and streams intermediate hidden-state activations across the pipeline over USB or Wi-Fi.
Companion Technical Post: Read Mergen v1.0: Running 9B and 27B LLMs Across Reclaimed Phones Over USB for the full deep-dive into the multi-token GQA KV bug fix,
llama.cppparity verification, 785 MiB hybrid DeltaNet/KV memory savings, and real 3-phone fleet benchmarks.
Supported Model Architectures & llama.cpp Parity
Every architecture is verified against upstream llama.cpp via scripts/llama_parity.py (42/42 greedy top-token parity across 6 models × 7 prompts) and third_party/llama_jinja (64/64 exact byte-for-byte chat-template parity):
- Llama / Qwen2 / Mistral (
Ornith,TinyLlama-1.1B): Full GQA attention, RoPE, SwiGLU FFN, and quantized KV caches (f32,f16,q8_0). - Qwen3.5 & Qwen3.8 (
qwen35:4B,9B,Qwen3.8-27B): Hybrid Gated DeltaNet recurrent layers interleaved with full GQA attention (attn_q_norm/attn_k_norm), MTP trunk slicing (block_count - nextn_predict_layers), and per-layer-kind state allocation (saving ~785 MiB RAM on Qwopus 9B at 4096 context). - LiquidAI
LFM2(LFM2.5-2.6B) &LFM3(LFM3-1.7B): Stateful short-convolution (shortconv) blocks + gated sliding-window GQA attention (256-token ring buffer saving ~2.5 GiB RAM at 32k context). - Xing4.0 (
xing4_0:29B-A4BMoE): DeepSeek-V3 Multi-head Latent Attention (MLA), 64 sigmoid-routed experts + 1 shared expert (top-4), and 4-stream Sinkhorn hyper-connections.
Real Hardware Fleet Validation (mergen-fleet.py)
Mergen runs on both Linux SBCs/phones (Alpine Linux GM 8, postmarketOS nodes) and unrooted Android phones via the packaged mergen-worker.apk (with bundled libsodium XChaCha20-Poly1305 transport encryption, PARTIAL_WAKE_LOCK / top-app cpuset scheduling, and --weight-load mmap).
Across a 3-phone physical USB 2.0 High-Speed (480 Mbps) fleet (Tecno Pova 4 CH6n + Reeder P13 + Huawei Y7 DUB-LX3), mergen-fleet.py automatically plans layer splits, provisions shards, serves an OpenAI-compatible /v1/chat/completions + SSE streaming API, and auto-heals mid-prompt USB disconnects or worker restarts (--recovery-seconds 120):
| Model | Quant | 3-Phone Split (CH6n / P13 / DUB) |
Context | Verified Hardware Performance |
|---|---|---|---|---|
| TinyLlama-1.1B | Q4_K_M |
15 / 4 / 3 (22 layers) |
2048 |
2.54 tok/s decode (1.07 s TTFT) |
| Qwen3.5-4B | Q4_K_M |
21 / 6 / 5 (32 layers) |
4096 |
~410 ms/tok (2.44 tok/s) |
| Qwopus3.5-9B | Q4_K_M |
20 / 7 / 5 (32 layers) |
4096 |
794 ms/tok (559 ms/tok / 1.47× with 4B speculative draft) |
| Qwen3.8-27B | UD-IQ2_S |
37 / 15 / 12 (64 layers) |
512 (×2 sessions) |
3.21 s/tok running across the 3 phones alone |
Technologies: C++20, GGML, GGUF, Linux, Android, CMake, Python
Project Tags: