Mergen v1.0: Running 9B and 27B LLMs Across Reclaimed Phones Over USB

When I first published Mergen, the goal was modest: prove that a C++20 / GGML runtime could split a GGUF language model into consecutive transformer layer shards, ship those shards to reclaimed phones, and generate a real token end-to-end across mixed Linux and Android devices.

Over the first week of November 2026, Mergen crossed the line from a proof-of-concept into v1.0.0:

• It now reaches 42/42 greedy top-token parity and 64/64 chat-template render parity against upstream llama.cpp across Llama/Qwen2, Qwen3.5 (4B / 9B / Qwen3.8-27B), LiquidAI LFM2.5 & LFM3, and Xing4.0 (29B-A4B MoE).
• It runs Qwopus3.5-9B (Q4_K_M, 4096 context) and even Qwen3.8-27B (UD-IQ2_S, 64 layers) directly across a 3-phone physical USB fleet (Tecno Pova 4 CH6n + Reeder P13 + Huawei Y7 DUB-LX3) supervised by a fault-tolerant USB orchestrator (mergen-fleet.py).

Here is the technical breakdown of the bugs, memory walls, and hardware bottlenecks that had to fall to make 9B and 27B models run on old phones.

1. The Silent Quality Killer: Multi-Token GQA Prefill KV Transposition

Early single-token smoke tests on TinyLlama looked plausible, but multi-token prompt completions degraded into repetitive gibberish whenever a model used Grouped-Query Attention (n_head_kv > 1).

Using the new eval-perplexity harness and scripts/llama_parity.py (which compares Mergen's raw logits against llama-completion on the exact same GGUF and prompt), I isolated the bug to the multi-token prefill path (n_tokens > 1):

• During multi-token prefill (ggml_Flash_Attn / explicit K/V cache views), the Key cache write and Value cache read strides had the n_head_kv and n_tokens dimensions transposed whenever n_head_kv > 1 and n_tokens > 1.
• Token 0 of the prompt landed at the right offset (which is why 1-token step tests passed!), while tokens 1..N-1 scrambled the KV heads across positions.

Fixing the K/V cache stride ordering dropped TinyLlama's multi-token prompt negative log-likelihood from 7.9 down to 1.686 (perplexity 5.398) and immediately turned "The capital of France is" greedy completions into "Paris." with exact llama.cpp logit parity (max |Δlogit| = 0.014).

2. Supporting Modern Hybrid Architectures & Saving 785 MiB on Qwopus 9B

Modern open-weight models have moved far beyond plain Llama attention. To run current 2026 models on memory-starved phones, I added native GGML graph builders and state managers for four new architecture families:

1. Qwen3.5 / Qwen3.8 (qwen35: 4B, 9B, and 27B)

Qwen3.5 interleaves Gated DeltaNet recurrent layers (ssm_* conv1d + delta-rule state) with full GQA attention layers (attn_q_norm, attn_k_norm, gated Q projection), plus Multi-Token Prediction (MTP) head blocks at the end of the GGUF.

Bringing Qwen3.5 to exact llama.cpp parity (max |Δlogit| = 0.025 on 9B, 0.022 on 27B) uncovered four critical details:

1. MTP Trunk Slicing: GGUFs with nextn_predict_layers > 0 store extra MTP layers after the main trunk. Slicing the trunk to block_count - nextn_predict_layers prevents running unscheduled MTP blocks during standard inference.
2. Contiguous GQA Query-Head Grouping: Qwen3.5 maps query heads to KV heads via contiguous grouping (h / (n_head / n_head_kv)) rather than modulo interleaving (h % n_head_kv).
3. Per-Layer-Kind State Allocation (-785 MiB RAM): Initially, workers allocated both a full-attention KV cache and a DeltaNet recurrent state buffer for every layer in their shard. In Qwen3.5, each layer is either DeltaNet or full attention—never both. Allocating KV cache only on attention layers and recurrent state only on DeltaNet layers saved ~785 MiB of RAM across a 32-layer Qwopus 9B pipeline at 4096 context, which is the difference between fitting on a 4 GB phone and triggering the Android LMK!

2. LiquidAI LFM2 (LFM2.5-2.6B) & LFM3 (LFM3-1.7B)

• Combines stateful short-convolution (shortconv) layers (keeping the last L_cache - 1 = 3 tokens per session) with gated sliding-window GQA attention.
• For LFM3, implementing a 256-position sliding-window KV ring buffer on the attention layers saves ~2.5 GiB of RAM at 32k context compared to a dense KV allocation.

3. Xing4.0 (xing4_0: 29B-A4B MoE)

• Implements DeepSeek-V3 Multi-head Latent Attention (MLA) (q_a/q_b LoRA + compressed kv_a_mqa/kv_b), 64 routed experts + 1 shared expert (sigmoid routing with e_score_correction_bias, top-4 active experts), and 4-stream Sinkhorn-Knopp hyper-connections (hc_pre, hc_post, hc_res).

4. Vendored llama.cpp Jinja Chat Template Engine (64/64 Parity)

Different instruct models (ChatML, Qwen3.5 thinking/non-thinking templates, LFM, Mistral/Ornith) embed complex Jinja2 chat templates inside their GGUF metadata. Rather than maintaining brittle regex approximations, I vendored llama.cpp's native C++ Jinja engine (third_party/llama_jinja) directly into mergen_gguf_tokenizer.cpp and linked it into /v1/chat/completions, passing 64/64 exact byte-for-byte template parity checks.

3. Hardware Bottlenecks on a Real 3-Phone USB Fleet

With model math verified against llama.cpp (42/42 top-token parity across 6 models × 7 prompts), I deployed the runtime across three physical Android phones connected over USB (adb forward):

Bringing 9B and 27B models up on these three phones exposed five real-world systems bottlenecks:

1. The 12 Mbps Full-Speed USB Hub Trap

During initial 3-phone tests over a cheap 4-port USB hub, TinyLlama decode crawled at 1.22 tok/s with a 4.29 s time-to-first-token (TTFT). Inspecting lsusb -t revealed that the hub negotiated at USB 1.1 Full-Speed (12 Mbps shared across all 3 phones)! Because the final stage initially streamed the entire float32 vocabulary logit vector (32,000 × 4 B = 128 KB, or 993 KB per token on Qwen3.5's 248,320 vocab!) back to the coordinator on every token, a 12 Mbps bus spent hundreds of milliseconds just serializing logits.

We fixed this at both layers:

• Hardware: Replaced the 12 Mbps hub with a 480 Mbps High-Speed USB 2.0 hub (doubling TinyLlama throughput to 2.54 tok/s and cutting TTFT 4× to 1.07 s).
• Protocol (logits_mode): When the coordinator uses greedy decoding (--temp 0) or speculative target verification, the output worker now performs the argmax locally on the phone and returns only the 4-byte winning token ID (tokens_out) instead of streaming a 1 MB float array over USB—cutting Qwopus 9B step latency from 890 ms/tok down to 794 ms/tok.

2. Android background cpuset Throttling (242 ms → 158 ms)

When a phone screen turns off or an Android worker runs without a foreground activity, Android's scheduler moves its threads into the background cpuset—pinning all GGML compute threads onto slow Cortex-A53 efficiency cores (LITTLE cluster).

• Having mergen-worker.apk acquire a PARTIAL_WAKE_LOCK and moving the worker into /dev/cpuset/top-app/tasks (while using all big.LITTLE cores during multi-token prefill and pinning decode to big cores) dropped P13's per-token step time from 242 ms down to 158 ms (1.53× speedup).

3. mmap Weight Loading + token_embd Planner Exemption

On models with huge vocabularies like Qwen3.5 (248,320 × 4096), the quantized token_embd.weight tensor alone is ~650 MiB, even though Stage 1 only ever performs single-row token embedding lookups (ggml_get_rows).

• Switching workers to --weight-load mmap lets the Linux kernel page in only the touched rows of token_embd.weight from flash storage instead of faulting the entire 650 MiB table into anonymous RAM.
• Updating plan-layers to exclude lookup-only token_embd bytes from Stage 1's RAM budget unlocked 22 layers on CH6n (instead of 14), enabling Qwopus3.5-9B (Q4_K_M, 32 layers at 4096 context) across the 3 phones (20/7/5 split).

4. Fitting Qwen3.8-27B (64 Layers) Across the 3 Phones Alone

With --weight-load mmap and the per-layer-kind DeltaNet/KV state split, we pushed the 3-phone fleet to its extreme limit: Qwen3.8-27B (UD-IQ2_S, 64 layers, 5120 hidden dim) running entirely on the 3 phones with zero host compute:

• CH6n hosts layers 0..36 (37 layers)
• P13 hosts layers 37..51 (15 layers)
• DUB-LX3 hosts layers 52..63 + output (12 layers)
• Verified with coherent 3-phone completions and 2 concurrent isolated sessions (--max-sessions 2 --max-tokens 512).

4. Fault-Tolerant Fleet Orchestration (mergen-fleet.py)

Running long inference jobs over USB cables on reclaimed phones means dealing with occasional loose micro-USB ports, transient adbd resets, or Android OOM kills.

To make the cluster self-managing, I built scripts/mergen-fleet.py and the packaged mergen-worker.apk (with bundled libsodium XChaCha20-Poly1305 transport encryption and stable Settings.Secure.ANDROID_ID node identities):

• mergen-fleet.py status: Queries adb, checks mergen-worker PIDs, MemAvailable, battery level/temperature, and each phone's recorded decode throughput (worker-speeds.tsv).
• mergen-fleet.py serve --model qwopus-9b: Automatically selects the optimal layer split (20/7/5) from live free RAM and historical worker tok/s, provisions shards (resuming from .part on interruption), and starts the OpenAI-compatible /v1/chat/completions server.
• mergen-fleet.py watch + --recovery-seconds 120: Continuously heals dropped adb forward tunnels and restarts crashed workers. In live kill-and-recover drills, killing mergen-worker on P13 mid-prompt caused the coordinator to wait for watch to re-establish the tunnel, re-ensure the cached shard, replay the prompt from token 0, and return the completed answer without failing the client request.

Summary of Hardware Benchmarks (v1.0.0)

Check out the updated Mergen Project Page for the architecture overview and links to the rest of the Phone Revival Lab.