Fleet Orchestrator: Building for Devices That Disconnect

Getting an old phone to boot Linux is a satisfying milestone. Making it useful over time creates a different set of problems. It needs to receive work, report what happened, reconnect after an interruption, and recover without someone watching a terminal.

I built fleet-orchestrator to manage a mixed collection of repurposed devices. The core is written in Rust: a controller, a small agent on each device, and the fleetctl command-line interface. It handles device enrollment, jobs, results, output files, supervised services, and staged rollouts.

The work became especially interesting when I started interrupting it on real hardware. A reboot exposed a job-retry bug. A clock reset broke TLS. A sandbox rule prevented BusyBox from starting normally. Those failures shaped the system more than a successful demonstration of remote command execution could.

A controller, an agent, and a durable record

The controller uses axum and SQLite to maintain the device registry, job state, leases, and operational records. Each agent polls for work, reports its capabilities, and sends back results and metrics. The CLI provides the operator-facing workflow for enrollment, submission, inspection, and recovery.

Polling means the device initiates communication with the controller. The controller does not need an inbound SSH connection to every worker to deliver a job. Polling, execution, and uploading also run independently, so a stalled result upload does not monopolize the agent's control loop.

The deployment has a single controller database. SQLite's write-ahead logging mode allows readers and a writer to proceed concurrently, while still allowing only one writer at a time. That is an explicit architectural boundary. High availability and sharding remain future work rather than claims attached to the current implementation.

On Linux, the agent can be built as a static musl binary for x86-64, AArch64, or ARMv7. There is also an Android foreground-service wrapper written in Kotlin around the Rust agent. Platform capabilities differ: Windows probing exists, for example, but Unix script execution does not become a Windows feature just because the agent builds there.

A lease answers who owns this attempt

A job needs more identity than a command and a device name. When an agent claims work, the controller creates a lease for that attempt. The lease has an expiry and an unguessable identifier, and the executing agent renews it through heartbeats.

For job callbacks, the controller checks the device credential, the owning device, and the lease identity. Possessing a valid device token does not grant authority to complete another device's job. If an expired attempt is reclaimed, its old authority cannot be reused to overwrite the replacement attempt.

This matters when the network delivers events out of order. An old result can arrive after a retry has started. A cancelled process can finish late. Those messages need a defined outcome: acknowledge a valid duplicate, reject a conflicting result, or retain a diagnostic without changing the job's terminal state.

The contract is at-least-once execution. A script may run again after a restart or lease expiry. Deduplicating result records cannot undo an external action the script already performed. A job that changes a file, calls another service, or increments a counter must account for retries itself.

Finishing work and delivering its result are separate events

An agent can finish a job while the controller is unreachable. Its output should not exist only in memory while it waits for a successful request.

The agent therefore writes undelivered results and artifact records to a durable outbox. The write sequence uses a temporary file, synchronization, and an atomic rename. A separate uploader retries delivery, and acknowledged records leave the pending queue. Corrupt records and permanent rejections are moved aside with a reason for inspection.

Storage is bounded. Retry deadlines and spool quotas mean the outbox is not an unlimited delivery guarantee. Queue size, age, dead letters, and quota discards are operational signals that need to be visible.

In a GM8 hardware drill, controller traffic was deliberately rejected for 30 seconds. The job completed, its result waited locally, and it reached the controller on attempt one after connectivity returned. That established recovery from the particular interruption tested. It did not establish behavior under every form of Wi-Fi failure or prolonged disconnection.

The reboot bug: an interruption looked like a completed failure

The next drill rebooted the device during an active script.

The original behavior was internally consistent but wrong for the intended recovery contract. During shutdown, the script was terminated. The agent recorded the termination as a failed result and uploaded it after boot. That terminal failure arrived before the lease expired, so the configured retry budget never got a chance to recover the interrupted work.

The fix makes a narrow distinction. If a script actually dies from a termination signal while agent shutdown is already in progress, that unfinished attempt is abandoned without publishing a terminal result. The lease can then expire, and the controller decides whether another attempt is allowed.

An ordinary script failure still behaves as a failure. Explicit numeric exit codes, timeouts, and controller cancellations keep their existing meanings. The change does not turn every error into an automatic retry.

On the patched hardware retest, attempt one was interrupted by reboot. The device returned, the controller reclaimed the expired lease, and attempt two completed successfully. Exactly one result row was recorded for that job: the successful second attempt.

That result still does not imply exactly-once effects. Anything the first attempt did before interruption could happen again.

A running time service was not enough

One recovery failure came from outside the Rust code. After reboot, the GM8's clock moved backwards far enough that newly issued TLS certificates appeared not yet valid.

The device ran Alpine and OpenRC on an Android-derived kernel. Its RTC read January 1970, while an existing clock-floor mechanism only restored an older image timestamp. The NTP account also lacked the network group required by that kernel for unprivileged sockets. Seeing an NTP process in the process list had not established successful time synchronization.

The device configuration was changed to preserve a software clock across orderly shutdown and boot, and the NTP account received the required group membership. A subsequent reboot and probe completed over verified TLS without manual clock correction. Certificate verification stayed enabled.

The saved timestamp is only a recovery floor. It does not advance while the device is powered off, and the drill did not prove network-time synchronization or abrupt-power-loss recovery. Those remained separate checks.

Sandboxing has to match the actual kernel

The GM8 also exposed a compatibility problem in the script sandbox. BusyBox resets its user and group IDs to their existing values during startup. The seccomp policy blocked identity-setting calls indiscriminately, causing even harmless shell workloads to terminate.

The correction permits those calls only when they retain the expected identity. Requests for another identity remain denied. A hardware trace and follow-up jobs verified the behavior.

However, fixing BusyBox did not add missing kernel features. The GM8 kernel still lacked user namespaces. In the default degraded mode, some isolation layers were unavailable; with the stricter sandbox requirement enabled, the agent correctly refused the workload.

This is why I describe isolation in terms of what the target enforces. A successful script and an enabled sandbox setting do not prove complete confinement. The project is an operator-controlled fleet tool, and these results do not establish safe execution of arbitrary untrusted tenants' workloads.

Experimental CPU tuning follows a separate privilege boundary. The ordinary agent remains unprivileged, while a small broker handles validated tuning requests. Tuning requires explicit enablement and remains disabled by default.

Unknown health should stop a rollout

The controller supports supervised service workloads and staged canary rollouts. Moving to the next group of devices depends on the configured health gate.

That gate is only useful if its inputs are current. A cool temperature reported before disconnection should not remain evidence that the device is healthy now. The controller measures freshness using receipt time. Missing or stale metrics produce unknown health, and the default rollout policy holds unknown devices.

The network drill exercised this distinction: health became unknown during the interruption, then returned to healthy after fresh telemetry arrived. Uploading a queued job result did not itself make the resource measurements fresh.

Rollback also needs persistence. The controller records cancellation and corrective-work intents before sending them, allowing recovery to resume after a controller restart. Service correction uses the last controller-acknowledged state for each device instead of guessing from whichever historical job looks most recent.

What is implemented, and what still needs a drill

The system now includes result buffering, artifacts, service supervision, staged rollouts, operational metrics, signed release verification, and offline recovery bundles. Bootstrap uses a one-time pairing flow and a scoped download token, so a new device does not need the fleet-wide administrator key.

The GM8 tests provide concrete evidence for controller interruption, queued delivery, orderly reboot, the clock correction, and interrupted-script retry. They also document an important isolation limitation.

The remaining acceptance work includes a sustained hardware soak, silent packet loss, abrupt power loss, Android-wrapper recovery coverage, and signed update/rollback drills. Implementing a recovery command and exercising recovery on a physical device are separate milestones.

For the next round of testing, I want to keep the same standard: define the interruption, record the state before and after it, and show what the system recovered. Every completed drill makes the fleet less dependent on someone being there when a device disappears.