QEngineKit
Persistent task queue for Swift — SwiftData-backed jobs with capped concurrency, retries, and crash recovery
Overview
QEngineKit solves a problem every non-trivial app hits: work that must survive process death. A photo upload interrupted by a force-quit, a sync that fails on a dead connection, an export the user expects to finish eventually — none of these can live in an in-memory queue. QEngineKit persists every job in SwiftData and rebuilds the execution state on launch, so work enqueued before a crash resumes after it.
Because closures can't be persisted, the design separates what (a kind string plus a Data payload, stored) from how (handlers registered at launch) — the same architecture used by server-side job systems like Sidekiq, translated into Swift's actor model. The scheduler fills a bounded concurrency pool by priority, retries failures on capped exponential backoff with jitter, and understands the device it runs on: a task marked Wi-Fi-only simply waits, burning no attempts, until the network monitor reports an eligible path.
- The problem
- Non-trivial apps need work that survives process death — an interrupted upload, a sync that fails on a dead connection, an export the user expects to finish — none of which can live in an in-memory queue.
- Why I built it
- I wanted a properly engineered, crash-safe task queue built natively on Swift's actor model and SwiftData, rather than reaching for an ad hoc solution each time this problem came up.
- Target users
- Swift developers building iOS, macOS, or visionOS apps who need reliable background job execution — uploads, syncs, exports — that survives app termination.
Features
Crash-safe persistence
Every job is persisted in SwiftData and execution state is rebuilt on launch, so work enqueued before a crash resumes after it.
Device-aware scheduling
Per-kind constraints (network, unmetered, charging) gate tasks without consuming retry attempts, with live NWPathMonitor integration re-evaluating the queue on connectivity changes.
At-least-once delivery with crash recovery
Interrupted tasks are detected and re-queued on launch, with per-task timeouts raced via structured task groups, dedup keys, delayed execution, and cooperative cancellation.
Observable task lifecycle
An AsyncStream-based event pipeline exposes task lifecycles, plus a drain(timeout:) API designed for BGTaskScheduler background execution windows.
Fully tested and documented
25 Swift Testing cases run deterministically on macOS and visionOS via GitHub Actions CI, with full DocC documentation and an MIT license.
Technical Architecture
- Persistence model
- Every job is persisted in SwiftData and the execution state is rebuilt on launch, so work enqueued before a crash resumes after it. Because closures can't be persisted, the design separates what (a kind string plus a Data payload, stored) from how (handlers registered at launch) — the same architecture used by server-side job systems like Sidekiq, translated into Swift's actor model.
- Scheduler
- The scheduler fills a bounded concurrency pool by priority and retries failures on capped exponential backoff with jitter.
- Device-aware constraints
- Per-kind constraints (network, unmetered, charging) gate tasks without consuming retry attempts. Live NWPathMonitor integration re-evaluates the queue whenever connectivity changes — a task marked Wi-Fi-only simply waits, burning no attempts, until an eligible network path appears.
- Delivery semantics
- At-least-once delivery with crash recovery: interrupted tasks are detected and re-queued on launch. Per-task timeouts are raced against handlers via structured task groups, with dedup keys, delayed execution, and cooperative cancellation.
- Observability
- An AsyncStream-based event pipeline exposes task lifecycle events, and a drain(timeout:) API is designed for BGTaskScheduler background execution windows.
Challenges
The module/type name collision
Naming the flagship actor QEngineKit inside a framework named QEngineKit broke compilation in a non-obvious way: Swift resolves qualified names against the type, not the module, and SwiftData's macros expand to module-qualified names — producing baffling "self-referential" errors. The fix required understanding Swift's name-resolution rules and how framework module discovery works: the module was renamed QEngineKitCore, and since the compiler locates modules by framework bundle name, the product had to be renamed to match.
Racing timeouts against handlers correctly
Per-task timeouts use withThrowingTaskGroup to race the handler against a timer, which demands careful reasoning about which error surfaces first, how group cancellation propagates, and how to keep external cancellation (mapped to .cancelled) distinguishable from a timeout (a retryable failure).
Honest delivery semantics
A crash between handler success and persistence will re-run a task — no design eliminates that window. Rather than pretend exactly-once, the library documents at-least-once delivery prominently and requires idempotent handlers, mirroring how mature distributed job systems handle the same physics.
Deterministic async tests
Testing a concurrent, timing-driven scheduler without flakiness meant building test infrastructure: actor-based gates to hold tasks mid-flight, polling assertions instead of fixed sleeps, in-memory SwiftData containers per test, and millisecond-scale retry delays — 25 tests run in roughly 0.3 seconds.
Scheduler correctness bugs found in review
Retry-wake timers accumulating unboundedly, fixed with a single deduplicated wake task, and silent SwiftData save failures, surfaced through os.Logger.
Lessons Learned
- Shipped v1.0.0 as a public SwiftPM package under the MIT license, with 25/25 tests green on macOS and visionOS and CI running on every push via GitHub Actions.
- Full DocC documentation — an API reference plus a "Setting Up QEngineKit" integration guide — makes the library approachable for other developers adopting it.
- A clean public API (one actor, one protocol for dependency injection and mocking, two constraint types) came directly out of treating naming and module boundaries as a first-class design concern, not an afterthought.