Case study / Summer 2026
TaskForge
Coordinate background work through short PostgreSQL transactions, renewable ownership, and a durable record of every attempt.
- Authority
- PostgreSQL tasks + attempt history
- Execution
- Go workers · one handler per process
- Recovery
- Go schedulers · leases + retries
System architecture
Background work must outlive an HTTP request and remain inspectable when a process fails. TaskForge separates admission, execution, and lifecycle maintenance around one durable database. Its current handlers are predefined synthetic/demo handlers, not arbitrary uploaded code.
PostgreSQL coordinates independent processes
01 / Admission
React console → FastAPI
The console uses a same-origin Nginx API proxy. FastAPI validates submissions and reads task, worker, and attempt history.
Submit / inspect / cancel ↔ PostgreSQL ↓
02 / Authority
PostgreSQL tasks + numbered attempts
Committed rows hold priority, schedule, owner, lease, current state, outcomes, and the history of each execution attempt.
Claim / renew / finalize ↔ workers · Recover / promote ↔ schedulers ↓
03 / Execution & maintenance
Go workers + Go schedulers
Each worker runs one handler at a time, with independent heartbeat and lease-renewal loops. Schedulers recover expired ownership and promote due retries; replicas coordinate through row locks.
Submission & idempotency
Admit a logical task
FastAPI validates task type, payload, queue, priority, total-attempt budget, and optional schedule. A committed task starts QUEUED with no attempts. An optional idempotency key makes uncertain client retries safe for admission.
Arbitrate matching replays
A global partial unique index reserves the key. A SHA-256 fingerprint covers the canonical request, including queue, priority, budget, and normalized schedule. Concurrent inserts resolve to the stored task: matching fingerprints return its current state; different requests with the same key conflict.
Keep the guarantee scoped
Keys survive process restarts while their task rows persist; there is no key TTL or tenant scope. Admission deduplication does not deduplicate external handler effects, and replaying a completed task does not create new work.
Competing workers
Workers select due QUEUED tasks with attempts remaining, ordered by descending priority, then creation time and ID. A short transaction uses FOR UPDATE SKIP LOCKED so replicas can claim different rows without a central dispatcher.
Worker A locks task X
It sets RUNNING, assigns its owner and lease, increments the attempt number, and inserts the matching RUNNING attempt. These changes commit together.
Worker B skips X
While X is locked, B can claim another eligible row or return no work. If A rolls back, X stays QUEUED without that attempt and becomes available to a later poll.
Execute after commit
The handler runs outside the claim transaction. Completion locks the task again and checks owner, attempt number, RUNNING state, and valid lease before committing task and attempt outcomes together.
Priority is a candidate-selection policy, not strict FIFO or fairness. A locked higher-priority task can be bypassed, sustained high-priority arrivals can starve others, and handlers are not preempted. Unique (task_id, attempt_number) prevents duplicate history identities, not repeated business operations.
Success & retry walkthrough
Successful attempt
QUEUED → RUNNING
Claim commits ownership and attempt N. The worker then invokes its allowlisted handler.
Renew while executing
An independent loop extends task ownership. Process heartbeats report liveness separately.
RUNNING → SUCCEEDED
Guarded completion persists task result and SUCCEEDED attempt output together, then clears ownership.
Retryable failure → success
Attempt N → FAILED · task → RETRYING
A typed retryable error records the failed attempt and schedules the same logical task, if its total-attempt budget remains.
RETRYING → QUEUED
The scheduler promotes it once due. Promotion creates no attempt; a later worker claim creates N + 1.
Next attempt → SUCCEEDED or FAILED
Success finalizes normally. Ordinary errors are terminal; a retryable error at the last allowed attempt makes the task FAILED.
Default backoff starts at 2 seconds, doubles by failed-attempt number, applies 20% jitter, and caps at 300 seconds. The default budget is three total attempts, including the first claim and crash replacements. scheduled_at is earliest eligibility, not an exact start-time promise; promotion, polling, and contention add delay.
Logical task identity remains stable across retries and recovery. Numbered attempts retain RUNNING, SUCCEEDED, FAILED, or ABANDONED history. A failed attempt and a failed logical task are different outcomes.
Lease renewal & recovery
Lease and heartbeat answer different questions
Defaults are a 30-second task lease renewed every 10 seconds, and a process heartbeat every 5 seconds. A heartbeat shows recent communication; it neither proves useful progress nor revokes a task. Ownership comparisons use database time.
Expired ownership triggers recovery
A crash stops renewals but causes no immediate state transition. Schedulers lock expired RUNNING tasks and matching attempts, mark attempts ABANDONED, and requeue within the remaining budget—or mark the logical task FAILED when exhausted. Crash recovery requeues directly rather than applying retry backoff.
Guard state; fence effects separately
The owner and attempt number reject stale database completion after replacement. A paused or partitioned old handler can still overlap a replacement, or have completed a remote effect before losing ownership. External idempotency or destination fencing is needed for stronger effect guarantees.
TaskForge supports bounded reattempt semantics, not exactly-once execution. Eventual success is not guaranteed: budgets can exhaust, healthy database/workers/schedulers are required, and a hung handler that keeps renewing can remain active indefinitely.
Benchmark methodology & results
The supplied audit revalidated saved historical E1–E6 bundles through their existing trust evaluators; it did not rerun the workloads. Accepted runs identify clean source commits, container images, hashed artifacts, independent reset blocks, and durable task/attempt evidence. Trial-only Prometheus deltas reconcile with exact counts; 100 warmup tasks are excluded.
E1 uses 5,000 no-op tasks per trial; E2 uses 1,000 synthetic 50 ms waits per trial. Both test 1, 4, 8, and 16 worker processes across three independently reset blocks: 12 trials each. The runs contain 60,000 and 12,000 tasks respectively.
Recorded environment: Apple M4 Pro · 12 logical CPUs · 24 GiB RAM · local Docker.
Processing tasks/sec = logical tasks / (latest attempt finish − earliest attempt start)
Charts show medians of per-trial processing throughput. Speedup is the ratio of aggregate medians; parallel efficiency is speedup divided by worker count. Each chart has its own zero-based scale.
E1 / No-op coordination
median processing tasks/sec
- 1 worker779.748 tasks/sec
- 4 workers1284.015 tasks/sec
- 8 workers1279.605 tasks/sec
- 16 workers1214.429 tasks/sec
E2 / Synthetic 50 ms waits
median processing tasks/sec
- 1 worker18.764 tasks/sec
- 4 workers74.668 tasks/sec
- 8 workers149.442 tasks/sec
- 16 workers297.264 tasks/sec
- E5 / Fail-once retries
- 3,000 tasks · 6,000 attempts
Three trials; 10 workers and 3 schedulers; fixed 100 ms retry/promotion configuration. Exactly FAILED → SUCCEEDED histories, with zero duplicate identities or stranded leases in these runs.
- E6 / Hard-kill recovery
- 30 abandoned attempts replaced
Three trials of 1,000 synthetic 500 ms waits; 20 workers, 3 schedulers, 10 killed owners per trial, and 5-second leases. Every captured abandoned attempt had exactly one later successful replacement.
E6 median-trial p95 recovery lag was 36.681 ms after lease expiration—not after process death, and not a pooled p95. The timestamp is recorded during the recovery SQL path and persisted on commit; it is not exact commit-ack latency. Median kill-to-final-drain was 25.271239 seconds.
Single-host Docker results on synthetic workloads do not establish production capacity or universal worker scaling. Three blocks do not cover broader hardware variance. Zero duplicate recovery effects refers to durable recovery-state transitions; these handlers do not test payment, email, or other remote-effect deduplication.
Historical bundles and reports were retained locally as ignored artifacts and may be absent from a fresh clone. This page uses the supplied manual’s evidence summaries; no public raw-artifact availability or fresh benchmark run is claimed.
Tradeoffs & lessons
Short transactions, explicit ownership
Release database locks before slow handler work while keeping claim and history consistent.
Remote execution is outside the transaction. Leases and attempt epochs protect database state, not external effects.
Treat destination idempotency as a separate part of handler design.
One durable authority
PostgreSQL can commit admission, ownership, outcomes, and history together.
Polling adds idle queries and discovery delay; write contention and database availability bound scale.
Measure the workload and database pressure before increasing worker counts or adding a broker.
Priority and bounded attempts
Prioritize urgency and stop repeated failures within an explicit budget.
No fairness aging; low priorities can starve. Exhaustion preserves FAILED history without a dead-letter replay facility.
Make scheduling and recovery policy visible instead of promising unconditional progress.
Repository
Explore the API, Go workers and schedulers, SQL migrations, console, and benchmark harness in the supplied repository. Architecture details reflect manual revision ecbb2fe, dated September 3, 2026; benchmark source revisions are historical and listed with the charts.