Skip to content

Jepsen's Conductor tests: lost workflows and duplicate task runs

By Quang Rau

Published
Reading time
7 min read

Jepsen, a testing group that injects faults into systems, tested Orkes Conductor, a workflow engine, and found three serious bugs. Some workflows never ran, some disappeared, and some were marked complete without a single task running. The three serious bugs are fixed in newer versions, so the design lesson matters more than any one version.

What Conductor does

Conductor runs workflows made of small tasks. You describe the tasks and their order, and the engine runs them, passes data between them and retries failures. Netflix built the first version, and it was open sourced in 2016, as Jepsen’s report explains. The open source project, Conductor OSS, uses the Apache 2.0 license. Its GitHub page showed about 32.4k stars when I read it today.

The servers keep no data of their own between requests. Workflow definitions and execution records live in a database such as Postgres. A queue, often Redis (an in-memory data store), hands work to the workers. Workers poll the servers, run tasks and report the results.

Conductor promises at-least-once delivery. A message arrives one or more times, so none is lost, but duplicates can happen. Its promise about results is narrower: once a successful output is visible, nobody should see a different one.

How Jepsen tested it

Jepsen built a cluster of ten Debian machines. One node ran Postgres. Redis ran as a single node or as a cluster. The other nodes split between the servers, standard workers and a custom worker written with the Java software development kit (SDK).

The test workflow had one task. That task slept for up to 30 seconds, then returned a random output and logged its run. Because every run was logged, the checker knew exactly what happened.

Each test had three phases. In the main phase, the harness started workflows while it killed and paused processes and cut network links. Those cut links are network partitions. In the recovery phase, it healed the faults and waited for the queue to empty. In the final phase, it read every workflow it had seen and checked for a long list of wrong behaviours.

The scope is narrow, and the report says so. The main tests used Postgres with Redis and single-task workflows. They did not cover failure workflows, high network latency or clock skew. Jepsen also says it can prove that bugs exist, but it cannot prove that none exist.

What broke

Lost and stuck workflows

  • Abandoned workflows. A workflow stayed in RUNNING forever and its task never ran. Conductor wrote the record to Postgres before it added the workflow to the decider queue. The sweeper, the background process that checks running workflows, read only that queue, so it never saw the record. Version 5.5.0 added a re-queue process.
  • Lost workflows. A partition between a server and Postgres made one server save a workflow, fail to enqueue it and then delete the record. The workflow was visible for 74 seconds. After that, getWorkflow returned null for the rest of the test. Version 5.5.0 also stopped deleting executions when a step fails.
  • Completed but never run. When Postgres crashed or got partitioned, a retry needed two separate writes. The first wrote a new task record. The second marked the old task as superseded, meaning replaced by the newer record. A failure between the two left no new task. The sweeper then saw only finished tasks and set the workflow to COMPLETED. The fix arrived in 5.5.6. Jepsen notes that one database transaction, which applies both writes or neither, would have prevented it, but the two writes sit in different layers of the code.
crash or partition herestartWorkflow callWrite workflow record toPostgresEnqueue in decider queueRecord exists, queue entrymissingSweeper reads only thequeueWorkflow stays RUNNINGforever

Tasks running twice

Healthy clusters ran the same task on two workers at once. Conductor uses a lock with a lease, meaning the lock expires after a set time so a crashed holder cannot block others forever. With the default one-minute lease, two servers, three workers and about 1,100 workflows a second, this happened roughly once every six hours. Cutting the lease to one second made it happen every few minutes. A race in the sweeper made the problem worse: it checked a task’s state, and that state changed before it acted on the result. Version 5.5.6 fixes this too.

Some workflows also stalled. A server cut off from Redis could leave a few workflows idle until the response timeout ran out. That timeout is how long Conductor waits for a worker to report back before it reschedules the task. The default is one hour. Jepsen accepted this as designed behaviour and did not ask for a change.

Confusing start errors

Calls to startWorkflow could return a null ID, a SQL error or a duplicate-key error. One SQL error came from a race while the database rotated its date-based table partitions (tables split by date), where the code built INSERT INTO null. Version 5.5.6 fixes that and the duplicate-key case.

The worst case was a silent one. In one run, 1,752 of 25,920 calls to startWorkflow reported a failure, yet the workflow had been created. A client that retries on that error can start the same workflow twice.

What others say

A short Hacker News thread, three comments when I checked, raised one gap. A reader said idempotency keys (unique IDs that let a server spot and skip a repeated request) were ignored on a self-hosted install. The report does not test them. I did not verify the rest of that thread, which is here.

Where I’d try it

GonnaCrash is a system design game where the system is going to crash, and the player has to fix it. A level could add the faults Jepsen used: kill the workflow server in the middle of a run, then cut its link to the database. The score would count jobs that never finish and jobs that finish twice.

My take

If you run Conductor 5.x, upgrade to 5.5.6 first. Jepsen recommends it, and it covers all three serious bugs.

Then review your own code. I would take three rules from this:

  1. Make every task idempotent, meaning running it twice gives the same result. Jepsen’s flight example shows the risk: a late retry can book a flight again after the cancel. Pass idempotency keys to downstream services.
  2. Before you retry a start call, check whether the workflow exists. Look it up with an ID you chose yourself.
  3. Keep your own record of intent. Before you call the engine for a payment or a booking, write the intent to your own database, then reconcile later.

Orkes funded the work, and the report says so. The test harness is public, and the report says the suite could measure OSS Conductor with some work. I would trust the results more once someone runs that suite against the open source build.

Storage is part of the design too. The report says Redis can lose writes that the server had already confirmed to the client, during a network partition. It also says some Postgres replication setups, where a second server copies the data so it can take over, can lose writes the database had reported as saved. The engine is only as safe as the stores under it.

My recommendation: upgrade to 5.5.6, then check every task and every start call for safe repeats before a workflow handles money or bookings. Happy coding!

Sources

  1. Jepsen: Orkes Conductor 5.3.2 (opens in new tab)
  2. jepsen-io/conductor: Jepsen tests for Orkes Conductor (opens in new tab)
  3. conductor-oss/conductor on GitHub (opens in new tab)