Our execution layer could run exactly one thing: Nextflow, wired to our internal conventions so tightly that "support WDL" was a rewrite, not a feature. A researcher asked for Snakemake and I realised the honest answer was "not without a month of work." That answer bothered me enough to spend the month building something that wouldn't need it again.
Conquest is what came out. You POST a workflow (Nextflow, WDL, or Snakemake), it runs on Kubernetes, and you get results and logs back through GA4GH-standard APIs. I've been building it since March and it's the most fun I've had on a project in a long time.
The constraint that shaped everything
I wanted it language-agnostic, exposing standard APIs (WES for submitting workflows, TES for individual tasks), running as a single binary, deployable on its own. That last one did the most work. When there's no service mesh to hide behind, you can't paper over a messy design with infrastructure. The binary has to make sense by itself.
It takes a --role flag: api, controller, or all. Production runs several API replicas behind a load balancer and a controller or two. Local dev runs --role all and everything lives in one process. The API is a chi HTTP server implementing GA4GH WES v1.1, with a generated OpenAPI validator so the spec, not my code, decides what's a valid request.
The controller is a goroutine polling Postgres for runs in actionable states (QUEUED, RUNNING, CANCELING) every five seconds. Pull-based, not event-driven. I'll defend that choice: if the controller crashes and restarts, it reads the current state and continues, no event replay, no missed messages. The cost is up to five seconds of latency between submission and start, which is a rounding error next to workflows that run for hours.
One interface, three very different engines
Each language needs different handling, so the per-language logic hides behind a deliberately small interface:
type Adapter interface {
Prepare(ctx context.Context, run *model.Run, workDir string) error
BuildJob(run *model.Run, workDir string) (*batchv1.Job, error)
ParseOutputs(run *model.Run, workDir string) ([]model.Output, error)
ChildPods(ctx context.Context, run *model.Run) ([]corev1.Pod, error)
}Prepare and BuildJob are what you'd expect: stage inputs, write the engine's config, hand back a Kubernetes Job that runs nextflow run or miniwdl run or snakemake --kubernetes. The Job's restartPolicy is Never, because the engines have their own retry logic and the last thing I want is Kubernetes restarting a half-finished pipeline underneath one of them.
ChildPods is where my mental model broke and had to be rebuilt. I assumed a running workflow was one pod. It isn't. Nextflow with the k8s executor spawns a pod per process invocation, so a single "running" job can be fifty pods scattered across the cluster. The controller finds them with a label selector each engine sets on its children (Nextflow uses nextflow.io/runName) and aggregates their statuses to decide whether the parent is still RUNNING, COMPLETED, or FAILED. I found this out the way you'd expect: a job my code reported as healthy had thirty dead child pods it wasn't looking at.
Why the state machine lives in Postgres, not a CRD
Every run is a row. Transitions go QUEUED -> INITIALIZING -> RUNNING -> COMPLETE | EXECUTOR_ERROR | SYSTEM_ERROR | CANCELED, enforced by a validTransitions map in Go. Every change is one UPDATE with the current state in the WHERE clause, so two controller replicas can't both win the same transition.
The Kubernetes-native move here is a CRD with a custom controller, and I looked hard at it. Postgres won on the query patterns. "Give me every run in state X, older than Y, created by user Z" is a WHERE clause and an index, not a custom informer with a cache to keep warm. And the controller already needs a database connection for the API, so choosing Postgres added nothing new to the dependency list. Choosing a CRD would have. On a small team, "no new moving parts" beats "the idiomatic pattern" most days.
Logs go in a JSONB column that accumulates controller lines, child pod events, and engine stdout/stderr. The WES GET /runs/{id} response has a run_log field, and the column maps onto it almost directly.
The part I badly underestimated: file staging
I budgeted a day for input handling. It took closer to a week. Workflows reference input files, and those files arrive as:
- HTTP/HTTPS URLs, to download and stage
s3://URIs, already staged, or in another bucket needing a cross-bucket copy- Inline base64 content in the WES submission, to write out
- Relative paths pointing at other files in the same bundle
Every one of those is a different code path, and workflows mix them freely in a single submission. The staging layer resolves all four into one consistent prefix, s3://{bucket}/runs/{run_id}/inputs/. Downloads run through a worker pool because genomics inputs are routinely tens of gigabytes, each one checksummed with MD5 (S3 uses it for ETag verification) and retried with backoff.
Cleanup runs on terminal transitions: delete the intermediate prefix, keep the outputs. It fires in a goroutine so it never blocks the state-transition response. That's the kind of detail you only add after the first time cleanup blocks an API call and a user watches a spinner for ten seconds.
Where it stands
It's honestly not done. Every run still shares one namespace and service account, so multi-tenant isolation is missing. There are no per-user CPU/memory quotas yet. The TES layer works but only internally; the public API isn't there. Those aren't polish, they're the difference between "runs my workflows" and "runs yours safely," and I haven't decided which to do first.
It's MIT-licensed and the README goes deep on architecture on purpose, because I'd rather someone deploy it and tell me where I'm wrong than star it and move on. If you run scientific workflows and any of this sounds like a problem you also have, I'd like to hear how you solved the multi-tenancy piece. I'm still staring at it.