A pipeline ran for eight hours and died at hour seven. Every intermediate file it had written went down with the pod, because our sync job only ran after the pipeline finished. The researcher got to start over from zero, and I got to explain why.
That failure is the whole reason RedRush exists. It watches a local directory and uploads changed files to S3-compatible storage as they appear. Not on a schedule, not on pipeline exit. Within milliseconds of the file hitting disk, via kernel filesystem events (inotify on Linux, kqueue on macOS). One Go binary, zero config files.
The setup we had before
Nextflow, the workflow engine we run for researchers, writes outputs to a local scratch directory during execution. Those files need to end up in object storage (MinIO in our case), sometimes mid-run so downstream steps on other nodes can pick them up. Our answer was a post-run s5cmd sync. Which works great, right up until the pipeline doesn't reach the post-run part.
RedRush flips it: run as a sidecar, start before the pipeline, upload continuously. If the pipeline crashes, everything written up to the crash is already in the bucket.
Filesystem events are messier than you'd hope
The Go implementation uses fsnotify, which wraps inotify and kqueue into a channel-based API:
watcher, _ := fsnotify.NewWatcher()
watcher.Add("/data/output")
for {
select {
case event := <-watcher.Events:
if event.Op&(fsnotify.Create|fsnotify.Write) != 0 {
debouncer.Submit(event.Name)
}
case err := <-watcher.Errors:
log.Error("watcher error", "err", err)
}
}Looks clean. The catch is that one file write from a bioinformatics tool generates a burst of events: CREATE when the descriptor opens, WRITE for each flush (there can be many), sometimes a CHMOD for good measure. Upload on every event and you'll ship a half-written 10 GB FASTQ file to S3 several times over. I did this in the first prototype. The bucket bill noticed before I did.
Debouncing
The fix is a per-file timer map. Event arrives for file X, start a 500ms timer for X. Another event for X before it fires, reset it. When the timer finally fires, the file has been quiet for 500ms and is probably done being written:
type Debouncer struct {
mu sync.Mutex
timers map[string]*time.Timer
out chan string
delay time.Duration
}
func (d *Debouncer) Submit(path string) {
d.mu.Lock()
defer d.mu.Unlock()
if t, ok := d.timers[path]; ok {
t.Reset(d.delay)
return
}
d.timers[path] = time.AfterFunc(d.delay, func() {
d.mu.Lock()
delete(d.timers, path)
d.mu.Unlock()
d.out <- path
})
}The 500ms default is tuned for bioinformatics tools that write large files in buffered chunks. Tools that fsync after every write (some database engines) want a shorter delay. It's a --debounce flag.
Is "the file went quiet" the same as "the file is complete"? No, and I want to be honest about that. It's a heuristic. It has held up in practice for our workloads, but a tool that pauses mid-write for longer than the debounce window would slip through. I haven't seen one do it yet.
The race that inotify hands you for free
Here's the part that actually cost me a weekend. inotify does not recurse. A watch on /data/output gives you events for that directory only, not for /data/output/sample_A/. Every subdirectory needs its own watch.
So you watch for directory CREATE events and add watches as directories appear. Except: between the moment the directory is created and the moment your watch lands on it, files can appear inside. Those files fire no events. They just silently don't exist as far as your watcher is concerned. For a pipeline that creates a fresh output directory per sample and immediately starts writing into it, that's not an edge case, that's the main case.
The fix is order-dependent:
case event := <-watcher.Events:
if event.Op&fsnotify.Create != 0 {
info, err := os.Stat(event.Name)
if err == nil && info.IsDir() {
watcher.Add(event.Name)
// Scan for files that arrived in the race window
filepath.WalkDir(event.Name, func(path string, d fs.DirEntry, err error) error {
if !d.IsDir() {
debouncer.Submit(path)
}
return nil
})
}
}Add the watch first, then scan. Files that appeared during the race get caught by the scan. Files that appear after the scan get caught by the watch. There's technically still a window between os.Stat and watcher.Add, but it's microseconds wide and I have genuinely tried and failed to trigger a miss.
Uploads
The upload path is a worker pool, four goroutines by default, reading from the debouncer's output channel. The S3 key mirrors the local path: /data/output/sample_A/aligned.bam becomes s3://{bucket}/{prefix}/sample_A/aligned.bam. Multipart uploads use 64MB parts with three parts in flight per file, so a 30 GB BAM file is about 480 parts uploaded three at a time.
Failed uploads go to a retry queue with exponential backoff and jitter (1s initial, 60s max, five attempts). After that, the path gets logged and the daemon moves on. I considered a dead-letter mechanism and decided against it: a log line plus the ability to run a manual sync covers the real failure case, and every component I don't build is a component that can't break.
Running it as a sidecar
The primary deployment is a Kubernetes sidecar sharing an emptyDir with the pipeline container:
spec:
containers:
- name: pipeline
image: nextflow/nextflow:24.04
volumeMounts:
- name: scratch
mountPath: /data/output
- name: redrush
image: anurag1201/redrush:latest
args: ["--watch", "/data/output", "--bucket", "results", "--prefix", "runs/$(RUN_ID)"]
volumeMounts:
- name: scratch
mountPath: /data/output
volumes:
- name: scratch
emptyDir: {}One detail that will bite you if you skip it: when the pipeline container exits, RedRush still has uploads draining from the debounce queue. On SIGTERM the daemon finishes in-flight uploads before exiting, so terminationGracePeriodSeconds needs to be long enough for the largest pending upload. Set it too low and Kubernetes will kill the sidecar mid-upload, which puts you right back in the "lost the last files" hole this tool was built to escape.
That callback is the test I hold RedRush to. The eight-hour pipeline that started all this? It crashed again a few weeks after we deployed the sidecar. This time the researcher lost nothing. That's the entire product.