---
title: "Small PRs to scientific infrastructure"
description: "Shipping a streaming zip endpoint to trs-filer, adding S3-compatible storage via boto3, and patching FOCA's MongoDB connection URI handling."
canonical: https://anuragxd.com/blogs/contributing-to-ga4gh-oss
date: 2025-08-20
---

# Small PRs to scientific infrastructure

*Published 2025-08-20 by Anurag Gupta — https://anuragxd.com/blogs/contributing-to-ga4gh-oss*

The third time I wrote the same file-by-file download loop against a TRS server, I stopped. GET one file, GET the next, stitch them together locally, repeat. It was tedious, and I was clearly not the first person to write it. Last August I spent a few weekends on [trs-filer](https://github.com/elixir-cloud-aai/trs-filer), the GA4GH Tool Registry Service implementation maintained by the ELIXIR Cloud & AAI group. I'd been around that community since my GSoC days, but back then as a mentor. This was the first time in a while I came back as a contributor. Most of what I shipped is boring. That's the point.

## The zip endpoint

TRS stores workflow descriptors, parameter files, and container recipes as individual objects. Before this change, every client that needed the full bundle had to call `GET /tools/{id}/versions/{version_id}/{type}/files` for each file type, then assemble them locally. So every TRS client reimplemented the loop I got sick of writing. Fixing it once, server-side, felt like the obvious trade.

The implementation is a streaming zip response using Python's `zipfile.ZipFile` writing into a `BytesIO` buffer. The tricky part is that trs-filer stores files in either MongoDB GridFS or on a local filesystem, depending on configuration. The zip handler had to abstract over both without knowing which one it was talking to:

```python
def get_tool_files_zip(tool_id: str, version_id: str, file_type: str):
    buffer = BytesIO()
    with ZipFile(buffer, 'w', ZIP_DEFLATED) as zf:
        accessor = get_file_accessor()  # GridFS or LocalFS, from config
        for file_meta in accessor.list_files(tool_id, version_id, file_type):
            data = accessor.read_file(file_meta.id)
            zf.writestr(file_meta.filename, data)
    buffer.seek(0)
    return Response(
        buffer.getvalue(),
        mimetype='application/zip',
        headers={
            'Content-Disposition': f'attachment; filename={tool_id}_{version_id}.zip'
        }
    )
```

The `FileAccessor` interface has two implementations. `GridFSAccessor` reads from GridFS using `gridfs.GridFS(db).get(file_id)`. `LocalFSAccessor` reads from a configured directory with `pathlib.Path.read_bytes()`. The zip handler stays ignorant of both.

I wrote the tests before I trusted the abstraction. A pytest fixture seeds a tool version with known files across both backends. The assertion unzips the response body and checks filenames and content against what was seeded:

```python
def test_zip_download(client, seeded_tool):
    resp = client.get(f'/tools/{seeded_tool.id}/versions/{seeded_tool.version}/WDL/files',
                      headers={'Accept': 'application/zip'})
    assert resp.status_code == 200
    with ZipFile(BytesIO(resp.data)) as zf:
        assert set(zf.namelist()) == {'workflow.wdl', 'inputs.json'}
        assert zf.read('workflow.wdl') == b'version 1.0\n...'
```

The CI matrix runs this test against both GridFS and local filesystem configurations. Connexion, the OpenAPI framework trs-filer uses, routes on the `Accept` header: `application/zip` goes to the new handler, `application/json` to the existing one. Same URL, two behaviors, no breaking change for anyone already parsing JSON.

## MinIO as a storage backend

The second PR added S3-compatible object storage as a file storage option. The two existing backends both had a ceiling I kept hitting. GridFS has a 16MB document limit per chunk. You can store larger files by splitting across chunks, but that's GridFS doing it internally, and debugging GridFS cursor issues is its own afternoon. Local filesystem simply doesn't work in multi-replica deployments, because replicas don't share disk. Neither is a real answer for a 2 GB container image.

The S3 backend uses `boto3` with a configurable endpoint URL, so it points at MinIO, Ceph, or actual S3 without code changes:

```python
class S3FileAccessor:
    def __init__(self, config):
        self.client = boto3.client(
            's3',
            endpoint_url=config.get('S3_ENDPOINT'),  # MinIO, Ceph, etc.
            aws_access_key_id=config['S3_ACCESS_KEY'],
            aws_secret_access_key=config['S3_SECRET_KEY'],
        )
        self.bucket = config['S3_BUCKET']

    def store_file(self, tool_id, version_id, file_type, filename, data):
        key = f"{tool_id}/{version_id}/{file_type}/{filename}"
        self.client.upload_fileobj(
            BytesIO(data),
            self.bucket,
            key,
            Config=TransferConfig(multipart_threshold=8 * 1024 * 1024)
        )

    def read_file_stream(self, tool_id, version_id, file_type, filename):
        key = f"{tool_id}/{version_id}/{file_type}/{filename}"
        response = self.client.get_object(Bucket=self.bucket, Key=key)
        return response['Body']  # StreamingBody, reads on demand
```

Downloads stream straight from S3 to the HTTP response through the `StreamingBody` wrapper, so the server never buffers a 2 GB Singularity image in memory. That detail mattered more than I expected. The first version I sketched read the whole object into a bytes buffer, which is fine on a laptop and a disaster on a shared node with other services fighting for RAM. Uploads use multipart with an 8 MB threshold; anything larger gets split into parts and uploaded with configurable concurrency.

## FOCA's MONGO_URI patch

[FOCA](https://github.com/elixir-cloud-aai/foca) is the Flask microservice framework most of these GA4GH services are built on. It assembled MongoDB connection strings from individual config fields: `host`, `port`, `databaseName`. That's fine until you point it at a managed provider. Atlas and Azure CosmosDB's Mongo API hand you a full connection URI with auth credentials, replica set options, `tls=true`, `retryWrites=true`, and read preferences already baked in:

```
mongodb+srv://user:pass@cluster0.abc123.mongodb.net/?retryWrites=true&w=majority&tls=true
```

You can't decompose that into host and port without losing the SRV lookup, the TLS setting, and the write concern. I tried, briefly, before admitting the assembly approach was the wrong shape for anything managed.

The fix was two commits. First, accept a `MONGO_URI` environment variable that bypasses assembly entirely:

```python
def create_mongo_client(config):
    uri = os.environ.get('MONGO_URI')
    if uri:
        client = MongoClient(uri)
        # Extract database name from URI if not explicitly configured
        db_name = config.get('databaseName') or uri_parser.parse_uri(uri).get('database')
    else:
        # Original host/port assembly path
        client = MongoClient(host=config['host'], port=config['port'])
        db_name = config['databaseName']
    return client[db_name]
```

`pymongo.uri_parser.parse_uri` pulls the database name out of the URI path component. If the URI has none, which is valid, the explicit `databaseName` config takes over. The old path is completely untouched, so existing deployments don't notice the change at all. That was the whole design constraint: add the new door without moving the old one.

None of this is glamorous. But this software runs federated genomics analysis across European compute centers, maintained by maybe a dozen part-time people. A zip endpoint or a connection string fix has a weirdly high impact-to-effort ratio next to a drive-by PR on some huge project, where your change sits in a queue behind 400 others and nobody's short-handed enough to care. Boring is where the payoff is. I keep coming back for exactly that.

---

Markdown representation of https://anuragxd.com. Request any page with `Accept: text/markdown`, or append `.md` to its URL. Index: https://anuragxd.com/llms.txt · API: https://anuragxd.com/api/index.json