+1 (478) 654-9062

The Deploy Nobody Documented: CI/CD, Reproducible Builds, and Rollback for an Inherited Meteor App

We have written here about build times, bundle size, hosting choices, and data migrations. This post is about the thing that sits between all of them: the release process. It is, in our experience, the single least-documented part of an inherited Meteor app.

The pattern is familiar. There is a deploy. It works. One person runs it, usually from a laptop, often with a shell script that has mup deploy or meteor build --server-only in it somewhere. Nobody has rebuilt last year's release from source in years, and nobody is certain they could. That is not negligence — it is what happens when the team that set it up has moved on and the thing kept working. But it quietly caps what you can do next. You cannot safely do a Meteor 3 migration, or a strangler migration off Meteor, if you cannot reliably produce and reverse a build.

This is the work we do early in most engagements, before touching application code.

Step one: find out what a build actually depends on

A Meteor build pulls from more inputs than a plain Node app, and an inherited one usually pins only some of them. Write down the real state of all six:

  • .meteor/release — the Meteor version. Pinned by definition. Good.
  • .meteor/versions — the resolved versions of every Atmosphere package, direct and transitive. This is the Atmosphere lockfile, and it is committed by default. Confirm it actually is committed; we have seen it gitignored by someone who mistook it for generated noise.
  • .meteor/packages — the direct Atmosphere deps, often unpinned (accounts-password with no version).
  • package.json / package-lock.json — the npm half. Check the lockfile exists, is current, and is lockfileVersion 2 or 3. A missing lockfile means your build is whatever npm felt like resolving that morning.
  • The Node version — bundled by Meteor, so determined by .meteor/release for the build, but you still need the right Node in the runtime image. meteor node --version tells you which one.
  • Local packages in packages/ — source, so they travel with the repo, but they may themselves have unpinned deps.

The test for whether you are pinned: check out the commit that produced the currently running release, build it on a clean machine, and diff the output's dependency manifest against production. If the versions match, you are reproducible. If they do not, find out which input drifted and pin it. This exercise takes an afternoon and it is the highest-value afternoon in the whole program of work.

Step two: move the build into CI, unchanged

Resist the urge to improve the build while you are moving it. Port the existing steps as-is into a CI job, verify it produces a bundle that runs, and only then start changing things. Two variables at once is how a week disappears.

A Meteor build in CI is just:

meteor npm ci
meteor build --server-only --directory /build/output

Then cd /build/output/bundle/programs/server && npm install --omit=dev to install the server's native and production deps against the target Node version. --server-only skips mobile builds; if the app has a Cordova target, that is a separate pipeline with its own signing problems (we wrote about inherited Cordova apps separately — the short version is that it belongs in its own job, not bolted onto the web deploy).

Two CI specifics that cost people a day each:

Cache ~/.meteor. Meteor's package cache is large and the download is slow. Key the cache on the hash of .meteor/versions and .meteor/release, not on a branch name. An uncached cold Meteor build in CI can run ten to twenty minutes; a warm one is a fraction of that.

Give it memory. meteor build is a Node process doing a large minification pass and it will fail with an out-of-memory error on a small runner, usually with a stack trace that implicates something irrelevant. Default CI runners are frequently too small for a 2015-era app with a decade of accumulated packages. Raise the runner size or TOOL_NODE_FLAGS=--max-old-space-size=4096 before you go hunting for a code problem.

Step three: build a container you could rebuild next year

A two-stage Dockerfile is the usual shape: a builder stage with Meteor installed that produces the bundle, and a slim runtime stage containing only Node plus bundle/. The runtime image needs no Meteor at all — the output of meteor build is a plain Node application.

The things that make it still rebuildable in three years:

  • Pin the base images by digest, not by tag. node:20-slim means something different every month; node:20-slim@sha256:... does not.
  • Match the runtime Node major to what Meteor bundled. Meteor 2.x lines ran on Node 14; Meteor 3.x runs on Node 20 and later. Running the bundle on a different major than it was built for produces native-module errors that read like application bugs.
  • Rebuild native modules in the image, not on the host. bcrypt and friends compile against the exact Node and libc of the machine that builds them. Doing that install inside the runtime image's base — or a stage built from it — is the difference between a clean start and a startup crash that only happens in production.
  • Tag by commit SHA, always. latest is not a release identity. If you cannot say which image is running from the git history alone, you do not have a rollback story.

Step four: decide where migrations run

This is the part that determines whether rollback is real.

If your deploy runs schema migrations automatically on app startup, every instance races to run them, and rolling back the code does not roll back the data. Our preference — and this is a trade-off, not a law — is to run migrations as a separate, explicit step before the new image goes live, and to require migrations to be backward-compatible with the previous release for at least one deploy cycle. Expand, deploy, backfill, contract: the new code can read the old shape, the old code can survive the new shape. We covered the data side of this in the post on changing data shape under a live app; here the point is narrower. If a deploy cannot be reversed by redeploying the previous image tag, it is not a deploy, it is a migration event, and it should be planned and announced like one.

Write down, for each release, the answer to one question: what is the exact command that puts the previous version back, and how long does it take? If nobody on the team can answer that in a sentence, that is the next thing to fix, ahead of any framework work.

Step five: the cutover details Meteor adds

Two Meteor-specific wrinkles on rolling deploys:

DDP is a long-lived WebSocket. When you replace an instance, every connected client drops and reconnects. Meteor handles this well — the client reconnects, resubscribes, and resumes its login token — but it is not free. On a busy app, replacing all instances at once produces a reconnect thundering herd against your database as every subscription re-initialises. Replace instances gradually and give connections time to redistribute.

Clients can be running old JavaScript. After a deploy, browsers that have not reloaded are talking to the new server with the old bundle. Meteor's hot code push mitigates this, but it is a reason method and publication changes should be additive across a release boundary: add the new method, deploy, move clients, remove the old one later. Same discipline as the data migrations, applied to the wire protocol.

What this is worth

None of this makes the app faster or adds a feature, which is why it tends not to get scheduled. What it buys is optionality. A pinned, reproducible, CI-built, rollback-capable deploy is the precondition for every larger decision in front of you: the async conversion is safe to attempt because you can reverse it; the package replacements are testable because the build is deterministic; the strangler migration can route traffic between two systems because you can deploy each of them on purpose.

If the assessment says keep the app as it is for another three years, you have a release process a new hire can run. If it says migrate, you have the foundation the migration runs on. It is the rare piece of work that is correct under every outcome, which is why we usually do it first.