Jan 18, 2026

What actually happens between git push and a running service

When I started deploying my own work, I treated “deploy” as one step: push code, something happens, site is live. That was enough until something failed in the gap: a container that started but could not serve traffic, a health check that lied, a process killed mid-request because nobody handled SIGTERM, or a service that worked on my laptop and OOM’d in staging.

Understanding what happens between git push and a user hitting your API changed how I write code, configure servers, and debug production. This is the full map, not tool-specific trivia but the mechanics underneath that stay true whether you use Vercel, Railway, Kubernetes, or a VPS.

The pipeline in one picture

git push
  → CI: install, lint, test, build artifact
  → image build (optional): Dockerfile layers
  → registry push
  → deploy orchestrator pulls image / artifact
  → new process starts (container or systemd unit)
  → bind port, pass health checks
  → load balancer adds instance to pool
  → user request → DNS → TLS → proxy → your app → database

Every arrow is a place things fail silently.

Step 1: CI is a contract

Your pipeline proves the repo is deployable. Minimum I trust:

jobs:
  check:
    steps:
      - install dependencies (frozen lockfile)
      - lint
      - typecheck
      - unit tests
      - integration tests (if feasible)

  build:
    needs: check
    steps:
      - build application
      - build container image
      - scan image for CVEs (optional but valuable)

  deploy:
    needs: build
    steps:
      - push image to registry
      - roll out with health gate
      - smoke test critical paths

Frozen lockfile (pnpm install --frozen-lockfile). Without it, today’s green build is tomorrow’s failure because a transitive dependency changed.

Cache node_modules or Docker layers in CI, but never skip reproducibility for speed.

Step 2: Docker, and what an image actually is

FROM node:22-alpine AS build
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

FROM node:22-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s \
  CMD wget -qO- http://127.0.0.1:3000/health/live || exit 1
CMD ["node", "dist/server.js"]

Multi-stage builds keep build tools out of the production image. Smaller images pull faster, start faster, less attack surface.

Layers: each RUN, COPY is a layer. Order matters: copy lockfile before source so dependency layer caches across builds.

Run as non-root: USER node limits damage if the process is compromised.

Step 3: Linux process model

When the container starts, your app becomes a process with a PID, memory space, file descriptors, and scheduler time.

PID 1 and signals

In containers, your app is often PID 1. Responsibilities:

  • Handle SIGTERM for graceful shutdown
  • Reap zombie child processes (Node does not always do this, so use dumb-init or tini as entrypoint if you shell out)
const server = app.listen(port, "0.0.0.0");

process.on("SIGTERM", async () => {
  logger.info("sigterm_received");
  server.close(); // stop accepting new connections
  await drainInFlightRequests(); // give existing requests time
  await db.$disconnect();
  await redis.quit();
  process.exit(0);
});

Orchestrators send SIGTERM, wait terminationGracePeriodSeconds, then SIGKILL. Without a handler, mid-transaction kills corrupt data.

File descriptors

Each TCP connection, DB pool connection, and open file uses a file descriptor. Default ulimit -n can be 1024. Under load:

Error: EMFILE, too many open files

Raise limits in container config. Fix connection leaks in code.

Memory

Term Meaning
RSS Physical RAM in use
VSZ Virtual address space (larger than RSS)
cgroup limit Hard cap in containers; exceeding it means an OOM kill

Node heap grows with traffic. A PDF generation job on the request path can exceed a 256MB container limit while working fine on a 16GB laptop.

Fix: move CPU/memory-heavy work to background workers with appropriate limits.

Step 4: How Node accepts connections (and why it matters)

Rough model:

  1. Kernel listens on port via listen()
  2. Incoming TCP connections accepted, and epoll / kqueue notifies readable events
  3. Node event loop reads HTTP, runs your handler, writes response
  4. One handler blocking the event loop blocks everything, including /health/live
// blocks entire process
const hash = bcrypt.hashSync(password, 12);

// yields to event loop between rounds
const hash = await bcrypt.hash(password, 12);

// still CPU-heavy; for bulk work, use worker_threads or separate service

Go uses goroutines. Python async has the GIL. Know your runtime’s concurrency model.

Step 5: Networking path to your app

User types api.example.com:

  1. DNS resolves to load balancer IP (TTL matters during failover)
  2. TLS handshake at load balancer or your app
  3. HTTP request forwarded to backend (often HTTP/1.1 or h2 behind proxy)
  4. Reverse proxy (nginx, Caddy, cloud LB) may buffer, timeout, rate limit

Common mistakes:

// wrong: only localhost, unreachable from outside container
app.listen(3000, "127.0.0.1");

// correct
app.listen(Number(process.env.PORT) || 3000, "0.0.0.0");

Trust X-Forwarded-For / X-Forwarded-Proto only from your proxy, not arbitrary clients, otherwise IP spoofing is trivial.

HTTPS usually terminates at the proxy. Your app sees HTTP on port 3000. Configure cookies Secure and HSTS at the edge.

Proxy timeouts: if your handler runs 120s but nginx proxy_read_timeout is 30s, clients see 502 while your app still runs.

Step 6: Health checks that tell the truth

Check Question On failure
Liveness Is the process alive? Restart container
Readiness Can it serve traffic now? Remove from LB pool 
app.get("/health/live", (_req, res) => {
  res.status(200).send("ok");
});

app.get("/health/ready", async (_req, res) => {
  try {
    await Promise.all([
      db.execute(sql`SELECT 1`),
      redis.ping(),
    ]);
    res.status(200).json({ ok: true });
  } catch (err) {
    logger.error("readiness_failed", { err });
    res.status(503).json({ ok: false });
  }
});

Liveness should be dumb. If it checks DB and DB blips, orchestrator restart-storms your entire fleet.

Readiness should reflect dependencies needed for real requests.

Startup probe: wait for migrations + warm connection pool before marking ready.

Step 7: Migrations in deploy pipeline

Running ALTER TABLE during peak traffic locks tables.

Expand/contract pattern:

Deploy 1: add nullable column phone_verified_at
Deploy 2: backfill, write from new code
Deploy 3: set NOT NULL, drop old column

Separate migration job before rolling app:

# run once per deploy, with advisory lock
pnpm db:migrate

Postgres advisory lock prevents two instances migrating simultaneously:

SELECT pg_advisory_lock(12345);
-- run migrations
SELECT pg_advisory_unlock(12345);

Step 8: Horizontal scaling

Adding instances works when HTTP layer is stateless:

Stateful (bad at scale) Stateless (good)
Session in server RAM Session in Redis / JWT
Uploads on local disk S3 / R2 / GCS
In-process cache without TTL Redis with tenant-scoped keys

Sticky sessions route same user to same instance, which hides state until that instance dies.

WebSockets need shared pub/sub (Redis) so any instance can message any connection, or accept stickiness with failover pain.

Autoscaling signals

  • CPU sustained above 70%
  • Request queue depth
  • p95 latency above SLO
  • Custom: queue consumer lag

Scale down slowly. Flapping costs money and clears warm caches.

Thundering herd on deploy

Ten new containers start → ten cold connection pools → ten migration attempts → DB connection spike.

Mitigations:

  • Gradual rollout: 10% → 50% → 100%
  • Pool size = (DB max_connections / max_instances) - overhead
  • Jitter in cron and cache TTL to prevent synchronized expiry

Step 9: Observability with logs, metrics, and traces

When deploy “succeeds” but users see errors, you need:

Structured logs

logger.info("request_complete", {
  requestId: req.id,
  method: req.method,
  path: req.path,
  status: res.statusCode,
  durationMs: Date.now() - start,
  tenantId: req.tenantId,
});

JSON logs. Searchable. Never console.log("here 3") in production.

Metrics

  • Request rate, error rate, latency histogram (p50, p95, p99)
  • DB pool: active, idle, waiting
  • Queue depth and consumer lag
  • CPU, memory per instance

Distributed tracing

OpenTelemetry trace ID propagated across services:

X-Request-Id / traceparent header

“Payment failed” across API → worker → Stripe webhook becomes one trace, not three log greps.

Step 10: Secrets and configuration

  • Secrets in vault / platform env, never in git
  • Different DB URLs per environment. Never point staging at production “to test quickly”
  • Fail fast at startup:
function required(name: string): string {
  const v = process.env[name];
  if (!v) throw new Error(`Missing env: ${name}`);
  return v;
}

Better crash at boot than serve 500 on every request.

Debug story: works locally, fails in staging

Dimension Local Staging
NODE_ENV development production
Database SQLite / local Postgres Managed Postgres, SSL required
Memory 16 GB laptop 512 MB container
Proxy timeout none 30s
Cold start N/A 2s JIT + empty cache

Symptom: API returns 502 after 30s.
Cause: report generation took 90s; proxy timed out.
Fix: async job + polling endpoint for result.

Symptom: container restart loop.
Cause: readiness checked DB before migrations ran.
Fix: init container or startup job ordering.

Symptom: OOM killed.
Cause: loaded entire CSV into memory.
Fix: stream processing.

Pre-production checklist

  1. Dockerfile builds from lockfile, runs non-root
  2. PORT from env, bind 0.0.0.0
  3. Separate /health/live and /health/ready
  4. SIGTERM handler drains requests and closes pools
  5. Migrations documented, with expand/contract for risky changes
  6. Structured logging with request ID middleware
  7. Staging mirrors production: same DB engine, TLS, similar data volume
  8. Load test one realistic endpoint before launch
  9. Runbook: how to roll back, who gets paged, where logs live

The point

Deploy is not magic. It is a sequence of mechanical steps (build, package, start process, bind port, pass health checks, receive proxied traffic) where small misunderstandings compound.

When you know the sequence, you stop guessing during incidents. Site “up” but logins fail? Check readiness, DB connectivity, migration version. Slow after deploy? Cold cache, connection pool, JIT warmup. Random 502s? Proxy timeout vs app timeout mismatch.

Tools change. Processes, networks, and operating systems behave the same underneath. That mental model is worth more than memorizing any single platform’s dashboard, and it stays useful across every stack you touch next.