Production engineering workshop · 45 MIN

DevOps / Cloud · Drain a service during deployment

Readiness, liveness and graceful shutdown solve different problems.

Readiness decides whether a service should receive new traffic. Liveness indicates whether restarting may help. On termination, fail readiness, stop accepting requests and allow a bounded drain before exit. Configure the platform’s grace period to exceed your drain deadline. Keep dependencies and secrets outside the image; log request IDs and latency without logging passwords or tokens.

First reproduce the normal path. Then force a failure at each boundary and inspect what remains true.

Read the example

import http from "node:http";
let draining=false;
const server=http.createServer((request,response) => {
  if (request.url==="/live") { response.end("alive"); return; }
  if (request.url==="/ready") { response.statusCode=draining?503:200; response.end(draining?"draining":"ready"); return; }
  if (draining) { response.statusCode=503; response.end("retry later"); return; }
  response.end("CodingNeed workshop");
});
server.listen(Number(process.env.PORT??3000));
function shutdown() {
  if (draining) return;
  draining=true;
  const deadline=setTimeout(()=>process.exit(1),10000);
  deadline.unref();
  server.close(error=>{ clearTimeout(deadline); process.exitCode=error?1:0; });
}
process.on("SIGTERM",shutdown);
process.on("SIGINT",shutdown);
// Production: close DB/queue connections after requests drain.
// Never include database passwords in an image or build argument.

Review the behavior in the stated project environment.

Your challenge

Run this Node service locally and in a non-root container. Send a request and terminate the process while it is active. Define readiness and liveness probes in your orchestrator and test deployment rollback with an intentionally failing readiness probe.

Solution cost: O(1) probe handling; drain bounded by a configured deadline. time · Active connections and per-request state; set admission and resource limits. space

Common trap

Using dependency health as liveness can restart every instance during a database outage, worsening recovery.

import http from "node:http";
let draining=false;
const server=http.createServer((request,response) => {
  if (request.url==="/live") { response.end("alive"); return; }
  if (request.url==="/ready") { response.statusCode=draining?503:200; response.end(draining?"draining":"ready"); return; }
  if (draining) { response.statusCode=503; response.end("retry later"); return; }
  response.end("CodingNeed workshop");
});
server.listen(Number(process.env.PORT??3000));
function shutdown() {
  if (draining) return;
  draining=true;
  const deadline=setTimeout(()=>process.exit(1),10000);
  deadline.unref();
  server.close(error=>{ clearTimeout(deadline); process.exitCode=error?1:0; });
}
process.on("SIGTERM",shutdown);
process.on("SIGINT",shutdown);
// Production: close DB/queue connections after requests drain.
// Never include database passwords in an image or build argument.

Further reading: Official documentation