Design a Code Deployment System: CI/CD, Canary and Rollback

Problem Statement

Design a code deployment system (like Spinnaker or AWS CodeDeploy) that takes code from a repository, builds it, runs tests, and deploys it to production using safe rollout strategies (canary, blue-green, rolling). The system must support automated rollback on error detection, artifact versioning, and multi-region deployment for thousands of microservices.

Requirements

Functional

• Build pipeline: on git push, compile code, run unit/integration tests, build container image, push to artifact registry
• Deployment strategies: canary (route 5% traffic to new version), blue-green (instant switch), rolling (gradual pod replacement)
• Automated rollback: detect error rate increase (>1% 5xx) during canary and automatically roll back
• Artifact management: version, tag, and promote container images through environments (dev -> staging -> prod)

Non-Functional

• Speed: Build + test completes in <10 minutes for most services
• Safety: No bad deployment reaches more than 5% of production traffic without human approval
• Scale: 5000 microservices, 500 deployments/day, multi-region (3+ regions)
• Reliability: Deployment system itself must be 99.99% available

Core Architecture

1. Build Service -- Triggered by git webhook. Pulls code from the repository, runs the build in an ephemeral container (Docker-in-Docker or Kaniko for rootless builds). Executes unit tests, integration tests, linting, and security scans in parallel stages. On success, builds a container image tagged with git SHA and pushes to the artifact registry (e.g., ECR, GCR). Build logs are streamed in real time.

2. Deployment Orchestrator -- Manages the deployment lifecycle. Receives a deployment request (service, version, strategy, target environment). For canary: creates a small deployment (5% of pods) running the new version behind the same load balancer, configures traffic splitting (Istio/Envoy), starts the monitoring window. For blue-green: provisions the green environment, runs health checks, switches the load balancer atomically.

3. Rollback Controller -- Monitors the canary deployment using metrics from the observability stack (Prometheus/Datadog). Compares error rate, latency p99, and custom health metrics between canary and baseline. If canary error rate exceeds baseline by >1% during the observation window (10-30 minutes), automatically rolls back by draining canary pods and restoring 100% traffic to the previous version. Pages the on-call engineer.

4. Artifact Registry and Promotion Pipeline -- Stores container images with immutable tags (git SHA). Promotion flow: an image built in CI is tagged "dev", manually promoted to "staging" after QA, then to "prod" after staging validation. Each promotion is audited (who promoted, when, approval chain). Old images are garbage-collected after 90 days.

Database Choice

PostgreSQL for deployment records (service, version, strategy, status, timestamps, rollback history), pipeline definitions, and audit logs. S3 for build logs and artifacts. Redis for pipeline status caching (which builds are running, queue depth) and distributed locks (prevent concurrent deployments of the same service). Kafka for deployment events consumed by the notification service (Slack alerts) and metrics aggregator.

Key API Endpoints

POST /api/v1/deployments
  -> Body: \{ service: "order-service", version: "abc123", strategy: "canary", canary_percent: 5, regions: ["us-east-1", "eu-west-1"] \}
  -> Returns: \{ deployment_id: "D-456", status: "IN_PROGRESS" \}

GET /api/v1/deployments/\{deployment_id\}/status
  -> Returns: \{ status: "CANARY_MONITORING", canary_error_rate: 0.2, baseline_error_rate: 0.15, time_remaining_min: 18 \}

POST /api/v1/deployments/\{deployment_id\}/promote
  -> Returns: \{ status: "ROLLING_OUT", progress: "5% -> 25% -> 50% -> 100%" \}

Scaling Insight

Progressive canary with automated promotion gates is the key safety mechanism. Instead of deploying to 5% and waiting for a human, the system uses a multi-stage canary: 5% for 10 minutes -> if healthy, 25% for 10 minutes -> 50% for 5 minutes -> 100%. At each gate, automated health checks compare canary vs. baseline metrics. A bad deploy is caught at 5% with only ~50ms of user impact (5% * average error duration). The entire rollout completes in 25 minutes with zero human intervention for healthy deployments.

Key Tradeoffs

Original DosiBridge notes for Design a Code Deployment System: CI/CD, Canary and Rollback

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design a Code Deployment System: CI/CD, Canary and Rollback is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Core Entity: For Design a Code Deployment System: CI/CD, Canary and Rollback, decide the core entity before choosing tools; otherwise the design looks clean but fails when code changes.
• Read/Write Path: For Design a Code Deployment System: CI/CD, Canary and Rollback, decide the read/write path before choosing tools; otherwise the design looks clean but fails when deployment changes.
• Contention Point: For Design a Code Deployment System: CI/CD, Canary and Rollback, decide the contention point before choosing tools; otherwise the design looks clean but fails when code changes.
• Abuse Path: For Design a Code Deployment System: CI/CD, Canary and Rollback, decide the abuse path before choosing tools; otherwise the design looks clean but fails when deployment changes.
• Recovery Workflow: For Design a Code Deployment System: CI/CD, Canary and Rollback, decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when code changes.

Design a Code Deployment System: CI/CD, Canary and Rollback correctness boundary

For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.

Design a Code Deployment System: CI/CD, Canary and Rollback scaling bottleneck

The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.

Production review question

If Design a Code Deployment System: CI/CD, Canary and Rollback suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?

Practical metric to track

Track code-deployment health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design a Code Deployment System: CI/CD, Canary and Rollback, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Design a Code Deployment System -- Reference
• Source: System-Design-Overview