CI/CD Best Practices for High-Velocity Teams

A deployment pipeline is only as good as the confidence it gives your team. If developers hesitate before merging, if deploys are events that require coordination, if rollbacks are manual and stressful — your pipeline is holding your team back. Here’s how we built Midnight’s CI/CD to ship dozens of times per day without breaking a sweat.

1. Make the Build Fast

The single most impactful improvement to any CI/CD pipeline is build speed. Every minute of build time is a minute a developer is context-switching to something else, which means another 15 minutes of ramp-up time when they come back.

Our targets:

• Lint + type check: under 30 seconds
• Unit tests: under 2 minutes
• Integration tests: under 5 minutes
• Full build + deploy: under 8 minutes total

We hit these targets through aggressive caching, parallel test execution, and incremental builds. The most impactful single change was switching from full rebuilds to incremental compilation — our build time dropped from 12 minutes to 3.

Key insight: Measure build time weekly. It creeps up slowly, and by the time someone complains, you’ve lost 5 minutes per build across the team.

2. Test in Layers

Not every test needs to run on every commit. We organize tests into three tiers:

• Tier 1 (every push): Lint, type check, unit tests. These run in under 2 minutes and catch 80% of issues.
• Tier 2 (every PR): Integration tests, E2E smoke tests. These run in 5 minutes and catch compatibility issues.
• Tier 3 (pre-deploy): Full E2E suite, performance benchmarks, security scans. These run in 15 minutes and gate production deploys.

This layering means developers get fast feedback on every push while still maintaining comprehensive coverage before production.

3. Deploy to Staging Automatically

Every merge to main triggers an automatic deploy to staging. No manual steps, no approval gates, no “schedule a deploy” rituals. The staging environment is always running the latest code.

This has two benefits:

• QA can test continuously without waiting for a “staging deploy window.”
• Issues surface faster because the delta between staging and production is always small.

We take this further with preview environments for every PR. Each pull request gets its own isolated environment with a unique URL. Reviewers can see the changes running, not just read the diff.

4. Canary Deploys to Production

We never deploy to 100% of production at once. Our deploy process:

1. Deploy to 5% of traffic (canary)
2. Monitor error rates, latency, and key metrics for 10 minutes
3. If metrics are healthy, roll out to 25%, then 50%, then 100%
4. If any metric degrades, automatically roll back

The automatic rollback is critical. Human rollback decisions are slow and error-prone because they require someone to be watching. Automated rollbacks based on pre-defined thresholds catch issues in minutes, not hours.

5. Infrastructure as Code, No Exceptions

Every piece of our infrastructure is defined in code: servers, databases, networking, DNS, monitoring alerts, even the CI/CD pipeline configuration itself. Nothing is configured through a web UI.

This means:

• Reproducibility. We can recreate any environment from scratch in 20 minutes.
• Auditability. Every infrastructure change has a PR, a review, and a commit history.
• Disaster recovery. If an entire region goes down, we can spin up in another region without manual steps.

The tool matters less than the principle. We use Terraform for infrastructure and GitHub Actions for CI/CD, but the same principles apply with any tool.

6. Observability from Day One

Monitoring isn’t something you add after you have problems — it’s something you build alongside features. Every deploy should be observable:

• Structured logging with request IDs that trace through the entire stack
• Metrics dashboards that show error rates, latency percentiles (p50, p95, p99), and throughput
• Alerts that page on-call for genuine incidents, not noise
• Deploy markers on dashboards so you can correlate metrics changes with deploys

The deploy markers are especially valuable. When you can see exactly when a deploy happened on every dashboard, debugging production issues becomes dramatically faster.

---

The common thread through all of these practices is speed with safety. Fast builds give developers confidence. Automated deploys reduce human error. Canary releases catch issues before they affect all users. Observability closes the feedback loop.

A great CI/CD pipeline doesn’t just ship code faster — it gives your entire team the confidence to move quickly without fear of breaking things. That confidence compounds into higher velocity, better morale, and ultimately, better software.