CI/CD Pipeline Design: GitHub Actions + Cache + Matrix Build

CI/CD is the backbone of engineering throughput. Poor design forces 40+ min PR waits and random failures; well-designed pipelines deliver trusted feedback under 5 minutes, run matrix tests across versions/OSes, and cache hits make repeat builds instant.

Stage Split and Path Filtering

Classic 5 phases: (1) changes filter: skip entire untouched subtrees (docs-only PR bypasses backend tests); (2) lint stage fast-fails within 8 min max; (3) test multi-version matrix parallel; (4) build artifacts; (5) publish on git tags only. Concurrency auto-cancels prior runs on same ref, saves Action minutes.

.github/workflows/ci.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
    tags: ['v*.*.*']
  pull_request:
    branches: [main, develop]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      backend: ${{ steps.filter.outputs.backend }}
      frontend: ${{ steps.filter.outputs.frontend }}
      infra: ${{ steps.filter.outputs.infra }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            backend:
              - 'apps/api/**'
              - 'packages/shared/**'
              - 'pyproject.toml'
            frontend:
              - 'apps/web/**'
              - 'packages/ui/**'
              - 'pnpm-lock.yaml'
            infra:
              - 'infra/**'
              - 'Dockerfile*'

  lint:
    needs: changes
    if: needs.changes.outputs.backend == 'true' || needs.changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with: { version: 9 }
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: pnpm
          cache-dependency-path: pnpm-lock.yaml
      - name: Install Node deps
        run: pnpm install --frozen-lockfile --prefer-offline
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
          cache-dependency-path: pyproject.toml
      - name: Install Python deps
        run: pip install -e ".[dev]" --retries 3
      - name: Run ESLint + Ruff
        run: |
          pnpm lint
          ruff check apps/api packages
      - name: Cache turbo setup
        uses: actions/cache@v4
        with:
          path: .turbo
          key: turbo-${{ github.sha }}
          restore-keys: turbo-

  test:
    needs: lint
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        python-version: ['3.11', '3.12', '3.13']
        node-version: ['18', '20', '22']
        os: [ubuntu-latest]
        include:
          - os: macos-latest
            python-version: '3.12'
            node-version: '20'
    services:
      postgres:
        image: postgres:16-alpine
        env: { POSTGRES_USER: test, POSTGRES_PASSWORD: test, POSTGRES_DB: test }
        ports: ['5432:5432']
        options: >-
          --health-cmd pg_isready
          --health-interval 10s --health-timeout 5s --health-retries 5
      redis:
        image: redis:7-alpine
        ports: ['6379:6379']
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4; with: { version: 9 }
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: pnpm
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - run: pnpm install --frozen-lockfile
      - run: pip install -e ".[test]"
      - name: Run tests with coverage
        run: |
          pnpm test -- --coverage --reporter=github
          pytest apps/api --cov=apps/api --cov-report=xml
        env:
          DATABASE_URL: postgresql://test:test@localhost:5432/test
          REDIS_URL: redis://localhost:6379
      - uses: codecov/codecov-action@v4
        if: matrix.os == 'ubuntu-latest' && matrix.node-version == '20' && matrix.python-version == '3.12'
        with: { fail_ci_if_error: false, token: ${{ secrets.CODECOV_TOKEN }} }

  build-and-push-image:
    needs: test
    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha,prefix=
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          platforms: linux/amd64,linux/arm64
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Cache + Matrix Build: Three Musketeers of Speed

(1) Dep cache: setup-node pnpm cache + pip cache skip install on 90%+ builds; (2) Task artifact cache: turbo cache shares builds across workflow runs; (3) Docker layer cache: buildx gha cache reuses layers, push drops from 5 min to 40s. Matrix spans Python 3.11-3.13 × Node 18-22 across OSes; matrix include injects macOS single run to catch Unix-compat regressions.

Best Practices

Every job sets timeout-minutes hard cap against hangs. services use Postgres/Redis healthcheck before test start. fail-fast:false keeps all matrix combos running to surface every failure. Only semver tags trigger publish; main branch stops at build+test only.