ssmd-deploy

Deploy ssmd components from code change through to running in GKE prod.

Overview

ssmd images are built by GitHub Actions on tag push. Deployments are GitOps via Flux: update image tags in clusters/gke-prod/ manifests, push, and Flux reconciles.

There are two repos involved:

• ssmd (899bushwick/ssmd/) — source code, where tags trigger image builds
• 899bushwick — deployment manifests and DQ/data scripts

Component Registry

ssmd repo (Rust + Deno components)

899bushwick repo (Python/DQ components)

Deployment Procedure

Step 1: Determine what to deploy

Identify which components changed. Common groupings:

• Connector schema change (e.g. new parquet column): connector + archiver (v*), parquet-gen (parquet-gen-v*)
• Connector-only change: connector + archiver (v*)
• DQ logic change: dq (dq-v*)
• parquet-gen only: parquet-gen (parquet-gen-v*)
• Operator CRD change: operator (operator-v*), also copy CRDs to clusters/gke-prod/apps/ssmd/operator/crds.yaml

Step 2: Determine next version numbers

Check the latest tags:

# For ssmd repo components
git -C <ssmd-path> tag --sort=-creatordate | grep "^v[0-9]" | head -3      # connector/archiver
git -C <ssmd-path> tag --sort=-creatordate | grep "^parquet-gen-v" | head -3
git -C <ssmd-path> tag --sort=-creatordate | grep "^operator-v" | head -3
git -C <ssmd-path> tag --sort=-creatordate | grep "^data-ts-v" | head -3

# For 899bushwick components
git -C <899bushwick-path> tag --sort=-creatordate | grep "^dq-v" | head -3

Increment the patch version (e.g. v0.9.10 -> v0.9.11).

Step 3: Commit and push source changes

# ssmd repo
git -C <ssmd-path> add <changed-files>
git -C <ssmd-path> commit -m "<type>: <description>"
git -C <ssmd-path> push origin main

# 899bushwick repo (for DQ/data changes)
git -C <899bushwick-path> add <changed-files>
git -C <899bushwick-path> commit -m "<type>: <description>"
git -C <899bushwick-path> push origin main

Step 4: Tag and push to trigger builds

# ssmd repo tags
git -C <ssmd-path> tag <tag>
git -C <ssmd-path> push origin <tag>

# 899bushwick tags
git -C <899bushwick-path> tag <tag>
git -C <899bushwick-path> push origin <tag>

Multiple tags can be pushed at once: git push origin v0.9.11 parquet-gen-v0.7.0

Step 5: Wait for GitHub Actions builds

gh run list --repo aaronwald/ssmd --limit 5 --json name,status,conclusion,headBranch
gh run list --repo aaronwald/899bushwick --limit 3 --json name,status,conclusion,headBranch

Poll until all relevant builds show "conclusion":"success". Rust builds take 3-5 minutes; Python/Deno builds take 1-2 minutes.

Step 6: Update deployment manifests

Update image tags in the manifest files listed in the Component Registry table above.

The image: field format is: ghcr.io/aaronwald/<image-name>:<version-without-prefix>

• Tag v0.9.11 -> image tag 0.9.11
• Tag parquet-gen-v0.7.0 -> image tag 0.7.0
• Tag dq-v0.8.0 -> image tag 0.8.0

Important: Connector and archiver share the same version number (same v* tag triggers both builds).

Step 7: Commit and push manifests

git -C <899bushwick-path> add <manifest-files> ssmd  # include ssmd submodule if updated
git -C <899bushwick-path> commit -m "deploy: <component> <version>[, <component> <version>...]"
git -C <899bushwick-path> push origin main

Step 8: Verify Flux reconciliation

Flux auto-reconciles on git push. To force or check:

flux reconcile kustomization ssmd --with-source
kubectl get pods -n ssmd -w  # watch for rolling restarts

Notes

• Connector and archiver always deploy together (same v* tag). Even if only one changed, both get rebuilt.
• The ssmd submodule pointer in 899bushwick should be updated when ssmd code changes are deployed.
• CronJobs (parquet-gen, DQ, secmaster, etc.) pick up new images on next scheduled run, not immediately.
• For Deployments (connector, archiver, operator, data-ts, worker), Flux triggers a rolling restart on image change.
• Never kubectl apply directly — all changes go through git + Flux.

Examples

Deploy a connector + parquet-gen change

Components: connector v0.9.11, parquet-gen v0.7.0
Tags: v0.9.11, parquet-gen-v0.7.0 (in ssmd repo)
Manifests: 3 connector yamls, 3 archiver yamls, 1 parquet-gen cronjob

Deploy a DQ-only change

Components: dq v0.8.0
Tags: dq-v0.8.0 (in 899bushwick repo)
Manifests: 1 dq-daily cronjob

Deploy operator CRD update

Components: operator v0.5.6
Tags: operator-v0.5.6 (in ssmd repo)
Extra step: copy CRDs from ssmd-operators to clusters/gke-prod/apps/ssmd/operator/crds.yaml
Manifests: 1 operator deployment + 1 crds.yaml