Observing What You've Built: Secrets, Alerting, and Blast Radius

Apologies for the radio silence since the last post. I started a new job recently, which has eaten into the time I'd normally spend on this kind of writing, and the rest of my spare hours have gone into Cycle Kirklees, the cycling campaign group I volunteer for. More on that platform soon, for now back to the self-hosted platform.

When you've built infrastructure on your own hardware, in your own home office, the stakes feel different. There's no "we'll fix it Monday": the pipes are yours to maintain. And there's no SRE team to page when something breaks at 2am.

This is part 4 in a platform engineering series. The previous posts covered CI/CD infrastructure, declarative VM provisioning, and Kubernetes cluster automation. But once you have a working cluster and pipelines that deploy code, you hit two problems that feel like afterthoughts but absolutely aren't:

1. How do you keep secrets secret? (Even from yourself, when you're tired and just want to ship something.)
2. How do you know what's happening when your infrastructure is running 26 services across multiple VMs?

And underneath both: how do you expose capabilities to other people without exposing the entire system?

The Secret You Meant to Revoke

Here's the thing about hardcoding a secret in your pipeline, with a mental note to "revoke it later": you won't remember. You'll intend to. You'll think "I'll just test this locally first, then rotate the credential before I commit." Then you push at 11pm because the thing works and you're tired. Six months later, that secret is still live.

This only matters if anyone ever sees your repository. My GitLab instance isn't on the public internet, so the blast radius is limited to: anyone who compromises my self-hosted platform, anyone on my LAN with physical access, anyone who gains shell access to a Proxmox VM. All real threats, none of them zero-probability.

But the lesson isn't "you're safe, so never mind." The lesson is: don't rely on future-you to remember operational discipline. Structure the system so that the easy path and the safe path are the same path.

GitLab CI/CD Variables: The Parallel to Azure Key Vault

In my day job, we use Azure Key Vault to manage secrets across infrastructure and applications. The pattern is straightforward:

• Secrets live in a centralised, encrypted store.
• Access is gated by identity and RBAC.
• Audit trails log every read and write.
• Applications request secrets at runtime; they're never committed to code.
• Secret rotation happens in one place; all consumers pick up the new value immediately.

GitLab's CI/CD variable system is the self-hosted equivalent. Not as featureful as Key Vault (no audit trail, no separate identity layer), but it solves the core problem: keep credentials out of your repository.

Here's how it works in practice.

Variable Scoping: Environment, Project, and Group

GitLab variables can be scoped at three levels:

Project-level variables apply to all CI/CD jobs in a project. Useful for values that don't change by environment.

# Available in all jobs in this project
REGISTRY_USERNAME: my_docker_registry_user

Environment-level variables apply only to jobs that explicitly deploy to that environment. This is where the separation of concerns lives.

# Only available when deploying to staging
STAGING_DATABASE_URL: postgres://staging-db:5432/app

# Only available when deploying to production
PRODUCTION_DATABASE_URL: postgres://prod-db:5432/app

Group-level variables (if you have multiple projects) are inherited by all projects in the group. Useful for shared credentials like container registry tokens.

In my Cycle Kirklees setup:

• The project has a shared CI/CD token (read-only) for pulling from my internal registry.
• The staging environment has credentials for pushing to the staging Docker registry and deploying to the k3s staging namespace.
• The production environment has separate credentials for the production registry and k3s namespace, and it's locked behind a manual gate: no auto-deploy to prod.

stages:
  - build
  - test
  - deploy_staging
  - deploy_production

build:
  stage: build
  script:
    - docker build -t app:$CI_COMMIT_SHA .
    - docker push $REGISTRY/$DOCKER_NAMESPACE/app:$CI_COMMIT_SHA

deploy_to_staging:
  stage: deploy_staging
  environment:
    name: staging
    kubernetes:
      namespace: cyclekirklees-staging
  script:
    - helm upgrade --install cyclekirklees ./helm
      --namespace cyclekirklees-staging
      --set image.tag=$CI_COMMIT_SHA
  only:
    - main

deploy_to_production:
  stage: deploy_production
  environment:
    name: production
    kubernetes:
      namespace: cyclekirklees-prod
  script:
    - helm upgrade --install cyclekirklees ./helm
      --namespace cyclekirklees-prod
      --set image.tag=$CI_COMMIT_SHA
  when: manual  # Explicit human gate
  only:
    - main

When the deploy_to_staging job runs, GitLab injects the staging environment variables. When someone manually triggers deploy_to_production, the production variables are injected instead. The same pipeline, different secrets.

Why This Matters: Blast Radius

The philosophy here is simple: if something goes wrong, limit the damage.

If a staging credential leaks (or is accidentally logged), only staging is affected. The attacker, or the tired sysadmin, can't accidentally deploy to production. That requires a separate credential, manually gated.

In my setup:

• Staging auto-deploys from main. If I push a commit, the staging environment is updated automatically. This is low-risk; staging can be broken.
• Production requires manual approval. Even if CI passes, even if staging works, production won't update until I explicitly click "Deploy" in the GitLab UI.
• Production credentials are environment-scoped. They're never used in the staging pipeline.

This is "least privilege" in practice: each role gets the minimum credentials it needs, and no more.

Protecting the Protected Branch

GitLab also lets you lock down which branches people can merge into, and require sign-off before changes go live. Here's how that works for Cycle Kirklees.

The main branch is what actually gets deployed to production, so it's protected: nobody can merge into it without approval from a Maintainer. Only committee members (and me) have that level of access.

Committee members add and edit website content through Sveltia, a CMS that isn't exposed to the open internet. Sveltia needs to push changes directly to a branch to save content, so I gave it its own branch, called content, that isn't locked down the same way main is.

That sounds like a hole: why leave a branch unprotected? Because changes on the content branch still have to go through an approval before they reach main. So a committee member can publish content updates any time, without waiting on me, but they can't merge straight to production. If something goes wrong, the damage is limited to draft content sitting on a branch, not a live deploy.

Everything is a trade-off. The point is: the trade-off is visible and justified.

Observability: Seeing What You Can't See

You have a cluster. Workloads are running. But:

• Is memory pressure building on a worker node?
• Which service is causing the most load?
• When did that pod start crashing, and why?
• How much network bandwidth is the database pod consuming?

Without observability, these questions have one answer: SSH into a node, run commands, hope you ask the right questions. With observability, you have a single pane of glass.

I use Prometheus and Grafana. Prometheus scrapes metrics from k3s nodes and workloads. Grafana visualises them and provides alerting.

Prometheus: Scraping the Cluster

Kubernetes exposes metrics on every node and pod via the /metrics endpoint. Prometheus polls those endpoints, stores time-series data, and makes it queryable.

The k3s setup includes the Prometheus Operator, which watches for ServiceMonitor and PrometheusRule resources and automatically wires them into the Prometheus configuration. No static config files; infrastructure as code.

A simple ServiceMonitor to scrape the Cycle Kirklees application:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cyclekirklees-app
  namespace: cyclekirklees-prod
spec:
  selector:
    matchLabels:
      app: cyclekirklees
  endpoints:
    - port: metrics
      interval: 30s

This tells Prometheus: "Every 30 seconds, scrape the metrics port on any service labelled app: cyclekirklees."

The Cycle Kirklees application (built with Node.js and Express) exports metrics via prom-client:

const prometheus = require('prom-client');

const httpRequestDuration = new prometheus.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.route?.path || 'unknown', res.statusCode)
      .observe(duration);
  });
  next();
});

app.get('/metrics', (req, res) => {
  res.set('Content-Type', prometheus.register.contentType);
  res.end(prometheus.register.metrics());
});

Now Prometheus knows: request latency, error rates, how many requests per route. That's data.

Grafana: Seeing the Data

Grafana queries Prometheus and renders dashboards. A simple dashboard for Cycle Kirklees might show:

• Request latency (p50, p95, p99)
• Error rate (5xx responses per minute)
• Pod restart count
• Memory and CPU usage
• Network I/O

When you look at a latency spike, you can cross-reference it with an error rate spike. If they align, you know the service is struggling under load. If error rate spikes without latency, you know the issue is application logic, not resources.

Example Grafana dashboard JSON (snippet):

{
  "panels": [
    {
      "title": "Request Latency (p95)",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))"
        }
      ]
    },
    {
      "title": "Error Rate",
      "targets": [
        {
          "expr": "rate(http_requests_total{status_code=~\"5..\"}[1m])"
        }
      ]
    },
    {
      "title": "Pod Restarts (24h)",
      "targets": [
        {
          "expr": "increase(kube_pod_container_status_restarts_total[24h])"
        }
      ]
    }
  ]
}

This is the "single pane of glass." When something feels off, you start here.

Alerting: The Right Message to the Right Person at the Right Time

Observability without alerting is a dashboard you check after something breaks. Alerting means you know before it affects users.

But alerting is easy to get wrong. Fire too many alerts, and you'll ignore them. Too few, and you'll miss the real problems. The goal is: signal, not noise.

In my setup, I use four alert rules, each routed to a different channel:

Alert 1: Pod Restart Spike → Telegram (Cycle Kirklees Team)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pod-restarts
spec:
  groups:
    - name: pod-restarts
      rules:
        - alert: PodRestartingTooOften
          expr: |
            increase(kube_pod_container_status_restarts_total{namespace="cyclekirklees-prod"}[15m]) > 3
          for: 5m
          annotations:
            summary: "Pod {{ $labels.pod }} restarted {{ $value }} times in 15 minutes"
            message: "High restart rate detected. Check pod logs for crashes."

When this fires, a Webhook sends a Telegram message to the Cycle Kirklees committee channel. The committee doesn't need to know why it restarted, just that it did, so they can tell me to fix it.

Why Telegram? It's a group channel. Committee members see the alert, they know something's up, they can ask questions before escalating to me.

Alert 2: High Error Rate → Email (Me)

- alert: HighErrorRate
  expr: |
    rate(http_requests_total{status_code=~"5.."}[5m]) > 0.05
  for: 2m
  annotations:
    summary: "{{ $labels.service }} error rate {{ $value | humanizePercentage }}"

Email goes to my personal account. It's slower than Telegram, but it's deliberate. A 5% error rate is bad, but it's not "wake me at 2am" bad. I'll see it in the morning and investigate.

Alert 3: Node Disk Space → Personal Email + AI Agent Alert

- alert: NodeDiskPressure
  expr: |
    (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) > 0.85
  for: 10m
  annotations:
    summary: "Node {{ $labels.node }} is {{ $value | humanizePercentage }} full"

When disk usage hits 85%, two things happen:

1. Email alert goes to me.
2. A webhook triggers an n8n flow (my local agentic automation platform) that:
   • Checks which pods are consuming the most space
   • Identifies old logs that could be safely rotated
   • Sends me an AI-generated summary: "Disk pressure on node-2. Postgres logs are 40GB. Consider rotating logs or adding storage."

The AI agent doesn't make decisions. It gathers context and presents options. But it's better than just "disk full," because it's already done half the troubleshooting.

Alert 4: Cluster API Unavailable → Telegram + Email + Heartbeat Monitor

- alert: KubernetesAPIDown
  expr: up{job="kubernetes-apiservers"} == 0
  for: 1m
  annotations:
    summary: "Kubernetes API is unreachable"

This is a "hard stop" alert. If the API is down, the cluster is down. It goes everywhere:

• Telegram (committee, so they know the platform is offline)
• Email (me, so I know to check the lab)
• A heartbeat monitor (externally hosted) pings my self-hosted platform every 5 minutes. If it doesn't respond, that sends a backup alert to my phone.

The heartbeat monitor is paranoia, a belt-and-suspenders approach. But the idea is sound: if my platform is down, the alerts on my platform won't fire. So I need an external check that says "hey, your platform just went dark."

Why This Structure Works

Look at the four alerts. They all solve a real problem:

• Pod restarts: The committee needs to know the service is flaky.
• Error rate: I need to know to check logs.
• Disk pressure: I need context, not just a binary "full/not full" signal.
• API down: Everyone needs to know immediately.

Each alert goes to a different channel because each has a different audience and urgency. Telegram for "the platform is acting weird"; email for "investigate when you get a chance"; AI context for "here's what's probably wrong."

No alert goes to a channel it doesn't belong in. No Telegram spam about disk space. No "alert about alerts." Just: the right information, to the right person, at the right time.

Bringing It Together: Platform Principles

This is what a self-hosted platform looks like:

1. Secrets are scoped and gated. Different environments get different credentials. Production requires explicit approval.
2. Visibility is built in. Every service exports metrics. Grafana shows the picture. Alerting catches problems early.
3. Blast radius is limited. A mistake in staging doesn't touch production. A misconfigured alert doesn't spam everyone.
4. Operational discipline is structural, not aspirational. You don't promise to remember to rotate credentials. The system makes it easy to avoid hardcoding them in the first place. You don't hope you notice a problem. Alerts tell you.

For Cycle Kirklees, this means:

• Committee members can edit content in Sveltia CMS without being able to deploy to production.
• Staging automatically reflects the latest content changes, so they can preview before going live.
• Production deployments require my explicit approval, so I'm in control of when changes ship.
• If something goes wrong, I see it in Grafana or via Telegram before users complain.
• If a node fills up, I get context-aware suggestions, not just a "full" alert.

This is how you keep hosting costs down (self-hosted hardware) whilst keeping the blast radius down (staged rollout, scoped secrets, alerting). And it's how you expose capabilities to other people, in this case a volunteer-run cycling advocacy organisation, without exposing the entire system to risk.

Closing the Loop

This started as a roadmap: declarative provisioning, configuration management, a cluster, and finally the two things that make a platform trustworthy rather than just functional. Secrets that don't leak. Problems that surface before someone complains about them.

None of this is finished in the sense that software is ever finished. Credentials still need rotating. Alert thresholds still need tuning as the platform grows. But the structure is in place, and that's the point of a platform: not that nothing goes wrong, but that when something does, the damage is contained and you find out about it on your terms.

That's where this series ends. The lab runs Cycle Kirklees, this portfolio, and a handful of internal tools on infrastructure I provisioned, configured, and now watch over, end to end, with no cloud safety net and no one else to call.

Resources

• GitLab CI/CD Variables Documentation
• Prometheus Operator: ServiceMonitor CRD
• Grafana Alerting
• prom-client (Node.js Prometheus client)
• Kubernetes Metrics Server

This post is part of my platform engineering portfolio. See more at spinstate.dev.

How do you scope secrets and alerts on a platform with no dedicated SRE team? I'd like to hear your approach in the comments.