Monitoring & Logging

Observability in Kubernetes

When a request slows down at 3 AM, kubectl get pods will not tell you why. Pods are ephemeral, replicas come and go, and a single user request may cross a dozen services. Observability is the practice of instrumenting a cluster so you can ask arbitrary questions about its behavior after the fact — without shipping new code. In Kubernetes this is built on three pillars: metrics (numeric time series like request rate and memory usage), logs (discrete events emitted by your apps), and traces (the path of a single request across services). This lesson focuses on the two you will stand up first: metrics with Prometheus and Grafana, and logs with Loki.

The Four Golden Signals

Before wiring up tools, decide what to measure. Google's SRE book recommends four signals that cover most user-facing failures:

• Latency — how long requests take (track successful and failed latency separately).
• Traffic — demand on the system, e.g. requests per second.
• Errors — the rate of failed requests (5xx responses, dropped connections).
• Saturation — how full your resources are (CPU, memory, queue depth).

Deploying the Prometheus Stack

The standard way to install Prometheus, Grafana, Alertmanager, and the node/kube-state exporters together is the community kube-prometheus-stack Helm chart. It also installs the Custom Resource Definitions (ServiceMonitor, PodMonitor, PrometheusRule) that let you configure scraping declaratively instead of editing a giant prometheus.yml.

bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword='change-me'

Once the pods are running, port-forward Grafana to log in locally:

bash
kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
# open http://localhost:3000  (user: admin)

Scraping Metrics with ServiceMonitor

The Prometheus Operator watches for ServiceMonitor objects and reconfigures Prometheus automatically. A ServiceMonitor selects Kubernetes Service objects by label, then tells Prometheus which port and path to scrape. Your Service must expose a named port that matches the port field below.

yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-metrics
  namespace: monitoring
  labels:
    release: monitoring   # must match the Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: api            # matches the Service's labels
  namespaceSelector:
    matchNames:
      - default
  endpoints:
    - port: metrics       # the Service's named port, not the number
      path: /metrics
      interval: 30s

Instrumenting Your Application

Prometheus is pull-based: your app exposes an HTTP /metrics endpoint in the text exposition format, and Prometheus scrapes it. Use an official client library rather than hand-writing the format. Here is a minimal Node.js example with prom-client:

ts
import express from "express"
import client from "prom-client"

const app = express()
const register = new client.Registry()
client.collectDefaultMetrics({ register }) // CPU, memory, event loop

const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "Request latency in seconds",
  labelNames: ["method", "route", "status"],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],
})
register.registerMetric(httpDuration)

app.use((req, res, next) => {
  const end = httpDuration.startTimer()
  res.on("finish", () =>
    end({ method: req.method, route: req.path, status: res.statusCode })
  )
  next()
})

app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType)
  res.end(await register.metrics())
})

app.listen(8080)

Querying with PromQL and Grafana

Prometheus stores everything as time series you query with PromQL. Because histograms are cumulative, you almost always wrap counters in rate() over a time window. These expressions map directly to the golden signals:

promql
# Traffic: requests per second per route
sum(rate(http_request_duration_seconds_count[5m])) by (route)

# Errors: fraction of 5xx responses
sum(rate(http_request_duration_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

# Latency: 95th percentile from histogram buckets
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
)

In Grafana, add Prometheus as a data source (the chart pre-configures it), then build panels from these queries or import a community dashboard by ID from grafana.com — for example 1860 (Node Exporter Full) or 315 (Kubernetes cluster monitoring). Store your own dashboards as JSON in Git so they are versioned and reproducible.

Centralized Logging with Loki

Metrics tell you that something is wrong; logs tell you what. Loki is a log aggregation system designed to pair with Grafana. Unlike Elasticsearch, it does not index the full text of every log line — it only indexes a small set of labels and stores the rest compressed, which makes it cheap to run. An agent such as Promtail (or Grafana Alloy) runs as a DaemonSet, tails every container's stdout/stderr, attaches Kubernetes labels, and pushes to Loki.

Because Loki and Prometheus share the same label vocabulary, you can jump straight from a spike on a Grafana metric panel to the exact logs for that pod and time range.

Alerting on SLOs

Dashboards only help when someone is looking. PrometheusRule objects define alerting rules that Prometheus evaluates continuously and forwards to Alertmanager for routing to Slack, PagerDuty, or email. Use for: to require a condition to persist before firing, which suppresses transient blips.

yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-slo-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: api.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_request_duration_seconds_count{status=~"5.."}[5m]))
              / sum(rate(http_request_duration_seconds_count[5m])) > 0.05
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "API 5xx error rate above 5%"
            description: "Error ratio is {{ $value | humanizePercentage }} for 10m."

Best Practices & Common Pitfalls

• Match your selectors. The number-one cause of "no data" is a ServiceMonitor whose release label does not match Prometheus's serviceMonitorSelector, or a Service port that has no name.
• Set resource requests and retention. Prometheus is memory-hungry. Give it a retention window (e.g. 15d) and back it with a PersistentVolume, or you will lose history on every restart.
• Avoid high-cardinality labels. They are the leading cause of Prometheus running out of memory. Audit new metrics before rolling them out.
• Alert on symptoms, page on the actionable. Route only severity: critical, user-impacting alerts to a pager; send warnings to a channel for review.
• Version your config. Keep dashboards, alerting rules, and Helm values.yaml in Git so your observability stack is as reproducible as your application.

Next Steps

You now have metrics, dashboards, logs, and alerts. Next you will use those metrics to scale the cluster automatically in response to real load.