Skip to main content

Logs

Stage 6 · Mission Operations

While metrics indicate that an incident is occurring and traces pinpoint where latency is accumulating, logs surface the explicit application error messages, database exceptions, and contextual event details needed for root-cause diagnosis.

What distinguishes structured logs from unstructured strings​

  • Unstructured text (Hard to parse & query): 
    2024-09-18 14:32:01 ERROR booking failed for user 1234: duplicate key
  • Structured JSON (Queryable, indexed, and correlated): 
    {
      "timestamp": "2024-09-18T14:32:01.423Z",
      "level": "error",
      "service": "booking",
      "trace_id": "abc123traceidentifier",
      "operation": "CreateBooking",
      "booking_reference": "AA-2024-001234",
      "error": "pq: duplicate key value violates unique constraint \"bookings_booking_reference_key\"",
      "duration_ms": 45
    }

Key advantages of structured fields:​

  • Trace correlation: Embedding trace_id enables jumping directly from a Tempo trace span into the exact matching log lines in Grafana.
  • LogQL filtering: Allows querying by specific operational codes without fragile regex matching.

Apollo's log ingestion pipeline​

Diagram OB-04 — containers print to stdout; containerd captures logs to disk; Grafana Alloy discovers, annotates with Pod labels, and ships to Loki.

  • Stdout standard: Applications print directly to standard output.
  • Node agent: Grafana Alloy runs as a DaemonSet, scraping container log paths on the host node.
  • Metadata enrichment: Alloy attaches Kubernetes metadata (namespace, app, pod) as stream labels before shipping to Loki.

Log retention rules: what to log vs. what to redact​

  • Always include:
    • trace_id and span_id.
    • Functional error reasons and exception types.
    • Safe business identifiers (order numbers, flight codes).
  • Never include (Security violations):
    • Raw JWT tokens or passwords.
    • Credit card numbers, CVVs, or passenger PII.

Evidence and limits​

  • 1. Direct container output: Check unbuffered application stdout: 
    kubectl logs -n apollo-airlines-apps deploy/booking --tail=50
  • 2. Query Loki via LogQL: Filter errors in Grafana Explore: 
    {app="booking", namespace="apollo-airlines-apps"} |= "error" | json | level="error"
  • 3. Search by booking reference: 
    {app="booking"} |= "AA-2024-001234"