Operational Telemetry with OpenTelemetry
Updated
IntermediateWhat you'll learn
~25 min- Initialize OpenTelemetry (a vendor-neutral way to emit traces, metrics and logs) in a Next.js server so the app speaks OTLP (the OpenTelemetry wire protocol) to a collector and nothing else
- Run a local collector (a small relay process that receives telemetry and forwards it to a backend) and prove that a span from the app reaches it
- Read secrets through the platform seam with platform.secrets.get(name) instead of environment variables baked into source
- Write down what never gets logged, and prove it with a test that plants a secret and checks it is gone
What telemetry is for
Operational telemetry answers one question: is the platform healthy right now, and if not, which request and which dependency is the cause. It is not analytics and it is not user tracking. You want to learn that a route handler is returning 403 to people who should pass, or that a database call has doubled in duration, before a help desk ticket tells you.
You will wire three things: spans flowing over OTLP to a collector, secrets read through one call, and a short list of fields that are never written to any log. The app has one telemetry path on every platform. It speaks OTLP to whatever listens at OTEL_EXPORTER_OTLP_ENDPOINT. Where the spans go next is the collector’s configuration, and that configuration belongs to your platform lesson.
Before the detail — the artifact is startTelemetry(serviceName) plus an allowlist, safeFields. The acceptance test is a span that appears in the collector with the right service.name, and a test that plants a secret and finds it gone. The OTLP path is a common baseline; the allowlist is reusable guidance.
| Problem | The pattern | Where your platform lesson lands it | You decide |
|---|---|---|---|
| How the app emits telemetry | OTLP to a collector; the app knows one endpoint variable | 34: in-process Microsoft OpenTelemetry Distro to Azure Monitor · 35: collector to X-Ray · 36: collector to Cloud Trace, if your boundary lists it | Which backend your boundary allows, and who edits the collector configuration |
| Where the backend address and credentials come from | The collector’s configuration, never the app’s source | 34: the portal’s connection string, never a typed host · 35: IAM role on the collector · 36: service account on the collector | Who is allowed to change that configuration |
| Where secrets live | platform.secrets.get(name) | 34: Key Vault · 35: Secrets Manager · 36: Secret Manager | Rotation owner and access reviewers |
| What reaches telemetry | An allowlist of field names | Same on every platform | Which names are on the list, and who reviews a change to it |
Plan the change with your agent
Open your workspace agent at the project root with the project-memory file from Lesson 2 in place, switch to Plan mode (Shift+Tab, /plan, or the desktop app’s mode picker), and ask:
Read src/platform/types.ts and our project-memory file, and plan server-side OpenTelemetry tracing over OTLP plus a field allowlist that strips secrets, headers, and query strings from logs. Ask me anything unclear.What to check in the plan before you approve
src/instrumentation.tsexportsregister()and, whenprocess.env.NEXT_RUNTIME === "nodejs", dynamically imports./lib/telemetry/registerand awaitsregisterTelemetry("intake-app"), which delegates toplatform.telemetry.start.src/lib/telemetry/start.tsexportsstartTelemetry(serviceName)usingNodeSDKandOTLPTraceExporterreadingOTEL_EXPORTER_OTLP_ENDPOINT(with no cloud-specific telemetry packages or hardcoded backend hosts), logging one warning and returning when the variable is unset.src/lib/telemetry/safe-fields.tsexportssafeFields(input), keeping onlymethod,path(with any query string stripped),status,durationMs,correlationId,route, anderrorName.- A unit test plants an
Authorizationheader, a session cookie, an email address, and a?token=query string and asserts none survivesafeFields().
If the first plan installs a vendor-specific telemetry SDK in core code or logs raw URLs with query parameters, nudge it before approving:
Do not hardcode a backend telemetry host; emit OTLP to OTEL_EXPORTER_OTLP_ENDPOINT and strip query strings in safeFields.What to expect
The app side is one function. It is real code with a passing test in the module’s runnable reference.
REFERENCE CODE src/lib/telemetry/start.ts: NodeSDK OTLP trace exporter initialization open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
// One telemetry path on every platform (lesson 16): the app speaks OTLP to a collector// at OTEL_EXPORTER_OTLP_ENDPOINT and knows nothing else. Where spans go from there is// the collector's configuration, which each platform lesson supplies (34 Azure Monitor,// 35 X-Ray, 36 Cloud Trace). Locally the collector just prints them.import { NodeSDK } from '@opentelemetry/sdk-node';import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';import { resourceFromAttributes } from '@opentelemetry/resources';
let sdk: NodeSDK | undefined;
export function startTelemetry(serviceName: string): NodeSDK | undefined { if (sdk) return sdk; if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT) { console.warn('telemetry disabled: OTEL_EXPORTER_OTLP_ENDPOINT is not set'); return undefined; } sdk = new NodeSDK({ resource: resourceFromAttributes({ 'service.name': serviceName }), traceExporter: new OTLPTraceExporter(), // reads OTEL_EXPORTER_OTLP_ENDPOINT itself }); sdk.start(); return sdk;}The startup hook does not call startTelemetry itself. It asks the platform through this reference helper in gov-dev-reference-app, so a platform whose backend wants something other than a collector (Azure, in lesson 34) can start that instead:
// What the app's startup hook calls (lesson 16: Next.js src/instrumentation.ts register()).// It goes through the seam, so each platform starts its own telemetry: local and the// collector-based platforms call startTelemetry() (OTLP); Azure starts its in-process distro.import { getPlatform } from '../../platform/index.ts';
export async function registerTelemetry(serviceName: string): Promise<void> { const platform = await getPlatform(); await platform.telemetry.start(serviceName);}And here is the reference startup hook your agent writes in src/instrumentation.ts:
export async function register() { if (process.env.NEXT_RUNTIME === "nodejs") { const { registerTelemetry } = await import("./lib/telemetry/register"); await registerTelemetry("intake-app"); }}The instrumentation.ts hook is Next.js’s own mechanism for running code at server start; that the OpenTelemetry Node SDK works from inside it was not tested for this lesson, so confirm that you see traces before you trust it.
The local collector
On a laptop the collector just prints what it receives. The platform lessons replace its exporters block; the app does not change.
REFERENCE CONFIG Local OpenTelemetry collector configuration (OTLP HTTP receiver to debug exporter) open reference hide reference Reference pattern for your workspace agent; open to inspect or compare this file. You do not type or paste this by hand.
# Local collector for lesson 16: receives OTLP from the app and prints it.# Platform lessons replace the exporters block; the app does not change.receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318exporters: debug: verbosity: detailedservice: pipelines: traces: receivers: [otlp] exporters: [debug]Start it with npm run local:up from the reference folder (it also starts the database and the sign-in provider from earlier lessons). Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 for the app.
The destination is not in this lesson’s code. Lesson 34 uses the in-process Microsoft OpenTelemetry Distro, which sends to Azure Monitor with Microsoft Entra authentication and takes its ingestion address from the portal’s connection string. Lesson 35 sends traces through a collector to X-Ray; the CloudWatch OTLP traces path needs a feature called Transaction Search that AWS GovCloud lists as unavailable. Lesson 36 sends through a collector to Cloud Trace, which the Assured Workloads page lists for FedRAMP High but not in its IL4 product table, so verify the product list for your control package first.
Secrets through the seam
A connection to a database, a client secret, a third-party key: none belongs in source, in a committed .env file, or in a plain pipeline variable. The platform seam gives the app one door: platform.secrets.get(name). The sign-in client secret, for example, is read with platform.secrets.get(platform.identity.clientSecretName), as in Lesson 3. Locally the secret is a file in ~/.config/m14-reference; on a cloud it is that cloud’s secret store.
Lesson 34 shows the Azure Government implementation: Key Vault, the vault suffix, and how a rotated value reaches the app. The pattern here is the call site, which does not change across platforms.
What never gets logged
Telemetry is a copy of your system’s behavior, stored somewhere with a different audience and a different retention period than your database. The rule is an allowlist: a log or span attribute carries only fields from a short list, and anything else is dropped by default.
| Never log | Why it is easy to leak |
|---|---|
| Authorization headers, session cookies, tokens, client secrets | Request-logging helpers dump all headers by default |
| Request and response bodies | They carry whatever users type, including identifiers and case details |
| Names, email addresses, government identifiers | Added “for debugging” and kept for the retention period |
| Full URLs with query strings | ?token= and ?email= style parameters ride along |
| SQL text with bound values, stack traces returned to a client | They reveal schema and data; Lesson 7 covers errors as a contract |
| Anything marked CUI (controlled unclassified information, the federal category for sensitive-but-unclassified data) | Handling rules follow the data into every system that stores it; take the determination from your CUI program office |
What you do log: method, route pattern, status, duration, a correlation id, and the error’s name. That is enough to find a failing request without storing who made it or what it said. Here is the reference allowlist function your agent generates in src/lib/telemetry/safe-fields.ts:
const ALLOWED = ["method", "route", "status", "durationMs", "correlationId", "errorName"] as const;
export function safeFields(input: Record<string, unknown>): Record<string, unknown> { const out: Record<string, unknown> = {}; for (const key of ALLOWED) if (key in input) out[key] = input[key]; if (typeof input.path === "string") out.path = input.path.split("?")[0]; return out;}See it in action
- Typed in the terminal:
claude - Mode:Plan mode
- You type in the session: Add telemetry to the server. The app should work on any platform we land it on.
- The tool replies: Before I write anything, two questions so I do not guess: 1. Backend: I will not import a vendor package. The app will send OTLP to whatever is at OTEL_EXPORTER_OTLP_ENDPOINT, and the collector decides where spans go. 2. Name: I will pass the service name into startTelemetry so every span carries service.name. I will create src/instrumentation.ts, src/lib/telemetry/start.ts and src/lib/telemetry/safe-fields.ts, plus a test. Proceed?
- You type in the session: Proceed. Also send them straight to the Government endpoint so we skip the collector.
- The tool replies: I will not type a backend address. Which host is right depends on the platform, and a wrong one fails silently. Your platform lesson supplies it, in the collector's configuration or, on Azure, in the connection string from the portal. Created: src/instrumentation.ts (register(), nodejs runtime only) src/lib/telemetry/start.ts (skips with one warning if the variable is unset) src/lib/telemetry/safe-fields.ts (allowlist) src/lib/telemetry/safe-fields.test.ts Note: I have not verified that the SDK starts from inside Next's register() hook. Run the app and confirm a span arrives before relying on it.
- You type in the session: Run it against the local collector and show me a span arrived.
- The tool replies: $ npm run local:up $ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run test:env 6 tests passed The telemetry test creates a span named intake.submit-<timestamp>, shuts the SDK down to flush it, then reads the collector's log. It found the span name and the resource attribute service.name: Str(m14-reference).
- You type in the session: Log the full request headers on 500s so we can debug faster.
- The tool replies: I would rather not. Request headers include the Authorization header and the session cookie, and they would be stored for the retention period of the telemetry backend. What I can log on a 500 is method, route, status, duration, the correlation id, and the error name through safeFields(). If you need more for one incident, I can add a temporary, named field to the allowlist behind a code review, and remove it afterward. Want that instead?
Illustrative session; exact output varies by tool and agency configuration.
Prove it
Two tests, one for each promise. From gov-dev-reference-app, with the local stack up:
npm run test:envPass: 6 tests pass. One of them creates a span, flushes it, and reads the collector’s output: the span’s name appears, and so does service.name: Str(m14-reference). That is the proof the app speaks OTLP and the collector hears it. Fail, which you should see once on purpose: stop the collector (docker stop m14-otel) and the telemetry test goes red.
Then the allowlist, in your own project:
npx vitest run src/lib/telemetry/safe-fields.test.tsPass: the test plants Authorization, cookie, an email address, and ?token=abc123 in a request-shaped object and asserts none appear in the result. Fail, which you should see once on purpose: add "cookie" to the allowlist and the test goes red.
Finally check the backend your platform lesson configured: send one request and look for it in that backend’s trace view. If nothing appears, check the collector’s log first, then its credentials, then the endpoint variable.
An AI tool offers to send spans from the app straight to a cloud backend's host so you can skip the collector. What is the right response?
Which log record fits the allowlist rule for a failed request?
Practice status
| Practice | Status | Also called |
|---|---|---|
| App speaks OTLP to a collector | common baseline | collector-fronted telemetry |
| Field allowlist for telemetry | strong optional | log scrubbing by default-deny |
| Secrets through one platform call | common baseline | secret-store abstraction |
Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging
Key takeaway
Telemetry is a second copy of your system’s behavior, so give it the same care as the first. The app emits OTLP to one endpoint, the collector decides where spans go, secrets come through platform.secrets.get, and every log field comes from an allowlist. Your platform lesson (34, 35 or 36) holds the backend, its authentication and its secret store. Next, Lesson 17 covers how the code that carries all of this reaches a repository without a long-lived token.
- LEADERSHIP DECISION
- Which telemetry backend your authorization boundary allows, and who may change the collector configuration.
- PRACTITIONER ACTION
- Run the env test until a span reaches the collector, then add the allowlist test to CI.
- SUCCESS MEASURE
- Contract or revenue risk: no secret or personal field reaches a store with a longer retention than the database.
Legend: LEADERSHIP DECISION = what a leader decides · PRACTITIONER ACTION = what the engineer does next · SUCCESS MEASURE = how you know, in calendar time, assessor-hours or cost, contract or revenue risk, or audit finding avoided