Applied AI-Accelerated Government Development

Operational Telemetry with OpenTelemetry

Updated

Intermediate

What you'll learn

~25 min
  • Initialize OpenTelemetry (a vendor-neutral way to emit traces, metrics and logs) in a Next.js server so the app speaks OTLP (the OpenTelemetry wire protocol) to a collector and nothing else
  • Run a local collector (a small relay process that receives telemetry and forwards it to a backend) and prove that a span from the app reaches it
  • Read secrets through the platform seam with platform.secrets.get(name) instead of environment variables baked into source
  • Write down what never gets logged, and prove it with a test that plants a secret and checks it is gone

What telemetry is for

Operational telemetry answers one question: is the platform healthy right now, and if not, which request and which dependency is the cause. It is not analytics and it is not user tracking. You want to learn that a route handler is returning 403 to people who should pass, or that a database call has doubled in duration, before a help desk ticket tells you.

You will wire three things: spans flowing over OTLP to a collector, secrets read through one call, and a short list of fields that are never written to any log. The app has one telemetry path on every platform. It speaks OTLP to whatever listens at OTEL_EXPORTER_OTLP_ENDPOINT. Where the spans go next is the collector’s configuration, and that configuration belongs to your platform lesson.

Before the detail — the artifact is startTelemetry(serviceName) plus an allowlist, safeFields. The acceptance test is a span that appears in the collector with the right service.name, and a test that plants a secret and finds it gone. The OTLP path is a common baseline; the allowlist is reusable guidance.

ProblemThe patternWhere your platform lesson lands itYou decide
How the app emits telemetryOTLP to a collector; the app knows one endpoint variable34: in-process Microsoft OpenTelemetry Distro to Azure Monitor · 35: collector to X-Ray · 36: collector to Cloud Trace, if your boundary lists itWhich backend your boundary allows, and who edits the collector configuration
Where the backend address and credentials come fromThe collector’s configuration, never the app’s source34: the portal’s connection string, never a typed host · 35: IAM role on the collector · 36: service account on the collectorWho is allowed to change that configuration
Where secrets liveplatform.secrets.get(name)34: Key Vault · 35: Secrets Manager · 36: Secret ManagerRotation owner and access reviewers
What reaches telemetryAn allowlist of field namesSame on every platformWhich names are on the list, and who reviews a change to it

Plan the change with your agent

Open your workspace agent at the project root with the project-memory file from Lesson 2 in place, switch to Plan mode (Shift+Tab, /plan, or the desktop app’s mode picker), and ask:

Read src/platform/types.ts and our project-memory file, and plan server-side OpenTelemetry tracing over OTLP plus a field allowlist that strips secrets, headers, and query strings from logs. Ask me anything unclear.

What to check in the plan before you approve

  • src/instrumentation.ts exports register() and, when process.env.NEXT_RUNTIME === "nodejs", dynamically imports ./lib/telemetry/register and awaits registerTelemetry("intake-app"), which delegates to platform.telemetry.start.
  • src/lib/telemetry/start.ts exports startTelemetry(serviceName) using NodeSDK and OTLPTraceExporter reading OTEL_EXPORTER_OTLP_ENDPOINT (with no cloud-specific telemetry packages or hardcoded backend hosts), logging one warning and returning when the variable is unset.
  • src/lib/telemetry/safe-fields.ts exports safeFields(input), keeping only method, path (with any query string stripped), status, durationMs, correlationId, route, and errorName.
  • A unit test plants an Authorization header, a session cookie, an email address, and a ?token= query string and asserts none survive safeFields().

If the first plan installs a vendor-specific telemetry SDK in core code or logs raw URLs with query parameters, nudge it before approving:

Do not hardcode a backend telemetry host; emit OTLP to OTEL_EXPORTER_OTLP_ENDPOINT and strip query strings in safeFields.

What to expect

The app side is one function. It is real code with a passing test in the module’s runnable reference.

REFERENCE CODE src/lib/telemetry/start.ts: NodeSDK OTLP trace exporter initialization open reference hide reference Reference implementation from gov-dev-reference-app; open to inspect or compare with what your agent builds.
src/lib/telemetry/start.ts
// One telemetry path on every platform (lesson 16): the app speaks OTLP to a collector
// at OTEL_EXPORTER_OTLP_ENDPOINT and knows nothing else. Where spans go from there is
// the collector's configuration, which each platform lesson supplies (34 Azure Monitor,
// 35 X-Ray, 36 Cloud Trace). Locally the collector just prints them.
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { resourceFromAttributes } from '@opentelemetry/resources';
let sdk: NodeSDK | undefined;
export function startTelemetry(serviceName: string): NodeSDK | undefined {
if (sdk) return sdk;
if (!process.env.OTEL_EXPORTER_OTLP_ENDPOINT) {
console.warn('telemetry disabled: OTEL_EXPORTER_OTLP_ENDPOINT is not set');
return undefined;
}
sdk = new NodeSDK({
resource: resourceFromAttributes({ 'service.name': serviceName }),
traceExporter: new OTLPTraceExporter(), // reads OTEL_EXPORTER_OTLP_ENDPOINT itself
});
sdk.start();
return sdk;
}

The startup hook does not call startTelemetry itself. It asks the platform through this reference helper in gov-dev-reference-app, so a platform whose backend wants something other than a collector (Azure, in lesson 34) can start that instead:

src/lib/telemetry/register.ts
// What the app's startup hook calls (lesson 16: Next.js src/instrumentation.ts register()).
// It goes through the seam, so each platform starts its own telemetry: local and the
// collector-based platforms call startTelemetry() (OTLP); Azure starts its in-process distro.
import { getPlatform } from '../../platform/index.ts';
export async function registerTelemetry(serviceName: string): Promise<void> {
const platform = await getPlatform();
await platform.telemetry.start(serviceName);
}

And here is the reference startup hook your agent writes in src/instrumentation.ts:

src/instrumentation.ts
export async function register() {
if (process.env.NEXT_RUNTIME === "nodejs") {
const { registerTelemetry } = await import("./lib/telemetry/register");
await registerTelemetry("intake-app");
}
}

The instrumentation.ts hook is Next.js’s own mechanism for running code at server start; that the OpenTelemetry Node SDK works from inside it was not tested for this lesson, so confirm that you see traces before you trust it.

The local collector

On a laptop the collector just prints what it receives. The platform lessons replace its exporters block; the app does not change.

REFERENCE CONFIG Local OpenTelemetry collector configuration (OTLP HTTP receiver to debug exporter) open reference hide reference Reference pattern for your workspace agent; open to inspect or compare this file. You do not type or paste this by hand.
# Local collector for lesson 16: receives OTLP from the app and prints it.
# Platform lessons replace the exporters block; the app does not change.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [debug]

Start it with npm run local:up from the reference folder (it also starts the database and the sign-in provider from earlier lessons). Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 for the app.

ℹWhere spans go on your cloud

The destination is not in this lesson’s code. Lesson 34 uses the in-process Microsoft OpenTelemetry Distro, which sends to Azure Monitor with Microsoft Entra authentication and takes its ingestion address from the portal’s connection string. Lesson 35 sends traces through a collector to X-Ray; the CloudWatch OTLP traces path needs a feature called Transaction Search that AWS GovCloud lists as unavailable. Lesson 36 sends through a collector to Cloud Trace, which the Assured Workloads page lists for FedRAMP High but not in its IL4 product table, so verify the product list for your control package first.


Secrets through the seam

A connection to a database, a client secret, a third-party key: none belongs in source, in a committed .env file, or in a plain pipeline variable. The platform seam gives the app one door: platform.secrets.get(name). The sign-in client secret, for example, is read with platform.secrets.get(platform.identity.clientSecretName), as in Lesson 3. Locally the secret is a file in ~/.config/m14-reference; on a cloud it is that cloud’s secret store.

Lesson 34 shows the Azure Government implementation: Key Vault, the vault suffix, and how a rotated value reaches the app. The pattern here is the call site, which does not change across platforms.


What never gets logged

Telemetry is a copy of your system’s behavior, stored somewhere with a different audience and a different retention period than your database. The rule is an allowlist: a log or span attribute carries only fields from a short list, and anything else is dropped by default.

Never logWhy it is easy to leak
Authorization headers, session cookies, tokens, client secretsRequest-logging helpers dump all headers by default
Request and response bodiesThey carry whatever users type, including identifiers and case details
Names, email addresses, government identifiersAdded “for debugging” and kept for the retention period
Full URLs with query strings?token= and ?email= style parameters ride along
SQL text with bound values, stack traces returned to a clientThey reveal schema and data; Lesson 7 covers errors as a contract
Anything marked CUI (controlled unclassified information, the federal category for sensitive-but-unclassified data)Handling rules follow the data into every system that stores it; take the determination from your CUI program office

What you do log: method, route pattern, status, duration, a correlation id, and the error’s name. That is enough to find a failing request without storing who made it or what it said. Here is the reference allowlist function your agent generates in src/lib/telemetry/safe-fields.ts:

src/lib/telemetry/safe-fields.ts
const ALLOWED = ["method", "route", "status", "durationMs", "correlationId", "errorName"] as const;
export function safeFields(input: Record<string, unknown>): Record<string, unknown> {
const out: Record<string, unknown> = {};
for (const key of ALLOWED) if (key in input) out[key] = input[key];
if (typeof input.path === "string") out.path = input.path.split("?")[0];
return out;
}

See it in action

SESSION Telemetry: a session with an AI CLI
  1. Typed in the terminal: claude
  2. Mode:Plan mode
  3. You type in the session: Add telemetry to the server. The app should work on any platform we land it on.
  4. The tool replies: Before I write anything, two questions so I do not guess: 1. Backend: I will not import a vendor package. The app will send OTLP to whatever is at OTEL_EXPORTER_OTLP_ENDPOINT, and the collector decides where spans go. 2. Name: I will pass the service name into startTelemetry so every span carries service.name. I will create src/instrumentation.ts, src/lib/telemetry/start.ts and src/lib/telemetry/safe-fields.ts, plus a test. Proceed?
  5. You type in the session: Proceed. Also send them straight to the Government endpoint so we skip the collector.
  6. The tool replies: I will not type a backend address. Which host is right depends on the platform, and a wrong one fails silently. Your platform lesson supplies it, in the collector's configuration or, on Azure, in the connection string from the portal. Created: src/instrumentation.ts (register(), nodejs runtime only) src/lib/telemetry/start.ts (skips with one warning if the variable is unset) src/lib/telemetry/safe-fields.ts (allowlist) src/lib/telemetry/safe-fields.test.ts Note: I have not verified that the SDK starts from inside Next's register() hook. Run the app and confirm a span arrives before relying on it.
  7. You type in the session: Run it against the local collector and show me a span arrived.
  8. The tool replies: $ npm run local:up $ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run test:env 6 tests passed The telemetry test creates a span named intake.submit-<timestamp>, shuts the SDK down to flush it, then reads the collector's log. It found the span name and the resource attribute service.name: Str(m14-reference).
  9. You type in the session: Log the full request headers on 500s so we can debug faster.
  10. The tool replies: I would rather not. Request headers include the Authorization header and the session cookie, and they would be stored for the retention period of the telemetry backend. What I can log on a 500 is method, route, status, duration, the correlation id, and the error name through safeFields(). If you need more for one incident, I can add a temporary, named field to the allowlist behind a code review, and remove it afterward. Want that instead?

Illustrative session; exact output varies by tool and agency configuration.


Prove it

Two tests, one for each promise. From gov-dev-reference-app, with the local stack up:

Terminal window
npm run test:env

Pass: 6 tests pass. One of them creates a span, flushes it, and reads the collector’s output: the span’s name appears, and so does service.name: Str(m14-reference). That is the proof the app speaks OTLP and the collector hears it. Fail, which you should see once on purpose: stop the collector (docker stop m14-otel) and the telemetry test goes red.

Then the allowlist, in your own project:

Terminal window
npx vitest run src/lib/telemetry/safe-fields.test.ts

Pass: the test plants Authorization, cookie, an email address, and ?token=abc123 in a request-shaped object and asserts none appear in the result. Fail, which you should see once on purpose: add "cookie" to the allowlist and the test goes red.

Finally check the backend your platform lesson configured: send one request and look for it in that backend’s trace view. If nothing appears, check the collector’s log first, then its credentials, then the endpoint variable.


KNOWLEDGE CHECK

An AI tool offers to send spans from the app straight to a cloud backend's host so you can skip the collector. What is the right response?

KNOWLEDGE CHECK

Which log record fits the allowlist rule for a failed request?

Practice status

PracticeStatusAlso called
App speaks OTLP to a collectorcommon baselinecollector-fronted telemetry
Field allowlist for telemetrystrong optionallog scrubbing by default-deny
Secrets through one platform callcommon baselinesecret-store abstraction

Scale: required | common baseline | strong optional | reference-shop (seen only at organizations that publish their own practice) | emerging

Key takeaway

Telemetry is a second copy of your system’s behavior, so give it the same care as the first. The app emits OTLP to one endpoint, the collector decides where spans go, secrets come through platform.secrets.get, and every log field comes from an allowlist. Your platform lesson (34, 35 or 36) holds the backend, its authentication and its secret store. Next, Lesson 17 covers how the code that carries all of this reaches a repository without a long-lived token.

LEADERSHIP DECISION
Which telemetry backend your authorization boundary allows, and who may change the collector configuration.
PRACTITIONER ACTION
Run the env test until a span reaches the collector, then add the allowlist test to CI.
SUCCESS MEASURE
Contract or revenue risk: no secret or personal field reaches a store with a longer retention than the database.

Legend: LEADERSHIP DECISION = what a leader decides · PRACTITIONER ACTION = what the engineer does next · SUCCESS MEASURE = how you know, in calendar time, assessor-hours or cost, contract or revenue risk, or audit finding avoided

Search lessons