Files
software-mansion__argent/packages/telemetry/test/otel-endpoint-live.test.ts
Ignacy Łątka 16ad44bb0e test(telemetry): pin the wire format and what a dead collector costs (#736)
Two test files that look at what #570's exporter actually *does*, plus
the configuration holes they exposed.

The suite on #570 proves the exporter is **configured** right - the url,
the headers, the timeouts, all read off the constructor. Nothing looked
at the request that comes out of it, or at what happens when the
collector does not answer. Both gaps are invisible from inside Argent by
design: a failed export never reaches the caller, and an OTLP exporter
counts any 2xx as delivered.

| | |
|---|---|
| **`otel-wire.test.ts`** | Captures the real exporter's request off a
loopback socket: uncompressed OTLP/JSON on `POST /v1/logs`, where each
field the ingestion schema reads actually sits, the batch-size cap, and
connection reuse. |
| **`otel-unreachable.test.ts`** | Drives the same exporter at a silent
socket, a refused port, a 503, a 401 and a 204, and asserts each drain
finishes inside its deadline and what the collector really received. |
| **`src/otel.ts`** | Two holes the tests exposed: `compression` and the
six OTLP certificate variables are now settled in code instead of left
to the environment. |

**No docs update needed** - no user-facing capability, CLI surface,
config key, flow file or MCP tool changed.

<details>
<summary><b>The two source fixes</b></summary>

**`compression` was the one OTLP knob still left to the environment.**
With `OTEL_EXPORTER_OTLP_COMPRESSION=gzip` (or its `_LOGS_` variant) set
- and a machine already running OpenTelemetry has it set for its own
collector - argent gzips its batches. Nothing downstream notices: a
gzipped 2xx is a delivered batch, while the request this suite and the
deploy-time smoke test are written against quietly stops being the
request argent sends. `createExporter` now passes `compression`
explicitly, which beats both variables. Unlike the header channel, where
the SDK *merges* whatever the code does not set and the variables have
to be cleared outright, one explicit value settles this.

**The certificate variables could hang the command outright.** The SDK
resolves
`OTEL_EXPORTER_OTLP_{,LOGS_}{CERTIFICATE,CLIENT_CERTIFICATE,CLIENT_KEY}`
with a synchronous `fs.readFileSync` while the exporter is constructed,
and only then discards the https agent it built from them in favour of
the explicit `httpAgentOptions`. The file is read for nothing - and a
path that never answers hangs the command that emitted the event,
uninterruptibly, on `getClient()`'s path. Measured with a fifo that has
no writer: `createExporter` never returned (killed at 12s); with the
variables cleared, 1.0 ms. They join the header pair in the list cleared
across the constructor and restored on the way out.

Some source prose was also wrong about the SDK and is corrected: it does
**not** reject a null attribute - it serializes one as an empty OTLP
value, `{"key":"cloud_agent","value":{}}` with `droppedAttributesCount:
0`, stored as and unrecoverable from a property that really was empty.
That is the actual reason `toAttributes` drops the key.

</details>

<details>
<summary><b>What the tests pin, and what they deliberately do
not</b></summary>

The wire test **mirrors** `OtelClient.emit` rather than calling it:
`getClient()` resolves its endpoint from the hard-coded
`OTLP_LOGS_ENDPOINT`, and that being unredirectable is the
anti-exfiltration property the client is meant to have, so there is
deliberately no seam to aim it at a test server. `otel-endpoint.test.ts`
covers the other half against the real emit path - `SERVICE_NAME`, the
logger name, event-name-as-body, the dropping of null/undefined
properties - so a rename is caught there rather than in the mirror.

What enforces the drain budgets is the exporter's `timeoutMillis`, *not*
the processor's `exportTimeoutMillis`, which `forceFlush()` awaits
straight past. Measured against a silent socket: `exportTimeout` 200 /
`timeoutMillis` 5000 drains in 5012 ms, the reverse in 202 ms.

503 is the one collector state where the batch is sent twice, and the
request count is the only thing that tells a retried failure from a
rejected one - the SDK reports the class, not the code. The retry itself
is only reachable while the next backoff fits in what is left of
`timeoutMillis`, so the jitter is pinned to its low draw and the
assertion turns on the round trip alone.

`httpAgentOptions.timeout` is the one option with no behavioural
coverage: it covers a socket stuck in *connect*, which needs an address
whose packets are dropped and is not portably reproducible here. It
stays pinned at the constructor and the docstring says so. Its
`keepAlive` half **is** covered, on the two batches of a single drain.

Real deadlines cost real time: `otel-unreachable.test.ts` waits ~5.1 s,
taking the package suite from ~0.9 s to ~5.3 s. Inherent to driving real
deadlines against real sockets.

</details>

<details>
<summary><b>Verification</b></summary>

- `npm test -w @argent/telemetry` - **336 passed**, 19 files
- `typecheck:tests`, `eslint --max-warnings 0`, `prettier --check`,
`knip`, `tsc --build` all clean; CI green
- **Mutation-tested**, whole-suite counts rather than one test each:

  | mutation in `src/otel.ts` | tests red |
  |---|---|
  | `compression` pin dropped | 2 |
  | `compression` pinned to `"gzip"` | 5 |
  | any one of the 8 cleared env-var names dropped | 1 each |
  | env-clearing loop deleted | 2 |
  | the `finally` that restores it deleted | 1 |
  | `timeoutMillis` dropped | 5 |
  | `httpAgentOptions` dropped, or just its `timeout` | 1 |
  | `keepAlive` flipped to `false` | 2 |
  | `installDiagLogger`'s `!isDebugEnabled()` guard removed | 1 |
| `maxExportBatchSize` 20 → 19 or 25; `scheduledDelayMillis` changed | 2
|
  | `SERVICE_NAME` renamed | 2 |
  | `LOGGER_NAME` changed | 1 |
  | record body no longer the event name | 4 |
  | `severityText` INFO → WARN | 1 |
  | null/undefined property skip removed | 2 |
  | `url` given a suffix the config did not set | 9 |

- **Absence assertions carry positive controls.** The certificate test
reads its answer out of a `Failed to read ` prefix that lives in a
transitive dependency, and the 204 case asserts an empty diag channel;
both would pass just as well against a dead observer. Each now proves
its own observer live first.
- **Flake-tested.** The 503 case's exact two-request assertion used to
turn on the jitter draw: standing a 500 ms collector delay in for a
loaded runner made it fail 20/20 (13 on the request count, 7 on
`Timeout` masking the error class). With the jitter pinned, 20/20 pass.

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved telemetry export reliability by preventing unintended
compression and certificate-related environment settings from affecting
exporter configuration.
- Ensured telemetry attributes with null or undefined values are omitted
rather than serialized as misleading empty values.
- Improved handling of unreachable or unresponsive telemetry collectors
with bounded failures and appropriate diagnostic reporting.
- Preserved connection reuse and batching behavior for more efficient
OTLP delivery.

- **Tests**
- Added comprehensive coverage for OTLP formatting, authorization,
batching, compression, certificates, connection handling, and failure
scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 12:39:35 +02:00

251 lines
11 KiB
TypeScript

/**
* What actually goes on the wire, proven against the REAL OpenTelemetry
* exporter rather than a mock of it.
*
* `otel-endpoint.test.ts` mocks `@opentelemetry/exporter-logs-otlp-http`, so it
* can only show what `otel.ts` passes to the constructor — it cannot show what
* the SDK then does with `OTEL_EXPORTER_OTLP_*`. An SDK
* upgrade that changed that handling would keep every assertion there green
* while telemetry started going somewhere else, or carrying something extra.
*
* So this drives `createExporter` — the real function, including its env
* handling — against loopback servers: one standing in for the hard-coded
* collector, one for what a hostile env var would name. No network is involved.
* `resolveConfig()` is what pins the endpoint itself to the production URL, and
* `otel-endpoint.test.ts` covers that.
*/
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import http from "node:http";
import { BatchLogRecordProcessor, LoggerProvider } from "@opentelemetry/sdk-logs";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { SeverityNumber } from "@opentelemetry/api-logs";
import { diag, DiagLogLevel } from "@opentelemetry/api";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-http";
import { createExporter } from "../src/otel.js";
import { listenLoopback, snapshotEnv } from "./helpers.js";
interface Capture {
url: string;
requests: Array<{ path: string; headers: http.IncomingHttpHeaders }>;
}
const closers: Array<() => Promise<void>> = [];
async function startCapture(): Promise<Capture> {
const requests: Capture["requests"] = [];
const server = http.createServer((req, res) => {
req.resume();
req.on("end", () => {
requests.push({ path: req.url ?? "", headers: req.headers });
res.writeHead(200, { "content-type": "application/json" });
res.end("{}");
});
});
const port = await listenLoopback(server);
closers.push(() => new Promise<void>((resolve) => server.close(() => resolve())));
return { url: `http://127.0.0.1:${port}/v1/logs`, requests };
}
const OTLP_ENV_VARS = [
"OTEL_EXPORTER_OTLP_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
"OTEL_EXPORTER_OTLP_HEADERS",
"OTEL_EXPORTER_OTLP_LOGS_HEADERS",
"OTEL_EXPORTER_OTLP_PROTOCOL",
"OTEL_EXPORTER_OTLP_COMPRESSION",
"OTEL_EXPORTER_OTLP_LOGS_COMPRESSION",
"OTEL_EXPORTER_OTLP_CERTIFICATE",
"OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE",
"OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE",
"OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE",
"OTEL_EXPORTER_OTLP_CLIENT_KEY",
"OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY",
];
let restoreEnv: () => void;
let code: Capture;
let hostile: Capture;
beforeEach(async () => {
restoreEnv = snapshotEnv(OTLP_ENV_VARS);
code = await startCapture();
hostile = await startCapture();
});
afterEach(async () => {
restoreEnv();
await Promise.all(closers.splice(0).map((close) => close()));
});
/** Emit one record through the same provider wiring `OtelClient` builds. */
async function exportOneRecord(endpoint: string, token: string): Promise<void> {
const provider = new LoggerProvider({
resource: resourceFromAttributes({ "service.name": "argent" }),
processors: [
new BatchLogRecordProcessor({
exporter: createExporter({ endpoint, token, isUsable: true }),
maxExportBatchSize: 20,
scheduledDelayMillis: 10_000,
exportTimeoutMillis: 1_500,
}),
],
});
provider.getLogger("@argent/telemetry").emit({
severityNumber: SeverityNumber.INFO,
severityText: "INFO",
body: "endpoint_invariance_probe",
attributes: { "distinct_id": "probe", "event.name": "endpoint_invariance_probe" },
});
// shutdown() force-flushes the batch and then tears the provider down, so the
// export has been attempted by the time it resolves.
await provider.shutdown();
}
describe("what reaches the collector, against the real OTLP exporter", () => {
it("delivers to the url passed in code while every OTLP env var names another host", async () => {
process.env.OTEL_EXPORTER_OTLP_ENDPOINT = hostile.url.replace("/v1/logs", "");
process.env.OTEL_EXPORTER_OTLP_LOGS_ENDPOINT = hostile.url;
// Inert against this SDK - protocol selection lives in @opentelemetry/sdk-node,
// which is not a dependency. Set so a version that starts reading it fails here.
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
await exportOneRecord(code.url, "real-ingest-token");
expect(hostile.requests).toHaveLength(0);
expect(code.requests).toHaveLength(1);
expect(code.requests[0]!.path).toBe("/v1/logs");
}, 15_000);
it("keeps the code-supplied bearer token when an env var supplies another", async () => {
// Getting redirected is one failure; shipping to the right collector under
// an attacker's token is another. The explicit header has to win.
process.env.OTEL_EXPORTER_OTLP_HEADERS = "authorization=Bearer steal-me";
process.env.OTEL_EXPORTER_OTLP_LOGS_HEADERS = "authorization=Bearer steal-me-logs";
await exportOneRecord(code.url, "real-ingest-token");
expect(code.requests).toHaveLength(1);
expect(code.requests[0]!.headers.authorization).toBe("Bearer real-ingest-token");
}, 15_000);
it("forwards no third-party header the environment names", async () => {
// A developer who already runs OpenTelemetry keeps their own vendor
// credential in exactly this variable. The SDK merges any key the code does
// not set, so without createExporter clearing it, that secret would ship to
// Software Mansion's collector on every batch.
process.env.OTEL_EXPORTER_OTLP_HEADERS = "x-honeycomb-team=hcaik_REAL_CUSTOMER_KEY";
process.env.OTEL_EXPORTER_OTLP_LOGS_HEADERS = "x-dt-api-token=dt0c01.LEAKED";
await exportOneRecord(code.url, "real-ingest-token");
expect(code.requests).toHaveLength(1);
const headers = code.requests[0]!.headers;
expect(headers["x-honeycomb-team"]).toBeUndefined();
expect(headers["x-dt-api-token"]).toBeUndefined();
expect(headers.authorization).toBe("Bearer real-ingest-token");
}, 15_000);
it("posts the encoding the code chose when an env var asks for gzip", async () => {
// A machine that already runs OpenTelemetry sets this for its own collector,
// and the encoding argent posts is not that machine's to pick.
// otel-wire.test.ts captures the uncompressed request the ingestion side is
// sized and smoke-tested against.
process.env.OTEL_EXPORTER_OTLP_COMPRESSION = "gzip";
process.env.OTEL_EXPORTER_OTLP_LOGS_COMPRESSION = "gzip";
await exportOneRecord(code.url, "real-ingest-token");
expect(code.requests).toHaveLength(1);
expect(code.requests[0]!.headers["content-encoding"]).toBeUndefined();
}, 15_000);
it("reads no file an OTLP certificate variable names", () => {
// The SDK resolves these with a synchronous fs.readFileSync while the
// exporter is constructed, and only then discards the https agent it built
// from them in favour of the explicit httpAgentOptions. So the read is for
// nothing - and a path that never answers, a dead network mount or a fifo
// with no writer, hangs the command that emitted the event instead
// (measured: the constructor never returned). createExporter clears the
// variables so the read is never attempted; the SDK emits a "Failed to
// read ..." warning only when it does attempt one and fails, so the absence
// of that warning is what says the clear held.
const warnings: string[] = [];
diag.setLogger(
{
error: () => {},
warn: (message) => warnings.push(String(message)),
info: () => {},
debug: () => {},
verbose: () => {},
},
DiagLogLevel.WARN
);
// Both spellings: the SDK reads `signalSpecific ?? nonSignalSpecific`, so
// the _LOGS_ form wins - and it is the one a machine already exporting logs
// elsewhere has set.
process.env.OTEL_EXPORTER_OTLP_CERTIFICATE = "/nonexistent/argent-probe-ca.pem";
process.env.OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE = "/nonexistent/argent-probe-logs-ca.pem";
process.env.OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE = "/nonexistent/argent-probe-cert.pem";
process.env.OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE =
"/nonexistent/argent-probe-logs-cert.pem";
process.env.OTEL_EXPORTER_OTLP_CLIENT_KEY = "/nonexistent/argent-probe-key.pem";
process.env.OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY = "/nonexistent/argent-probe-logs-key.pem";
try {
createExporter({ endpoint: code.url, token: "real-ingest-token", isUsable: true });
} finally {
diag.disable();
}
// Only the certificate-read warnings, not the whole diag stream: an invalid
// OTEL_EXPORTER_OTLP_COMPRESSION or _TIMEOUT in the ambient environment
// warns unconditionally, and createExporter deliberately leaves those two in
// place (the explicit option already beats them), so matching every warning
// would fail this cert test on an unrelated env var.
const certReadWarnings = warnings.filter((message) => message.startsWith("Failed to read "));
expect(certReadWarnings).toEqual([]);
// Positive control: that prefix is a string in a transitive dependency, and
// if it is ever reworded this test starts passing by seeing nothing. The
// variables are still set - createExporter restores them - so an exporter
// built without the clear has to produce one.
const warningsWithoutClear: string[] = [];
diag.setLogger(
{
error: () => {},
warn: (message) => warningsWithoutClear.push(String(message)),
info: () => {},
debug: () => {},
verbose: () => {},
},
DiagLogLevel.WARN
);
try {
new OTLPLogExporter({ url: code.url, headers: { authorization: "Bearer x" } });
} finally {
diag.disable();
}
expect(warningsWithoutClear.filter((m) => m.startsWith("Failed to read "))).not.toEqual([]);
});
it("leaves the OTLP environment as it found it", () => {
// Clearing the header variables is a means, not a side effect to inflict on
// the rest of the process — anything else in this CLI that reads them later
// must still see what the user set. The compression variable is here for the
// opposite reason: the explicit option already beats it, so winning by
// clobbering it would be a strictly worse implementation this suite would
// otherwise be unable to tell from the shipped one.
process.env.OTEL_EXPORTER_OTLP_HEADERS = "x-vendor=keep-me";
delete process.env.OTEL_EXPORTER_OTLP_LOGS_HEADERS;
process.env.OTEL_EXPORTER_OTLP_CERTIFICATE = "/etc/ssl/corp-ca.pem";
process.env.OTEL_EXPORTER_OTLP_COMPRESSION = "gzip";
createExporter({ endpoint: code.url, token: "real-ingest-token", isUsable: true });
expect(process.env.OTEL_EXPORTER_OTLP_HEADERS).toBe("x-vendor=keep-me");
expect("OTEL_EXPORTER_OTLP_LOGS_HEADERS" in process.env).toBe(false);
expect(process.env.OTEL_EXPORTER_OTLP_CERTIFICATE).toBe("/etc/ssl/corp-ca.pem");
expect(process.env.OTEL_EXPORTER_OTLP_COMPRESSION).toBe("gzip");
});
});