feat: buffer connection logs and flush them on failure (#1100)

Implements RFC requirement 13 / DEVEX-669: buffer connection debug logs
in memory below the current log level and flush them on a genuine
connection failure, so a support bundle captures the detail leading up
to the failure without the user having enabled debug logging beforehand.

## What this does

- Adds a `BufferingLogger` decorator (`src/logging/logBuffer.ts`) that
wraps the "Coder" output channel and keeps a bounded, in-memory ring of
the entries that sit **below** the channel's current level, which it
would otherwise drop. Only below-level entries are buffered, so nothing
already written is duplicated. Entries are formatted (via
`safeStringify`) at record time.
- On a connection failure, `flush(reason)` re-emits the captured entries
into the output channel at the least-verbose level the channel still
persists. The first physical line of each entry carries a `[buffered]`
marker with its original ISO timestamp and level; continuation lines
carry the bare marker. Capture is best-effort — the channel writes on
its own schedule — but a later failure flush replays anything a bundle
missed.
- Collecting a support bundle flushes the buffer
(`flush("support_bundle", { retain: true })`) before the CLI runs, and
**retains** the ring so a later failure flush still has the entries.
- When the channel is at **Off**, flush keeps the entries buffered
instead of discarding them.
- The buffer is bounded by both entry count and characters
(`MAX_BUFFERED_CHARS = 2_000_000`); `trim()` enforces both budgets with
oldest-eviction, and `flush()` replays in chunks of 100 to avoid a
single oversized write.
- Wires the buffer into `ServiceContainer` and adds the
`coder.connectionLogBuffer.size` setting (default `1000`, capped at
`10000`, `0` disables). The setting readers live in
`src/settings/logger.ts` and the container reads/watches the size
through a single `readSize()` helper via `watchConfigurationChanges`.
The setting is included in `COLLECTED_SETTINGS`.
- Flushes only on genuine connection failures, gated by an explicit
`failure?: boolean` on the connection-log reason (not on transient
reconnects, a handshake `401`, or intentional teardown):
- a reconnecting WebSocket terminal failure (`unrecoverable_close`,
`unrecoverable_http` where the status is not `401`, or
`certificate_error`);
- a failure while opening a workspace (canceled build, missing agent,
timeout, or CLI/certificate error), funnelled through
`Remote.closeRemote()`.
- Connection-failure flushes are funnelled through
`BufferingLogger.onConnectionFailure(reason, route)` (part of
`ConnectionLogBuffer`), so the extension and remote paths flush
`${reason} ${route}` identically; the socket option defaults to a noop
when no observer is supplied.
- The flush reason carries the failing **route** for attribution; the
route is seeded on the socket so even a first-connect failure logs a
real route instead of `unknown`.
- Handshake status is parsed by a shared `handshakeStatus(error)`
(`src/websocket/utils.ts`) covering both `ws` (`Unexpected server
response: <code>`) and `eventsource` (`Non-200 status code (<code>)`),
so a host/port such as `127.0.0.1:4040` is no longer misread as HTTP 404
and the SSE path is handled. `CoderApi.is404Error` compares against
`HttpStatusCode.NOT_FOUND`.
- Redacts `registration_access_token` in HTTP body logging alongside the
other sensitive fields.
- Documents the behavior, config, hard-kill/OOM loss limitation, and SSH
log scope in `CONTRIBUTING.md`.

## WebSocket event fix

`OneWayWebSocket` now registers `open`/`close`/`error` via DOM-style
`addEventListener`, so `close` consumers receive a real `CloseEvent`
with `.code`/`.reason`. This makes `unrecoverable_close` reachable in
production (message events still use `ws.on("message", ...)` for JSON
parsing). Server-initiated normal closes (`1000`/`1001`) now go through
`scheduleReconnect` rather than parking the socket, so the
now-unreachable `normal_close` reason is dropped from the telemetry
unions and `EVENTS.md`.

## Scope notes

- Extension SSH debug logs that pass through the shared logger are
buffered; the CLI `ProxyCommand` file logs under
`coder.proxyLogDirectory` are not, since support bundles already collect
them from disk.
- The buffer lives in memory, so a hard kill or out-of-memory event
loses it (documented).
- Flushing after N consecutive failed reconnect attempts against an
unreachable server is deferred to follow-up #1112.
- Reading the handshake status from the libraries' public event APIs
(instead of parsing their error strings, which a real-library test now
guards) is deferred to follow-up #1118.

## Testing

- `pnpm typecheck`, `pnpm format:check`, and `pnpm lint` are clean.
- Affected/dependent unit suites pass (`logBuffer`, `settings/logger`,
`formatters`, `reconnectingWebSocket`, `oneWayWebSocket`, `coderApi`,
`workspaceMonitor`, `workspaceStateMachine`, `remote`,
`commands.supportBundle`, `instrumentation/websocket`).

<details>
<summary>Implementation plan &amp; design decisions</summary>

### Design

- **Buffer:** bounded by entry count and characters; captures only calls
whose severity is below the channel's current level; formats entries at
record time; oldest-eviction; live-resizable via config; replays in
chunks.
- **Flush target (D3):** replay into the existing "Coder" output channel
at a level that still persists, with a `[buffered]` marker plus original
level/timestamp, chronologically next to the real failure logs. Support
bundles already collect the on-disk VS Code logs and also trigger a
flush before appending them, so no separate sink is needed. At Off,
entries are retained rather than discarded.
- **Flush reasons (D4):** genuine, surfaced connection failures only,
gated by an explicit `failure?: boolean` — reconnecting-socket terminal
failures (except a `401` handshake) and a workspace-open failure
funnelled through `closeRemote()`. Never on transient `retrying` drops
or intentional teardown (`manual_disconnect`, `replaced`,
dispose/deactivate/reload). The callback carries the failing route.
- **SSH scope (D5):** buffer extension SSH debug passing through the
shared `Logger`; do not buffer CLI `ProxyCommand` file logs already
handled via `coder.proxyLogDirectory`.

### Decisions

- D1: buffer all below-level session logs.
- D2: bound by entry count (and a character budget to cap memory).
- D3: replay into the existing Coder output channel at a persisted level
with `[buffered]` marker/original level/timestamp; retain at Off; also
flush on support-bundle collection.
- D4: flush only on genuine connection failure; not transient, not a
`401`, and not intentional teardown.
- D5: buffer extension SSH debug through the shared `Logger`; not CLI
ProxyCommand file logs.

</details>

---

🤖 Generated with Coder Agents. Reviewed and authored on behalf of
@aqandrew.

---------

Co-authored-by: Ehab Younes <ehab.alyounes@gmail.com>

Author: aqandrew

CHANGELOG.md

@@ -5,6 +5,23 @@
      from published versions since it shows up in the VS Code extension changelog
      tab and is confusing to users. Add it back between releases if needed. -->
 
+## Unreleased
+
+### Added
+
+- Add `coder.connectionLogBuffer.size` to keep recent below-level Coder output
+  channel entries in memory and replay them into the channel when a connection
+  fails terminally or you collect a support bundle, so Support can diagnose
+  connection failures without asking you to reproduce with debug logging on. Set
+  to `0` to disable.
+
+### Fixed
+
+- Stop retrying after an unrecoverable WebSocket close (`1002`/`1003`) and
+  replay the buffered connection logs instead. Close codes never reached the
+  reconnect logic, so these closes retried forever. Server-initiated normal
+  closes (`1000`/`1001`) keep reconnecting.
+
 ## [v1.16.3](https://github.com/coder/vscode-coder/releases/tag/v1.16.3) 2026-09-14
 
 ### Changed

CONTRIBUTING.md

@@ -132,6 +132,32 @@ next to the code:
 
 **[`src/instrumentation/CONVENTIONS.md`](src/instrumentation/CONVENTIONS.md)**
 
+## Logging
+
+The extension logs to the "Coder" output channel, a `LogOutputChannel` that gates
+messages by the level chosen in its gear menu. To help Support diagnose
+connection failures without asking users to reproduce with debug logging enabled,
+a `BufferingLogger` ([`src/logging/logBuffer.ts`](src/logging/logBuffer.ts))
+wraps the channel and keeps a bounded, in-memory ring of the entries that sit
+**below** the current level, which the channel would otherwise drop.
+
+When a WebSocket fails terminally, a remote session closes after a failed or
+canceled open, or you collect a support bundle, the extension replays the ring
+into the channel. The first physical line of each replayed entry carries a
+`[buffered]` marker with its original timestamp and level, and any continuation
+lines carry the bare marker.
+Capture is best-effort: the channel writes on its own schedule, so a bundle may
+miss the most recent lines, but a later failure flush still replays them.
+Transient reconnects and intentional teardown never flush, and neither does a
+handshake `401` (a 401 explains itself, and with OAuth a refresh reconnects the
+same socket). Nothing is recorded or flushed while the channel is at `Off`.
+
+The buffer size is set by `coder.connectionLogBuffer.size` (number of entries;
+`0` disables it) and lives in memory, so a hard kill or out-of-memory event
+loses it. Extension SSH debug logs that pass through the shared logger are
+buffered; the CLI `ProxyCommand` file logs under `coder.proxyLogDirectory` are
+not, since support bundles already collect them from disk.
+
 ## Testing
 
 There are a few ways you can test the "Open in VS Code" flow:

package.json

@@ -215,6 +215,13 @@
 					"minimum": 0,
 					"default": 250
 				},
+				"coder.connectionLogBuffer.size": {
+					"markdownDescription": "Maximum number of log entries below the Coder output channel's log level to keep in memory. When a connection fails terminally or you collect a support bundle, the extension replays them into the channel so a bundle can carry the detail leading up to the failure without debug logging enabled beforehand. Capture is best-effort, since the channel writes on its own schedule. Set to `0` to disable.",
+					"type": "number",
+					"minimum": 0,
+					"maximum": 10000,
+					"default": 1000
+				},
 				"coder.httpClientLogLevel": {
 					"markdownDescription": "Controls the verbosity of HTTP client logging. This affects what details are logged for each HTTP request and response.",
 					"type": "string",

src/api/coderApi.ts

@@ -26,12 +26,13 @@ import {
 } from "../logging/httpLogger";
 import { HttpRequestsTelemetry } from "../logging/httpRequestsTelemetry";
 import {
-	HttpClientLogLevel,
 	type RequestConfigWithMeta,
+	type HttpClientLogLevel,
 } from "../logging/types";
 import { sizeOf } from "../logging/utils";
 import { AuthConfigTracker } from "../settings/authConfig";
 import { getHeaderCommand } from "../settings/headers";
+import { readHttpClientLogLevel } from "../settings/logger";
 import {
 	NOOP_TELEMETRY_REPORTER,
 	type TelemetryReporter,
@@ -48,6 +49,7 @@ import {
 	type SocketFactory,
 } from "../websocket/reconnectingWebSocket";
 import { SseConnection } from "../websocket/sseConnection";
+import { handshakeStatus } from "../websocket/utils";
 
 import { getRefreshCommand, refreshCertificates } from "./certificateRefresh";
 import {
@@ -69,6 +71,7 @@ import type {
 } from "coder/site/src/api/typesGenerated";
 import type { ClientOptions } from "ws";
 
+import type { ConnectionStateReason } from "../instrumentation/websocket";
 import type { Logger } from "../logging/logger";
 import type {
 	CloseEvent,
@@ -125,6 +128,10 @@ export class CoderApi extends Api implements vscode.Disposable {
 		private readonly telemetry: TelemetryReporter,
 		private readonly httpRequestsTelemetry: HttpRequestsTelemetry,
 		private readonly authConfigTracker: AuthConfigTracker,
+		private readonly onConnectionFailure?: (
+			reason: ConnectionStateReason,
+			route: string,
+		) => void,
 	) {
 		super();
 		wrapWithValidation(this);
@@ -145,6 +152,10 @@ export class CoderApi extends Api implements vscode.Disposable {
 		token: string | undefined,
 		output: Logger,
 		telemetry: TelemetryReporter = NOOP_TELEMETRY_REPORTER,
+		onConnectionFailure?: (
+			reason: ConnectionStateReason,
+			route: string,
+		) => void,
 	): CoderApi {
 		const httpRequestsTelemetry = new HttpRequestsTelemetry(telemetry);
 		const authConfigTracker = new AuthConfigTracker();
@@ -153,6 +164,7 @@ export class CoderApi extends Api implements vscode.Disposable {
 			telemetry,
 			httpRequestsTelemetry,
 			authConfigTracker,
+			onConnectionFailure,
 		);
 		client.getAxiosInstance().defaults.timeout = DEFAULT_REQUEST_TIMEOUT_MS;
 		client.getAxiosInstance().defaults.headers.common[BAGGAGE_HEADER] =
@@ -282,9 +294,10 @@ export class CoderApi extends Api implements vscode.Disposable {
 		watchTargets: string[],
 		options?: ClientOptions,
 	) => {
-		return this.createReconnectingSocket(() =>
+		const apiRoute = "/api/v2/notifications/inbox/watch";
+		return this.createReconnectingSocket(apiRoute, () =>
 			this.createOneWayWebSocket<GetInboxNotificationResponse>({
-				apiRoute: "/api/v2/notifications/inbox/watch",
+				apiRoute,
 				searchParams: {
 					format: "plaintext",
 					templates: watchTemplates.join(","),
@@ -296,9 +309,10 @@ export class CoderApi extends Api implements vscode.Disposable {
 	};
 
 	watchWorkspace = async (workspace: Workspace, options?: ClientOptions) => {
-		return this.createReconnectingSocket(() =>
+		const apiRoute = `/api/v2/workspaces/${workspace.id}/watch-ws`;
+		return this.createReconnectingSocket(apiRoute, () =>
 			this.createStreamWithSseFallback({
-				apiRoute: `/api/v2/workspaces/${workspace.id}/watch-ws`,
+				apiRoute,
 				fallbackApiRoute: `/api/v2/workspaces/${workspace.id}/watch`,
 				options,
 			}),
@@ -309,9 +323,10 @@ export class CoderApi extends Api implements vscode.Disposable {
 		agentId: WorkspaceAgent["id"],
 		options?: ClientOptions,
 	) => {
-		return this.createReconnectingSocket(() =>
+		const apiRoute = `/api/v2/workspaceagents/${agentId}/watch-metadata-ws`;
+		return this.createReconnectingSocket(apiRoute, () =>
 			this.createStreamWithSseFallback({
-				apiRoute: `/api/v2/workspaceagents/${agentId}/watch-metadata-ws`,
+				apiRoute,
 				fallbackApiRoute: `/api/v2/workspaceagents/${agentId}/watch-metadata`,
 				options,
 			}),
@@ -530,24 +545,26 @@ export class CoderApi extends Api implements vscode.Disposable {
 	 * Check if an error is a 404 Not Found error.
 	 */
 	private is404Error(error: unknown): boolean {
-		const msg = error instanceof Error ? error.message : String(error);
-		return msg.includes(String(HttpStatusCode.NOT_FOUND));
+		return handshakeStatus(error) === HttpStatusCode.NOT_FOUND;
 	}
 
 	/**
 	 * Create a ReconnectingWebSocket and track it for lifecycle management.
 	 */
 	private async createReconnectingSocket<TData>(
+		apiRoute: string,
 		socketFactory: SocketFactory<TData>,
 	): Promise<ReconnectingWebSocket<TData>> {
 		const options: ReconnectingWebSocketOptions = {
+			route: apiRoute,
 			onCertificateRefreshNeeded: async () => {
 				const refreshCommand = getRefreshCommand();
 				if (!refreshCommand) {
 					return false;
 				}
 				return refreshCertificates(refreshCommand, this.output);
 			},
+			onConnectionFailure: this.onConnectionFailure,
 			telemetry: this.telemetry,
 		};
 
@@ -814,12 +831,5 @@ function getSize(headers: AxiosHeaders, data: unknown): number | undefined {
 }
 
 function getLogLevel(): HttpClientLogLevel {
-	const logLevelStr = vscode.workspace
-		.getConfiguration()
-		.get(
-			"coder.httpClientLogLevel",
-			HttpClientLogLevel[HttpClientLogLevel.BASIC],
-		)
-		.toUpperCase();
-	return HttpClientLogLevel[logLevelStr as keyof typeof HttpClientLogLevel];
+	return readHttpClientLogLevel(vscode.workspace.getConfiguration());
 }

src/commands.ts

@@ -84,6 +84,7 @@ import type { MementoManager } from "./core/mementoManager";
 import type { PathResolver } from "./core/pathResolver";
 import type { SecretsManager, SessionAuth } from "./core/secretsManager";
 import type { DeploymentManager } from "./deployment/deploymentManager";
+import type { ConnectionLogBuffer } from "./logging/logBuffer";
 import type { Logger } from "./logging/logger";
 import type { LoginCoordinator, LoginMethod } from "./login/loginCoordinator";
 import type { TelemetryService } from "./telemetry/service";
@@ -168,6 +169,7 @@ export class Commands {
 	private readonly authTelemetry: AuthTelemetry;
 	private readonly diagnosticTelemetry: DiagnosticTelemetry;
 	private readonly workspaceOpenTelemetry: WorkspaceOpenTelemetry;
+	private readonly connectionLogBuffer: ConnectionLogBuffer;
 
 	// These will only be populated when actively connected to a workspace and are
 	// used in commands.  Because commands can be executed by the user, it is not
@@ -193,6 +195,7 @@ export class Commands {
 			this.telemetryService,
 		);
 		this.logger = serviceContainer.getLogger();
+		this.connectionLogBuffer = serviceContainer.getConnectionLogBuffer();
 		this.pathResolver = serviceContainer.getPathResolver();
 		this.mementoManager = serviceContainer.getMementoManager();
 		this.secretsManager = serviceContainer.getSecretsManager();
@@ -486,6 +489,11 @@ export class Commands {
 					: [];
 
 				progress.report({ message: "Collecting diagnostics..." });
+				// Flush buffered below-level connection logs before the CLI runs so
+				// the channel has time to write them to disk; retain the ring so a
+				// later failure flush still replays them. Best-effort: the channel
+				// writes on its own schedule, so the tail may not land in this bundle.
+				this.connectionLogBuffer.flush("support_bundle", { retain: true });
 				await cliExec.supportBundle(env, workspaceId, {
 					outputPath: outputUri.fsPath,
 					agentName,

src/core/container.ts

@@ -1,10 +1,19 @@
 import * as vscode from "vscode";
 
+import { watchConfigurationChanges } from "../configWatcher";
 import { AuthTelemetry } from "../instrumentation/auth";
+import {
+	BufferingLogger,
+	type ConnectionLogBuffer,
+} from "../logging/logBuffer";
 import { prefixLogger } from "../logging/prefixLogger";
 import { shortId } from "../logging/utils";
 import { LoginCoordinator } from "../login/loginCoordinator";
 import { OAuthCallback } from "../oauth/oauthCallback";
+import {
+	CONNECTION_LOG_BUFFER_SIZE_SETTING,
+	readConnectionLogBufferSize,
+} from "../settings/logger";
 import { buildSession, extractExtensionVersion } from "../telemetry/event";
 import { TelemetryService } from "../telemetry/service";
 import { LocalJsonlSink } from "../telemetry/sinks/localJsonlSink";
@@ -29,7 +38,8 @@ import type { Logger } from "../logging/logger";
  */
 export class ServiceContainer implements vscode.Disposable {
 	private readonly outputChannel: vscode.LogOutputChannel;
-	private readonly logger: Logger;
+	private readonly logger: BufferingLogger;
+	private readonly connectionLogBufferConfigSubscription: vscode.Disposable;
 	private readonly pathResolver: PathResolver;
 	private readonly mementoManager: MementoManager;
 	private readonly secretsManager: SecretsManager;
@@ -48,9 +58,21 @@ export class ServiceContainer implements vscode.Disposable {
 		this.outputChannel = vscode.window.createOutputChannel("Coder", {
 			log: true,
 		});
-		this.logger = prefixLogger(
+		const readSize = () =>
+			readConnectionLogBufferSize(vscode.workspace.getConfiguration());
+		this.logger = new BufferingLogger(
+			prefixLogger(this.outputChannel, `[session ${shortId(sessionId)}]`),
 			this.outputChannel,
-			`[session ${shortId(sessionId)}]`,
+			readSize(),
+		);
+		this.connectionLogBufferConfigSubscription = watchConfigurationChanges(
+			[
+				{
+					setting: CONNECTION_LOG_BUFFER_SIZE_SETTING,
+					getValue: readSize,
+				},
+			],
+			() => this.logger.setCapacity(readSize()),
 		);
 		this.pathResolver = new PathResolver(
 			context.globalStorageUri.fsPath,
@@ -148,6 +170,11 @@ export class ServiceContainer implements vscode.Disposable {
 		return this.logger;
 	}
 
+	/** The connection log buffer that replays below-level entries on failure. */
+	getConnectionLogBuffer(): ConnectionLogBuffer {
+		return this.logger;
+	}
+
 	getCliManager(): CliManager {
 		return this.cliManager;
 	}
@@ -193,6 +220,7 @@ export class ServiceContainer implements vscode.Disposable {
 		this.commandManager.dispose();
 		this.contextManager.dispose();
 		this.loginCoordinator.dispose();
+		this.connectionLogBufferConfigSubscription.dispose();
 		try {
 			await this.telemetryService.dispose();
 		} finally {

src/extension.ts

@@ -141,6 +141,7 @@ async function doActivate(
 		deploymentSessionAuth?.token,
 		output,
 		telemetryService,
+		serviceContainer.getConnectionLogBuffer().onConnectionFailure,
 	);
 	ctx.subscriptions.push(client);
 

src/instrumentation/EVENTS.md

@@ -428,7 +428,7 @@ Emitted by `WebSocketTelemetry`.
 These events share one value set, **ConnectionStateReason**: `initial_connect`,
 `manual_reconnect`, `certificate_refresh`, `scheduled_reconnect`, `open`,
 `disconnect`, `dispose`, `unrecoverable_close`, `unrecoverable_http`,
-`certificate_error`, `connection_error`, `normal_close`, `unexpected_close`.
+`certificate_error`, `connection_error`, `unexpected_close`.
 
 ### Logs
 
@@ -451,11 +451,11 @@ These events share one value set, **ConnectionStateReason**: `initial_connect`,
 Emitted as an error log (with the `error` block) when a socket error caused
 the drop.
 
-| Attribute                              | Values                                                                                                          |
-| -------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
-| `cause`                                | `manual_disconnect`, `replaced`, `unrecoverable_close`, `normal_close`, `unexpected_close`, `disposed`, `error` |
-| `close_code`                           | WebSocket close code (when known)                                                                               |
-| `connection_duration_ms` (measurement) | time the connection was open                                                                                    |
+| Attribute                              | Values                                                                                          |
+| -------------------------------------- | ----------------------------------------------------------------------------------------------- |
+| `cause`                                | `manual_disconnect`, `replaced`, `unrecoverable_close`, `unexpected_close`, `disposed`, `error` |
+| `close_code`                           | WebSocket close code (when known)                                                               |
+| `connection_duration_ms` (measurement) | time the connection was open                                                                    |
 
 #### `connection.reconnect_resolved`
 

src/instrumentation/websocket.ts

@@ -16,14 +16,12 @@ export type ConnectionStateReason =
 	| "unrecoverable_http"
 	| "certificate_error"
 	| "connection_error"
-	| "normal_close"
 	| "unexpected_close";
 
 export type ConnectionDropCause =
 	| "manual_disconnect"
 	| "replaced"
 	| "unrecoverable_close"
-	| "normal_close"
 	| "unexpected_close"
 	| "disposed"
 	| "error";

src/logging/formatters.ts

@@ -24,6 +24,7 @@ const SENSITIVE_BODY_FIELDS: ReadonlySet<Lowercase<string>> = new Set([
 	"id_token",
 	"password",
 	"refresh_token",
+	"registration_access_token",
 	"token",
 ]);