Observability

Silgi has its own logger, request id, request log, tracing, metrics, Server-Timing header, error context and client reports. All of it is written with Web APIs and has no dependency. Nothing that costs measurable throughput is on by default in a build: a feature that is off adds no work to a request.

What is on by default

Tests check the "cost when off" column: with the defaults of a build, a request calls neither crypto.randomUUID nor crypto.getRandomValues, adds no header, starts no AsyncLocalStorage run and writes no log line. What each feature costs when it is on is in What it costs.

Logging

Server code logs through event.log, the server logger with the request's id bound to every record. useLogger() without an event is the server logger without a request. There is no lookup of "the current request": pass the event, or use event.log. A loader has it as ctx.event.log.

import { defineHandler, useLogger } from "silgi/server";

export const placeOrder = defineHandler({
  route: "/api/orders",
  method: "POST",
  raw: true,
  async handler(_, { event }) {
    event.log.info("Order received", { items: 3, token: "not-written" });

    const audit = useLogger().child({ name: "audit" });
    audit.warn("Large order", { items: 3 });

    try {
      await Promise.reject(new Error("Payment provider timed out"));
    } catch (error) {
      event.log.error(error, "Payment failed", { provider: "acme" });
    }

    return { ok: true };
  },
});

A logger method takes a message and fields, or an error first: event.log.error(error, "Payment failed", { provider: "acme" }) writes the error as err. child(bindings) gives a logger that writes its bindings on every record, and enabled(level) tells whether a call would write.

The levels are trace, debug, info, warn, error and fatal, and silent turns the logger off. A call below the level returns after one compare and builds nothing. Set the level and the format in the config, or at run time with SILGI_LOG_LEVEL and SILGI_LOG_FORMAT (json, pretty or false), which win over the config:

import { defineConfig } from "silgi";

export default defineConfig({
  framework: "html",
  server: {
    logLevel: "info",
    logFormat: "json",
    logRedact: ["ssn", "card_number"],
    requestIdHeader: "x-request-id",
    trustProxy: true,
  },
});

A build writes JSON lines, one per record, with the keys in this order: time (ISO 8601 UTC), level, msg, name, reqId, traceId, spanId, your bindings and fields, and err. A binding or field named time, level or msg is written as _time, _level or _msg, so app data cannot forge them. silgi dev writes the pretty format: [silgi] <msg> followed by key=value per field and the error's stack. logFormat: false installs no default sink, which suits an app that sends its records somewhere else with addLogSink. A value in the environment that is not a level or a format logs one warning and is ignored.

Redaction. Before a record is formatted, the value of every binding, field and err.data key is replaced by "[redacted]", at any depth, when the key's name (lower case, with - and _ removed) is one of authorization, proxyauthorization, cookie, setcookie, xapikey, password, passwd, secret, token, apikey, accesstoken, refreshtoken, clientsecret, privatekey, sessionid and csrf, or ends with password, secret, token or apikey. server.logRedact adds names; the defaults cannot be removed. Redaction goes by key only: a secret inside the message is not found, so pass values as fields. Request headers are never logged unless your code passes them.

A sink that is not the default one is added by a server plugin with server.addLogSink(sink) (see SilgiServer). It gets every record after redaction.

Request id

getRequestId(event) returns the id of the request. It is made on first read (by getRequestId, event.log, the request log, a span or the error context) and then fixed. When nothing reads it, no id exists and no header is sent.

import { defineHandler, getRequestId } from "silgi/server";

export const requestId = defineHandler({
  route: "/api/request-id",
  method: "GET",
  raw: true,
  handler(_, { event }) {
    return { requestId: getRequestId(event) };
  },
});

The id is the value of the server.requestIdHeader request header (x-request-id by default) when server.trustProxy is on and the value is 1 to 200 characters of A-Z a-z 0-9 . _ : @ / + = -. On cloudflare, vercel and netlify the platform's own header is used (cf-ray, x-vercel-id, x-nf-request-id). Otherwise it is crypto.randomUUID(). Once an id exists, the answer carries it in the server.requestIdHeader header, unless the route set that header. A page that calls an API route through $fetch in process shares its id with that route.

Request log

server.requestLog writes one record per request when fetch returns the answer: GET /about 200 3.1ms, with the fields method, path, status, duration and route. It is at error level for a 5xx, warn for a 4xx and info otherwise. It is on in silgi dev and off in a build. SILGI_REQUEST_LOG (1 or true, 0 or false) overrides the config at run time.

import { defineConfig } from "silgi";

export default defineConfig({
  framework: "html",
  server: {
    requestLog: { query: false, assets: false, skip: ["/healthz", "/api/ping"] },
  },
});

duration is the time to the answer's headers, so the later bytes of a streamed body are not counted. The query is left out of path unless query: true; then the values of query names that match the redaction rule are [redacted]. Not logged: files from public/ and the build's assets (unless assets: true), paths that match a skip pattern, the Prometheus endpoint, the client report endpoint and the dev endpoints.

Tracing

server.tracing records a span for each sampled request and for the work it causes. SILGI_TRACING=1 turns it on at run time and 0 off; the other keys come from the config.

Silgi makes a span for the request and for server middleware, route middleware, the route handler, loaders, the render, each $fetch, each storage operation and each cached function lookup. The names and attributes follow the OpenTelemetry HTTP conventions where one exists; the query string is never an attribute. A request that is not sampled makes no span at all. The decision is made once, at the request span: an incoming traceparent with the sampled flag is recorded, one without it is not, and a new trace is recorded with the chance sample.

A sampled page carries <meta name="traceparent"> so the client reports join its trace. A prerendered page and a page that a route rule caches do not.

bodyTiming stays off by default because ending the span at the end of a streamed body means piping that body, and that costs throughput on the bun preset (see What it costs).

Send spans to a collector

Silgi sends spans itself, as OTLP over HTTP with the JSON encoding. There is no gRPC and no protobuf. The exporter is on when tracing is on and an endpoint is known: tracing.otlp.endpoint as written, else OTEL_EXPORTER_OTLP_TRACES_ENDPOINT as written, else OTEL_EXPORTER_OTLP_ENDPOINT with /v1/traces added. tracing.otlp: false turns it off.

This runs an OpenTelemetry Collector that prints what it receives, and an app that sends to it:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

docker run --rm --name otelcol -p 4318:4318 \
  -v "$PWD/otel-collector.yaml:/etc/otelcol/config.yaml:ro" \
  otel/opentelemetry-collector:0.115.1 --config /etc/otelcol/config.yaml

# in another terminal, from the app
silgi build
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 SILGI_TRACING=1 bun .output/server/index.mjs

Open the app, then stop it (or wait five seconds): the collector's log shows the spans, with service.name and the trace id. The recipe was run once against otel/opentelemetry-collector:0.115.1, image digest sha256:f6842984eafb35be5198d080e40ca3f6b3b45c6c2456318a41ce7f2fa71ba973, with Silgi at commit 852d8f1.

Other settings follow the OpenTelemetry environment variables: OTEL_EXPORTER_OTLP_TRACES_HEADERS or OTEL_EXPORTER_OTLP_HEADERS for headers (a=1,b=x%20y), OTEL_EXPORTER_OTLP_TIMEOUT (10000 ms by default), OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES. The config keys win over them. A backend that needs a token takes it in otlp.headers:

import { defineConfig } from "silgi";

export default defineConfig({
  framework: "html",
  server: {
    tracing: {
      sample: 0.1,
      serviceName: "shop",
      otlp: { endpoint: "http://localhost:4318/v1/traces", headers: { authorization: "Bearer token" } },
    },
  },
});

Ended spans wait in a queue of 2048 and go out in batches of at most 512 every 5 seconds, when the queue holds 512, and when the server drains. A network error, 429, 502, 503 or 504 is retried once; after that, or on any other status, the batch is dropped, and the request never sees it. The warnings tracing: span export failed: … and tracing: dropped <n> spans (queue full) appear at most once a minute. The error message never holds the endpoint, since it may carry a token.

To send spans somewhere that does not speak OTLP, add your own exporter in a server plugin. It gets the ended spans and the resource:

import { definePlugin } from "silgi/server";

export default definePlugin((server) => {
  server.addSpanExporter({
    async export(spans, resource) {
      for (const span of spans) console.log(resource["service.name"], span.name, span.endTime - span.startTime);
    },
  });
});

On Workers

The cloudflare preset reads the OTEL_* names and SILGI_TRACING from the Worker's variables at the first request, so set them under vars (or as secrets, for a token) in wrangler.jsonc. Spans go out through event.waitUntil after the answer has left, so the request is never held back and the isolate stays alive until the export is done. The preset already writes the nodejs_compat flag, which the async context needs; without an async context, storage and cache spans are left out and the others are made. The same holds for vercel and netlify.

Your own spans

withSpan(name, fn, { event, kind, attributes }) runs fn in a child span of the request and rethrows what it throws, after recording it as an exception event with status error. With tracing off, or on a request that is not sampled, fn gets a frozen no-op span and nothing is made.

import { defineHandler, withSpan } from "silgi/server";

async function chargeCard(amount: number): Promise<string> {
  return `charge-${amount}`;
}

export const charge = defineHandler({
  route: "/api/charge",
  method: "POST",
  raw: true,
  handler(_, { event }) {
    return withSpan(
      "charge card",
      async (span) => {
        span.setAttribute("shop.amount", 42);
        const id = await chargeCard(42);
        span.addEvent("charged", { id });
        return { id };
      },
      { event, attributes: { "shop.provider": "acme" } },
    );
  },
});

Metrics

server.metrics counts every request in the histogram http.server.request.duration (seconds, by http.request.method, http.route and http.response.status_code) and keeps the gauge http.server.active_requests. Metrics live in one process, so they work on the bun and node presets and in silgi dev. On cloudflare, vercel, netlify and static the build warns that server.metrics is ignored, since each request may run in another instance, and the bundle has no metrics code; use the platform's analytics there.

useMetrics() gives your own counters, histograms and gauges. Keep attribute values to a small set: past 2000 sets per instrument, new ones are folded into one point.

import { defineHandler, useMetrics } from "silgi/server";

const orders = useMetrics().counter("shop.orders", { description: "Orders placed." });
const basket = useMetrics().histogram("shop.basket.items", { description: "Items per order.", buckets: [1, 2, 5, 10, 20] });

export const placeOrder = defineHandler({
  route: "/api/orders",
  method: "POST",
  raw: true,
  handler() {
    orders.add(1, { plan: "pro" });
    basket.record(3, { plan: "pro" });
    return { ok: true };
  },
});

Prometheus

With prometheus on, the metrics are served at /_silgi/metrics (after app.baseURL; path changes it) in the Prometheus text format.

import { defineConfig } from "silgi";

export default defineConfig({
  framework: "html",
  server: {
    metrics: { interval: 30000, prometheus: { path: "/_silgi/metrics" } },
  },
});

A scrape needs authorization: Bearer <token>. The token is prometheus.token, else the environment variable SILGI_METRICS_TOKEN; a missing or wrong token gets 401. With no token anywhere the endpoint answers 404 and the server logs one warning at start. token: false makes the endpoint public, and is for a path only your network can reach. Put the token in a file and point Prometheus at it:

scrape_configs:
  - job_name: shop
    metrics_path: /_silgi/metrics
    scrape_interval: 15s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/silgi-token
    static_configs:
      - targets: ["shop.internal:3000"]

A scrape is not counted, logged or traced. To push metrics somewhere else, add an exporter with server.addMetricExporter(exporter) in a server plugin: it gets a snapshot every server.metrics.interval ms (60000 by default) and once at drain.

Server-Timing

server.serverTiming adds a Server-Timing header that the browser's devtools show: mw (server middleware), handler, loader, render, cache;desc="hit" or "stale" when a cache answered, and total, in milliseconds. It is on in silgi dev and off in a build, because it tells anyone who asks how long each phase took. SILGI_SERVER_TIMING=1 turns it on at run time.

setServerTiming(event, name, duration, description?) adds an entry of your own, and does nothing while the feature is off:

import { defineHandler, setServerTiming } from "silgi/server";

async function queryUsers(): Promise<string[]> {
  return ["ada"];
}

export const listUsers = defineHandler({
  route: "/api/users",
  method: "GET",
  raw: true,
  async handler(_, { event }) {
    const start = performance.now();
    const users = await queryUsers();
    setServerTiming(event, "db", performance.now() - start, "users");
    return { users };
  },
});

Reporting errors

The server error hook gets the raw thrown value, the event and an ErrorContext: the phase (request, middleware, route, loader, function, render and others), the route, the request id, the trace and span ids when the request is traced, what the client got (masked), and whether an answer carries the error (handled). It also gets the errors that no answer carries, such as a rejected event.waitUntil promise or a failed background refresh of a cache entry. Each error is logged once, before the hooks run.

A Sentry-style reporter is a server plugin on this hook. It reports the 5xx errors and every error that no answer carries, and sends the request id and trace id so the report links to the log lines and the trace. reporter.ts stands for your vendor's SDK:

export interface ReportContext {
  requestId?: string;
  traceId?: string;
  phase?: string;
  route?: string;
  url?: string;
  source?: string;
}

export function captureException(error: unknown, context: ReportContext): void {
  const body = JSON.stringify({ message: error instanceof Error ? error.message : String(error), context });
  void fetch("https://reporter.example/api/events", { method: "POST", body, keepalive: true }).catch(() => {});
}

import { definePlugin } from "silgi/server";
import { captureException } from "./reporter.ts";

export default definePlugin((server) => {
  server.hooks.hook("error", (error, event, context) => {
    if (context.masked.status < 500 && context.handled) return;
    captureException(error, {
      requestId: context.requestId,
      traceId: context.traceId,
      phase: context.phase,
      route: context.route,
      url: event.url.pathname,
    });
  });
});

The browser side is an app plugin on the app:error hook. Its info.source says where the error came from: render, navigation, plugin, hook, chunk, window or rejection. Silgi listens for window error and unhandledrejection events once per page and calls the hook for them; it neither prevents the default nor shows the error page.

import { definePlugin } from "silgi/app";
import { captureException } from "./reporter.ts";

export default definePlugin((app) => {
  app.hooks.hook("app:error", (error, info) => {
    if (error.status < 500) return;
    captureException(error, { source: info.source });
  });
});

Client reports

If you would rather not add a vendor script, server.clientReports makes the browser send its errors, web vitals and client navigations to your own server. It is off by default; true is { errors: true, vitals: true, navigations: true }.

import { defineConfig } from "silgi";

export default defineConfig({
  framework: "html",
  server: {
    clientReports: { errors: true, vitals: 0.1, maxPerPage: 10 },
  },
});

vitals as a number is the share of page loads that report them. A report goes to the client:report server hook and to the logger with name: "client": an error at error level, a vital at info as LCP 1834ms good /about. With metrics on, a vital is also recorded in the histogram silgi.web_vitals, and with tracing on an error report that carries the page's traceparent becomes a client error span in that trace.

import { definePlugin } from "silgi/server";

export default definePlugin((server) => {
  server.hooks.hook("client:report", (report, event) => {
    if (report.type === "vital" && report.rating === "poor") {
      event.log.warn("Poor web vital", { name: report.name, value: report.value, path: report.path });
    }
  });
});

What it costs:

• Bytes. Off, the client entry does not import the code and the server has no route there. On, the client adds about 1.3 KB gzipped (the build checks a budget of 1.5 KB). Even with reports off, the client carries about 300 to 350 B gzipped for the window listeners and the error source, since the app:error hook has them unconditionally.
• Requests. One beacon, navigator.sendBeacon (else fetch with keepalive), to /_silgi/reports: errors 1 second after the last one, vitals once when the page is first hidden, together with any error not yet sent. A batch is at most 20 reports and 16384 bytes. The same message and first stack line are sent once per page load, and at most maxPerPage (10) reports.
• Server. The endpoint takes POST only, and refuses a request whose Sec-Fetch-Site is not same-origin. It takes at most 100 reports per second per instance and drops the rest. It makes no log line, span or metric point of its own.

Vitals are measured by Silgi's own PerformanceObserver code, with web.dev's thresholds for good, needs-improvement and poor: LCP, INP, CLS, FCP and TTFB for the document load.

In silgi dev

silgi dev prints every server record in the pretty format through the dev runner, so the logs of the server and the runner come in one stream, in order, and the request log is on: [silgi] GET /about 200 3.1ms. silgi dev --debug sets the level to debug. The runner keeps the last 1000 records and streams them at /_silgi/logs (?reqId=<id> keeps one request's records, ?once=1 sends the buffered ones and closes). The error overlay shows the log lines of the request that failed, and the client overlay has a closed "Server log" section for the same.

What it costs

bun bench/run.ts observability measures the three features that run on every request, on the React app of docs/benchmarks.md: the defaults, server.requestLog: true, and server.tracing: { sample: 1, otlp: false } with an exporter that does nothing. The table is in the "Observability overhead" section there, with the load average of the run. In the run recorded there, the defaults were inside the range of the same app measured twice, on both presets, and server.requestLog: true was inside it too (its line goes to a standard output that the harness discards). Tracing at sample: 1 with an exporter that does nothing was about 5 to 6 % below the reference median on both presets, and that is smaller than the spread between rounds of that run (the reference row ranged from 3,317 to 4,130 req/s on bun), so repeat it on a quiet machine before you quote it. Tracing makes a span object and a queue entry for each span of each request, and with bodyTiming: true it also pipes streamed bodies, so on a busy server set sample below 1.