Skip to content

observability.md

Latest commit

 

History

History
285 lines (240 loc) · 17.2 KB

observability.md

File metadata and controls

285 lines (240 loc) · 17.2 KB
Raw

The morph::observe observability seam — design

morph::observe is a lightweight, injectable metrics/trace seam modelled on morph::log's replaceable-sink pattern (logger.md). A host installs a MetricSink and/or a TraceSink once at startup; the framework's hot paths — RemoteServer dispatch/register/deregister, LocalBackend dispatch/register/deregister, SyncWorker::run(), and ReconnectCoordinator::onOnline() — feed them. With no sink installed, every call site rejects behind a single relaxed atomic load and constructs nothing: an unconfigured build has zero observability overhead and behaves exactly as if the seam did not exist. RemoteServer additionally exposes a health() readiness query and ServerConfig::healthHandler.

Contents

Metrics

Metric is a closed enum class of nine observation kinds:

Enumerator Kind Meaning
executeLatencyMs value One dispatch's wall time in milliseconds (model strand).
executeInFlight gauge Concurrent in-flight executes.
executeErrors counter Executes that resolved via an error path.
registerCount counter register calls (attempted, not just successful).
deregisterCount counter deregister calls.
queueDepth gauge IOfflineQueue pending items at the start of a SyncWorker::run().
reconnectAttempts counter ReconnectCoordinator::onOnline's tryReconnect attempts.
reconnectOutcome counter One per onOnline() call, tagged by outcome.
queueOverflow counter An IOfflineQueue::enqueue() call rejected because the queue was at its configured maxDepth() (see docs/spec/offline/offline.md).

Each observation is a MetricEvent{metric, value, tags}; tags is a std::span<const std::pair<std::string_view, std::string_view>> carrying dimensions (modelType, actionType, outcome, …) without baking them into the enum. setMetricSink(MetricSink) installs the process-wide sink (a std::function<void(const MetricEvent&)>, thread-safe mutex-guarded swap, exactly like morph::log::setLogger); metricsEnabled() is the lock-free [[nodiscard]] bool hot-path check, backed by a relaxed std::atomic<bool> set whenever a non-null sink is installed. setMetricSink(nullptr) disables metrics again.

Tracing

Tracing reuses Context::requestId (session.md) as the correlation id. TraceSink has two members:

struct TraceSink {
    std::function<SpanId(std::string_view requestId, std::string_view modelType,
                         std::string_view actionType)> beginSpan;
    std::function<void(SpanId, bool ok)> endSpan;
};

SpanId is a std::uint64_t; 0 is a reserved "no span" sentinel. Both beginSpan and endSpan must be set for setTraceSink to enable tracing — a sink with only one of the two behaves as if none were installed, and the internal beginSpan helper keeps returning 0. Call sites unconditionally call beginSpan(...) at the start of a dispatch and endSpan(id, ok) at the end; when tracing is disabled (or id == 0), both are single cheap checks that touch neither a mutex nor the trace sink.

Health / readiness

RemoteServer (backend.md) exposes:

struct HealthStatus {
    bool ready;
    std::size_t liveModels;
    std::size_t inFlight;
};

[[nodiscard]] morph::async::Completion<HealthStatus> health(morph::exec::IExecutor& replyExec);
// ServerConfig::healthHandler — std::function<void(const HealthStatus&)>

health(replyExec) is answered on the server strand, where the model registry and the in-flight count live, and delivered on replyExec, the executor the caller attaches from: liveModels from the registry, inFlight from the in-flight count — the same count LimitPolicy::maxInFlightExecutes enforces, the executeInFlight metric reports, and RemoteServer::drainedWithin() waits on (see Thread safety and backend.md): one count, four consumers, never double-counted. ServerConfig::healthHandler, given at construction, is called once from the constructor with the initial snapshot (so a subscriber never has to wait for a transition to see a baseline) and again, on the server strand, on any subsequent readiness change. ready starts true and is flipped to false, once and for good, by RemoteServer::beginShutdown() — there is no un-shutdown, so once a server has begun shutting down ready stays false for the rest of its lifetime. morph does not embed an HTTP health endpoint; a deployment's transport (e.g. QtWebSocketServer) is expected to expose health() over whatever probe protocol it serves.

Call sites

Site Metrics emitted Trace hook
RemoteServer::dispatchMessage's register branch registerCount —
RemoteServer::dispatchMessage's deregister branch deregisterCount —
RemoteServer::dispatchExecute's in-flight admit/complete executeInFlight (on admit and again when the reply is delivered) —
RemoteServer::dispatchExecute's strand task executeLatencyMs, executeErrors beginSpan/endSpan around ActionDispatcher::dispatch
LocalBackend::registerModel registerCount —
LocalBackend::deregisterModel deregisterCount —
LocalBackend::execute's strand task executeLatencyMs, executeInFlight, executeErrors beginSpan/endSpan around localOp
SyncWorker::run() queueDepth (once, at drain) —
ReconnectCoordinator::onOnline() reconnectAttempts (per attempt), reconnectOutcome (once, tagged outcome) —
InMemoryOfflineQueue::enqueue, FileOfflineQueue::enqueue, SqliteOfflineQueue::enqueue queueOverflow (once per rejected call, at the queue's configured maxDepth()) —

registerCount/deregisterCount count every call, not just successful ones — an unauthorized or malformed register still increments it, so the counter reflects load on that path rather than only outcomes. executeLatencyMs/ executeErrors are scoped to the strand-posted dispatch itself; an execute that fails before dispatch even starts (unknown model id, failed authorization, LimitPolicy rejecting with "server busy"/"too many models") does not touch the latency/error metrics or the trace hooks, mirroring the fact that no strand task ever ran. On RemoteServer, a LimitPolicy::executeTimeout firing before the strand task finishes still lets that task run to completion later (the model action itself is never interrupted, per backend.md's LimitPolicy docs); when it does, its executeLatencyMs/executeErrors/trace-span instrumentation still fires, even though the client may have already received the timeout reply — the in-flight gauge's paired decrement, by contrast, is centralized so it fires exactly once regardless of which path (timeout or dispatch) resolves the call first.

Thread safety

Metrics and tracing each have their own mutex (metricMtx, traceMtx) guarding installation and retrieval of their sink, plus their own lock-free std::atomic<bool> enabled flag (metricsOn, traceOn), independent of each other and of morph::log's state — installing a metric sink never contends with logging or tracing.

A sink is never invoked while the mutex is held, and never allowed to throw. Each sink is stored behind a std::shared_ptr, so an emitter takes a reference-count copy under the lock, releases it, and only then calls the sink, inside a catch (...). Both properties are load-bearing, because a sink is host code called from the framework's hot paths:

  • Outside the lock. std::mutex is not recursive, so a sink that emits a metric of its own — or reinstalls itself via setMetricSink — would deadlock against the very mutex its caller holds.
  • Inside catch (...). A throwing sink must not escape into the framework. LocalBackend::execute brackets each dispatch with beginSpan / emitMetric / endSpan and settles the caller's Completion after them, precisely so a completion callback cannot observe the dispatch as finished before its metrics land. An exception escaping instrumentation would therefore skip setValue/setException entirely and be swallowed by the strand's catch-and-log (LoggedTask), leaving that Completion unsettled forever — a hung caller with neither a value nor an error, caused by a bug in a metrics callback.

A sink that throws is otherwise ignored (a failed beginSpan degrades to the 0 sentinel every call site already handles): observability may not change program behavior, and reporting the failure through morph::log would invite the same re-entrancy this tolerates. RemoteServer's in-flight count (the one LimitPolicy::maxInFlightExecutes gates, see backend.md) is server-strand state: incremented at admission and decremented by a task each reply posts back, so every executeInFlight sample is emitted on the server strand and none is torn or out of order with another. This seam reuses that counter for the executeInFlight metric and HealthStatus::inFlight rather than introducing a second, separately-maintained counter for the same concept. LocalBackend's in-flight counter is a std::shared_ptr<std::atomic<std::size_t>> rather than a plain member: its strand-posted tasks capture a copy of the shared_ptr, never this, so the counter stays valid even if the backend is destroyed while a task is still queued or running (see backend.md's Lifetime & ownership).

API reference

Metric

enum class Metric : std::uint8_t — see Metrics for the nine enumerators.

MetricEvent

Member Type Notes
metric Metric Which metric this observation is for.
value double The observed value.
tags std::span<const std::pair<std::string_view, std::string_view>> Dimensions; empty for untagged metrics.

Metric configuration

Symbol Signature Notes
MetricSink using MetricSink = std::function<void(const MetricEvent&)> Sink type.
setMetricSink void setMetricSink(MetricSink) Thread-safe (mutex-guarded swap). nullptr disables metrics.
metricsEnabled [[nodiscard]] bool metricsEnabled() noexcept Lock-free hot-path check (relaxed atomic load).

SpanId / TraceSink

Symbol Signature Notes
SpanId using SpanId = std::uint64_t 0 is the "no span" sentinel.
TraceSink::beginSpan std::function<SpanId(std::string_view requestId, std::string_view modelType, std::string_view actionType)> Called at dispatch start (on the strand).
TraceSink::endSpan std::function<void(SpanId, bool ok)> Called at dispatch end.
setTraceSink void setTraceSink(TraceSink) Both callbacks must be set to enable tracing. Thread-safe.

ScopedObserveOverride

RAII guard, snapshot-only constructor, restores the previous metric and trace sinks on destruction. Mirrors morph::log::ScopedLoggerOverride's snapshot-only constructor; used by tests to avoid leaking a sink across test cases. Not copyable or movable.

RemoteServer::HealthStatus / health() / ServerConfig::healthHandler

See backend.md's RemoteServer API reference table.

Internal detail (not for direct use)

Symbol Signature Notes
detail::ObserveState struct Process-wide singleton holding both sinks, their mutexes, and their enabled atomics.
detail::observeState ObserveState& observeState() Meyers singleton accessor.
detail::emitMetric void emitMetric(Metric, double, std::span<const std::pair<std::string_view, std::string_view>> tags = {}) Framework-internal emit entry point; no-op (one atomic load) if no sink installed.
detail::beginSpan [[nodiscard]] SpanId beginSpan(std::string_view, std::string_view, std::string_view) Framework-internal; returns 0 if tracing disabled.
detail::endSpan void endSpan(SpanId, bool) Framework-internal; no-op if id == 0 or tracing disabled.
offline::detail::reconnectOutcomeName constexpr std::string_view reconnectOutcomeName(ReconnectOutcome) noexcept Maps a ReconnectOutcome to the reconnectOutcome metric's "outcome" tag value.

Design decisions

Decision Choice Why
API surface Mirrors morph::log exactly: public config functions + internal detail:: emit helpers Consistency with the framework's one existing observability seam; engineers who know morph::log already know this API's shape.
Two independent sinks (metric, trace), each with its own mutex/atomic No shared state between them Installing/using one never contends with the other; matches the fact that they observe different things (numbers vs. spans) and can be wired to entirely different backends.
registerCount/deregisterCount count every call, not just successes Counts attempts Gives an accurate load/rate signal on that path (an unauthorized or malformed register still costs the server work); executeErrors is the separate counter for outcome-scoped failure signal on the execute path.
RemoteServer's in-flight metric reuses the in-flight count No second counter added LimitPolicy::maxInFlightExecutes already needs an in-flight count with exactly this admit/complete lifecycle; a parallel one for the metric alone would have to be kept in lockstep for no benefit. health()'s inFlight field reads the same count.
LocalBackend's in-flight counter is a shared_ptr, not a plain atomic member Avoids capturing this in a strand-posted lambda LocalBackend's existing strand tasks already avoid this captures for lifetime safety (see Limitations); the counter follows the same rule rather than becoming a new dangling-pointer risk. RemoteServer's equivalent count is a plain member of its server-strand state, reached through tasks that capture self = shared_from_this(), keeping the whole object alive.
ServerConfig::healthHandler is called from the constructor Once, synchronously, with the initial status A subscriber gets a baseline status without waiting for the first transition; nothing else can reach the server yet, so the call needs no strand.
ready's sole mutator is beginShutdown() RemoteServer sets _ready to false only from beginShutdown(), never back to true This seam landed ahead of beginShutdown() specifically so that the later change (backend.md) needed no API change — just a store and a handler re-invocation inside the method already documented as the trigger.

Limitations

  • No sampling/aggregation/histograms in morph. The framework emits raw observations (a value or a count of 1); rate-limiting, histogram bucketing, and sampling are the host sink's job, exactly as morph::log ships no log rotation or filtering beyond level.
  • No metrics/tracing/HTTP dependency shipped. The seam is a std::function pair the host wires to Prometheus, StatsD, OpenTelemetry, or a test spy; there is no bundled backend.
  • ready is always true. See Health / readiness and the corresponding design decision — no code path in this release flips it.
  • No preemption or control. Metrics/health observe; they never throttle or cancel work. Bounding behavior belongs to backend.md's LimitPolicy, which this seam only makes visible.
  • Does not change dispatch semantics or the wire. The hooks wrap existing call sites; there are no new envelope fields and no behavior change when unconfigured.
  • executeLatencyMs/executeErrors (and, on LocalBackend, executeInFlight) are scoped to the strand-posted dispatch. A fast-fail before the strand task is posted (unknown model id, failed authorization, a LimitPolicy rejection) emits none of them — see Call sites.

Cross-references

  • logger.md — the replaceable-sink pattern (global std::function sink, mutex-guarded swap, lock-free hot-path check) this seam copies, and the sink-signature limitation ((LogLevel, string_view) carries no structured numbers or spans) that makes a separate seam necessary.
  • backend.md — RemoteServer/LocalBackend dispatch call sites the metric/trace hooks wrap, the in-flight count this seam reuses, and where HealthStatus/health()/ServerConfig::healthHandler live.
  • offline.md — SyncWorker/ReconnectCoordinator and the IOfflineQueue depth the queueDepth/reconnectAttempts/ reconnectOutcome metrics report on.
  • session.md — Context::requestId, reused as the trace correlation id.
  • profiler.md — compile-time Tracy instrumentation behind MORPH_ENABLE_TRACY. A build option rather than a sink: it is not wired through this seam, and it links the phases of one call by the same requestId.