diff options
| author | Stefan Boberg <[email protected]> | 2026-05-05 15:47:48 +0200 |
|---|---|---|
| committer | GitHub Enterprise <[email protected]> | 2026-05-05 15:47:48 +0200 |
| commit | 01286c6233347d561064fc9e6cf9deaf2087ceb7 (patch) | |
| tree | bdbfdf01725baa2d2dd3d73727e6506b41421dff /src/zenhttp | |
| parent | hub async s3 client (#1024) (diff) | |
| download | archived-zen-main.tar.xz archived-zen-main.zip | |
Branch started as a sessions-service overhaul (persistence, client liveness, UE_LOGFMT intake) and grew to pick up adjacent infrastructure work: an early-startup log backlog, a hardened `MemoryArena`, the `zen trace serve` viewer gaining a counter view + compact timeline + tabbed callsite panel, defensive fixes in the third-party `tourist` trace parser, a series of allocation reductions across the HTTP and compact-binary hot paths, and a new `zen sessions` CLI command tree.
## Sessions service
**Persistence.** Each session lives on disk under `<DataRoot>/sessions/<id>/` as `info.cb` (metadata) plus `log.bin` (length-prefixed CbObject log records). On startup the service scans that directory and loads prior sessions as ended sessions, preloading the tail of each log so historical views work after a restart. `SessionLog` is noexcept-constructed and falls back to a disabled state on disk errors, so a bad disk can't take down `RegisterSession`. `GetSession` falls back to the ended-sessions list (fixes historical log fetches over HTTP). `LoadTail` counts only successfully-parsed records.
**Pruning.** Periodic cleanup task drops ended sessions once any of three caps is exceeded: age (default 1 year), count (default 1000), or total on-disk footprint (default 50 MiB). Runs 30 s after startup, hourly thereafter. Active sessions never pruned; disk removal and directory stat happen outside the exclusive lock so a slow filesystem can't stall lookups.
**Client liveness.** Sessions carry a `ProcessHandle` for the client-reported pid, captured at registration time so Windows pid recycling can't produce false positives. A 30 s asio timer probes liveness and ends dead sessions through the normal remove path, producing a synthetic `Session ended: process exited (...)` line persisted to `log.bin`. Windows decodes common NTSTATUS exit codes to human names (Ctrl-C, access violation, stack overflow, ...); POSIX stays at plain `process exited`. Clients auto-fill `ClientPid` only for local targets (unix socket / loopback); the server defensively accepts pids only from `IsLocalMachineRequest()` peers. zenserver also reports its own pid when registering its self-session, so it shows up with a real pid in the dashboard and `zen sessions ls`.
**Synthetic end-of-session line.** `RemoveSession` takes an optional reason; before the session moves to the ended list it appends an Info-level `Session ended[: reason]` entry through the normal log path (released outside `m_Lock`). Current reasons: `client request` (HTTP DELETE), `server shutdown` (self-session), `process exited (...)` (liveness).
**UE_LOGFMT structured entries.** `POST /sessions/{id}/log` now accepts `{level, logger, format, fields}` alongside the existing `{level, logger, message}` shape. New `logtemplate.{h,cpp}` implements UE's `StructuredLog.cpp` template grammar (field paths with `.name` / `[N]`, `{{`/`}}` escapes, `$text` / `$format` / `$locformat` object conventions, bounded recursion). Renders to a displayable message at intake while persisting raw format + fields so a future UI can drill into fields without another schema bump. Hot path is zero-alloc — renders into `ExtendableStringBuilder<256>` using stack-buffered `Oid::ToString` / `IoHash::ToHexString` overloads. UI shows a `{…}` marker with the raw template + JSON-pretty fields on hover.
**Parent sessions.** `SessionInfo` gains `parent_session_id`; hub-managed storage server child processes inherit the hub's session id via `--parent-session=<id>`. `ZEN_SESSIONS_URL` env var becomes a fallback for `--sessions-url` / config when neither is provided. The in-process session log sink is disabled when a remote sessions target is configured (logs flow through `SessionsServiceClient` instead). The sessions UI groups child sessions under their parent (collapsible/expandable, sorts as a unit, supports nesting).
**Platform reporting.** `SessionInfo` gains `Platform`, flowed end-to-end: client auto-fills via `GetRuntimePlatformName()`, server persists in `info.cb` (`plat`) and emits on GET. UI renders as a SimpleIcons-style inline SVG (windows / macOS / iOS / linux / wine / android / playstation / xbox / nintendo) with case-insensitive alias resolution (Win32/Win64, PS4/PS5, XSX/XSS, NintendoSwitch, iPhone/iPad, Darwin/OSX). Unknown values fall back to text; sorting runs on the underlying string.
**WebSocket log streaming.** Sessions UI moves from 2 s polling to a WebSocket push model. New `WsSubscriber` has a stable id + helper methods. UI caps the log-line DOM at 5 000 entries with a shared cursor-regression helper, factored out of two call sites. Per-broadcast allocations trimmed on the push path; fixed a stack overrun in the WS log broadcast hex-id buffer.
**Log memory.** `LogEntry::Level` is now `logging::LogLevel` (1 byte) instead of `std::string` (~32 B) — saves ~310 KB per full 10 k-entry deque and eliminates a per-message allocation in the in-proc sink. On-disk format writes an int32 and accepts either int or legacy string on read. `LogEntry` strings now live in a `MemoryArena`; logger names are interned across the deque. `SessionLog::Append` and `WriteSessionInfoFile` drop their `UniqueBuffer` round-trip and write `CbObject::GetView()` straight through `BasicFile` / `SafeWriteFile`. Multi-entry `POST /log` batched under one lock + one push.
**In-proc log timestamps.** `InProcSessionLogSink::TimePointToDateTime` previously preserved only whole seconds, so every in-proc entry rendered at `.000` ms in the dashboard and `zen sessions tail`. It now adds the sub-second part (nanoseconds → 100 ns ticks) to keep ms precision end-to-end.
**UI.** Side "Session Details" panel is gone — its info is inline in the table (appname, mode, platform, id, timestamps, this/log pills, active dot). Bottom panel is a tabbed `Log | Metadata` view with a right-side "Session Information" panel beside metadata; log-only controls (filter, newest-first, follow, log-level filter, expand/collapse) hide when Metadata is active, polling keeps running across tab switches. Wide-mode toggle fills the viewport edge-to-edge. Log lines show the logger category; timestamps render in 24 h with zero-padded fields regardless of locale. Sessions list defaults to All / 10 per page / created-desc, gains click-to-sort headers on the full dataset, a header filter box, and a pager aligned to the table's right edge. Duplicate auto-injected `<h1>Sessions</h1>` removed.
## `zen sessions` CLI
New command tree on the `zen` client for inspecting the sessions service from the terminal:
- **`zen sessions ls`** — lists sessions (active first, ended next; newest-first within each group) with id, status, app/mode, pid, created, duration, and log count. Supports `--status active|ended|all` (default `all`).
- **`zen sessions status`** — prints the sessions service summary: self id, active / ended counts, and the read/write/delete/list/request/bad-request counters from `/stats/sessions`.
- **`zen sessions tail [session]`** — tails a session's log. With no argument it tails zenserver's own session (resolved via `/sessions/list`'s `self_id`); an explicit 24-hex id targets any session, including ended ones (historical replay). `--lines N` (default 50, 0 = all buffered) trims the initial dump client-side. `--follow` prefers a WebSocket push subscription on `/sessions/ws` for sub-second latency; on upgrade failure (older server, blocked port, unix-socket transport) it falls back to HTTP cursor polling at `--interval-ms` (default 500), with sleeps chunked to 50 ms so Ctrl-C reacts quickly. Output matches `zen::logging::FullFormatter` (`[YY-MM-DD HH:MM:SS.mmm] [lvl] [logger] message`); on a TTY the level is colored and the logger is bold, with continuation lines indented under the message column using the *visible* prefix width. 404 surfaces as `(session ended)` and connection errors as `(server gone)` — both clean exits, so stopping the server mid-tail no longer prints a stack trace.
- **`zen sessions ui`** — opens `<host>/dashboard/?page=sessions` in the user's default browser. Rejects unix-socket hosts.
A small `ZenServiceClient::IsUnixSocket()` helper now wraps the unix-socket check used by `ui`, `sessions tail` (WS path), and `sessions ui`.
## Logging
`BacklogSink` captures early-startup log entries in a fixed-capacity ring so late-attached sinks (session sink, file sink) can replay them. Detaches from the broadcast list when disabled; backed by destructor-only cleanup (no `unique_ptr` indirection per entry). Tuned defaults so the backlog covers typical bring-up without unbounded growth.
## `zen trace serve` viewer
- Compact timeline mode for high-density views.
- New `TRACE_INT_VALUE` / `TRACE_FLOAT_VALUE` counter trace points + a counters page in the viewer.
- Callsite tables collapsed into a single tabbed panel.
- Lossless `Oid <-> Guid` bridge for trace session ids; trace `SessionId` plumbed through.
- `tourist` parser hardening: bounds-check `BufferStream::read`, validate `Type::info_size` before `patch()`, convert `parse_important_aux` to a loop (avoids deep recursion), widen `ParserPool` index to `uint32`, bounds-check field offsets in the dispatcher, pin `Types::parse` buffer up-front.
## `MemoryArena`
Configurable chunk size, inline chunk list, oversize requests routed to truly-dedicated chunks (no slack waste, no fragmentation when one allocation is much larger than the chunk).
## Allocation cleanups across hot paths
- `zenhttp::HttpRequestRouter::HandleRequest` and `FormatPackageMessageInternal`: drop heap allocations.
- Compact-binary validation: `eastl::fixed_vector` + `eastl::sort`; eliminate `std::vector` churn.
- `zenserverprocess`: trim transient allocations in spawn paths.
- Sessions HTTP intake / broadcast: drop transient `std::string` allocs.
Diffstat (limited to 'src/zenhttp')
| -rw-r--r-- | src/zenhttp/httpserver.cpp | 8 | ||||
| -rw-r--r-- | src/zenhttp/include/zenhttp/httpserver.h | 6 | ||||
| -rw-r--r-- | src/zenhttp/packageformat.cpp | 34 |
3 files changed, 24 insertions, 24 deletions
diff --git a/src/zenhttp/httpserver.cpp b/src/zenhttp/httpserver.cpp index 03117ee6c..31c94eb3b 100644 --- a/src/zenhttp/httpserver.cpp +++ b/src/zenhttp/httpserver.cpp @@ -939,9 +939,8 @@ HttpRequestRouter::HandleRequest(zen::HttpServerRequest& Request) const std::vector<int>& Matchers = Handler.ComponentIndices; bool IsMatch = true; - std::vector<std::string_view> CapturedSegments; - - CapturedSegments.emplace_back(Uri); + RouterRequest.m_CapturedSegments.clear(); + RouterRequest.m_CapturedSegments.emplace_back(Uri); for (size_t MatcherOffset = 0; MatcherOffset < Matchers.size(); MatcherOffset++) { @@ -991,7 +990,7 @@ HttpRequestRouter::HandleRequest(zen::HttpServerRequest& Request) if (m_MatcherFunctions[MatcherIndex](Segment)) { - CapturedSegments.push_back(Segment); + RouterRequest.m_CapturedSegments.push_back(Segment); } else { @@ -1019,7 +1018,6 @@ HttpRequestRouter::HandleRequest(zen::HttpServerRequest& Request) } #endif - RouterRequest.m_CapturedSegments = std::move(CapturedSegments); Handler.Handler(RouterRequest); return true; // Route matched diff --git a/src/zenhttp/include/zenhttp/httpserver.h b/src/zenhttp/include/zenhttp/httpserver.h index 955b8ed15..bccabf95b 100644 --- a/src/zenhttp/include/zenhttp/httpserver.h +++ b/src/zenhttp/include/zenhttp/httpserver.h @@ -17,6 +17,8 @@ #include <zentelemetry/hyperloglog.h> #include <zentelemetry/stats.h> +#include <EASTL/fixed_vector.h> + #include <filesystem> #include <functional> #include <gsl/gsl-lite.hpp> @@ -405,8 +407,8 @@ private: HttpRouterRequest(const HttpRouterRequest&) = delete; HttpRouterRequest& operator=(const HttpRouterRequest&) = delete; - HttpServerRequest& m_HttpRequest; - std::vector<std::string_view> m_CapturedSegments; + HttpServerRequest& m_HttpRequest; + eastl::fixed_vector<std::string_view, 8, /*bEnableOverflow=*/true> m_CapturedSegments; friend class HttpRequestRouter; }; diff --git a/src/zenhttp/packageformat.cpp b/src/zenhttp/packageformat.cpp index 267ce386c..4f001a14f 100644 --- a/src/zenhttp/packageformat.cpp +++ b/src/zenhttp/packageformat.cpp @@ -130,7 +130,7 @@ FormatPackageMessageBuffer(const CbPackage& Data, FormatFlags Flags, void* Targe static void MarshalLocal(CbAttachmentEntry*& AttachmentInfo, - const std::string& Path8, + std::string_view Path8, CbAttachmentReferenceHeader& LocalRef, const IoHash& AttachmentHash, bool IsCompressed, @@ -156,7 +156,7 @@ IsLocalRef(tsl::robin_map<void*, std::string>& FileNameMap, bool DenyPartialLocalReferences, void* TargetProcessHandle, CbAttachmentReferenceHeader& LocalRef, - std::string& Path8) + StringBuilderBase& Path8) { const SharedBuffer& Segment = AttachmentBinary.GetSegments().front(); IoBufferFileReference Ref; @@ -172,9 +172,11 @@ IsLocalRef(tsl::robin_map<void*, std::string>& FileNameMap, return false; } + Path8.Reset(); + if (auto It = FileNameMap.find(Ref.FileHandle); It != FileNameMap.end()) { - Path8 = It->second; + Path8.Append(It->second); } else { @@ -193,7 +195,7 @@ IsLocalRef(tsl::robin_map<void*, std::string>& FileNameMap, if (OK) { DuplicatedHandles.push_back((void*)TargetHandle); - Path8 = fmt::format("{}{}", HandlePrefix, reinterpret_cast<uint64_t>(TargetHandle)); + fmt::format_to(StringBuilderAppender(Path8), "{}{}", HandlePrefix, reinterpret_cast<uint64_t>(TargetHandle)); UseFilePath = false; } } @@ -205,21 +207,19 @@ IsLocalRef(tsl::robin_map<void*, std::string>& FileNameMap, #endif // ZEN_PLATFORM_WINDOWS if (UseFilePath) { - ExtendablePathBuilder<256> LocalRefFile; - std::error_code Ec; - std::filesystem::path FilePath = PathFromHandle(Ref.FileHandle, Ec); + std::error_code Ec; + std::filesystem::path FilePath = PathFromHandle(Ref.FileHandle, Ec); if (Ec) { ZEN_WARN("Failed to get path for file handle {} in IsLocalRef check, reason '{}'", Ref.FileHandle, Ec.message()); return false; } - LocalRefFile.Append(std::filesystem::absolute(FilePath)); - Path8 = LocalRefFile.ToUtf8(); + PathToUtf8(std::filesystem::absolute(FilePath), Path8); } - FileNameMap.insert_or_assign(Ref.FileHandle, Path8); + FileNameMap.insert_or_assign(Ref.FileHandle, Path8.ToString()); } - LocalRef.AbsolutePathLength = gsl::narrow<uint16_t>(Path8.size()); + LocalRef.AbsolutePathLength = gsl::narrow<uint16_t>(Path8.Size()); LocalRef.PayloadByteOffset = Ref.FileChunkOffset; LocalRef.PayloadByteSize = Ref.FileChunkSize; @@ -302,8 +302,8 @@ FormatPackageMessageInternal(const CbPackage& Data, FormatFlags Flags, void* Tar bool MarshalByLocalRef = EnumHasAllFlags(Flags, FormatFlags::kAllowLocalReferences) && (Compressed.GetSegments().size() == 1); bool DenyPartialLocalReferences = EnumHasAllFlags(Flags, FormatFlags::kDenyPartialLocalReferences); - CbAttachmentReferenceHeader LocalRef; - std::string Path8; + CbAttachmentReferenceHeader LocalRef; + ExtendableStringBuilder<128> Path8; if (MarshalByLocalRef) { @@ -321,7 +321,7 @@ FormatPackageMessageInternal(const CbPackage& Data, FormatFlags Flags, void* Tar const bool IsCompressed = true; bool IsHandle = false; #if ZEN_PLATFORM_WINDOWS - IsHandle = Path8.starts_with(HandlePrefix); + IsHandle = Path8.ToView().starts_with(HandlePrefix); #endif MarshalLocal(AttachmentInfo, Path8, LocalRef, AttachmentHash, IsCompressed, ResponseBuffers); ZEN_DEBUG("Marshalled '{}' as file {} of {} bytes", Path8, IsHandle ? "handle" : "path", Compressed.GetSize()); @@ -357,8 +357,8 @@ FormatPackageMessageInternal(const CbPackage& Data, FormatFlags Flags, void* Tar EnumHasAllFlags(Flags, FormatFlags::kAllowLocalReferences) && (AttachmentBinary.GetSegments().size() == 1); bool DenyPartialLocalReferences = EnumHasAllFlags(Flags, FormatFlags::kDenyPartialLocalReferences); - CbAttachmentReferenceHeader LocalRef; - std::string Path8; + CbAttachmentReferenceHeader LocalRef; + ExtendableStringBuilder<128> Path8; if (MarshalByLocalRef) { @@ -376,7 +376,7 @@ FormatPackageMessageInternal(const CbPackage& Data, FormatFlags Flags, void* Tar const bool IsCompressed = false; bool IsHandle = false; #if ZEN_PLATFORM_WINDOWS - IsHandle = Path8.starts_with(HandlePrefix); + IsHandle = Path8.ToView().starts_with(HandlePrefix); #endif MarshalLocal(AttachmentInfo, Path8, LocalRef, AttachmentHash, IsCompressed, ResponseBuffers); ZEN_DEBUG("Marshalled '{}' as file {} of {} bytes", Path8, IsHandle ? "handle" : "path", AttachmentBinary.GetSize()); |