| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: Hub hydration packs small files into raw CAS pack blobs to reduce request count for modules dominated by tiny metadata files
- `--hub-hydration-enable-pack` (Lua: `hub.hydration.enablepack`, default true)
- `--hub-hydration-pack-threshold-bytes` (Lua: `hub.hydration.packthresholdbytes`, default 256 KiB)
- `--hub-hydration-max-pack-bytes` (Lua: `hub.hydration.maxpackbytes`, default 4 MiB)
- Feature: Hub hydration and dehydration can be disabled per direction
- `--hub-enable-hydration` (Lua: `hub.enablehydration`, default true)
- `--hub-enable-dehydration` (Lua: `hub.enabledehydration`, default true)
- Feature: Hub hydration accepts a configurable file exclude list via `HydrationOptions` `excludes` (array of wildcards). Built-in defaults skip transient runtime files (`.lock`, `.sentry-native/*`, `state_marker`, `*.bak`, `gc/reserve.gc`, `auth/*`) so they no longer participate in dehydrate scans. Override semantics: a present field replaces the default outright; explicit `[]` opts out of all defaults.
- Improvement: Hub hydration completion logs now report per-request average and max latency, peak in-flight workers, queue wait, and hash-cache hit percentage; loose and pack-blob transfers are reported separately
- Improvement: Hub hydration pre-creates unique parent directories before scheduling parallel writes
- Improvement: S3 hydration retries transient HTTP failures (timeouts, 429 throttling, 5xx server errors, connection errors) up to 3 times via the HTTP client retry layer
- Improvement: S3 hydration multipart chunk size is persisted in `state.cbo` per module so hydrate replays the partitioning used at dehydrate; default raised to 64 MiB (was 32 MiB)
- Improvement: Hub hydration `Obliterate` retries backend delete once before falling back to local cleanup
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Integrates the **tourist** trace analysis library and builds a full `zen trace` command suite for working with Unreal Engine `.utrace` files.
### Trace analysis library (`thirdparty/tourist/`)
- Adds the tourist library as a third-party dependency with three modules: **foundation** (platform primitives, memory, scheduling), **trace** (UE Trace protocol decoding), and **analysis** (event dispatching and analyzer framework).
- Cross-platform support for Windows, Linux, and macOS.
### `zen trace` CLI commands (`src/zen/cmds/`, `src/zen/trace/`)
- **`zen trace analyze`** — Summarize a `.utrace` file: session metadata, thread inventory, command line + build configuration, CPU profiling scopes, timing, event rates, log messages, and (with symbols) memory allocation metrics including live-allocs dumps, callstack-keyed aggregation, and allocation churn. Optional HTML output for memory reports.
- **`zen trace inspect`** — Dump the event schema (declared types, fields, sizes) from a trace file.
- **`zen trace trim`** — Extract a time-window from a trace into a new `.utrace` file.
- **`zen trace serve`** — Launch a local HTTP server hosting an interactive trace viewer; opens in the default browser.
### Symbolication (`src/zen/trace/symbol_resolver.*`, `thirdparty/raw_pdb/`)
- Pluggable resolver with multiple backends: `pdb` (in-tree raw_pdb), `dbghelp` (Windows), `llvm-symbolizer` (all platforms), `atos` (macOS). An `auto` backend picks the best available tool per platform.
- Microsoft Symbol Server support: downloads PDBs on demand using a redirect-aware HTTP client.
- Local PDB cache keyed by image GUID preserves symbols across binary recompilation.
- Callstack trimming heuristic strips UE internal noise from reports.
- Binary analysis cache (`.ucache_z`) avoids re-resolving the same trace.
### Interactive trace viewer (`src/zen/frontend/html/`, `src/zen/trace/trace_viewer_service.*`)
- Timeline: scope-level detail, horizontal zoom/pan, vertical scrolling, viewport-driven loading with pre-computed LOD for responsive navigation of large traces.
- Thread grouping (collapsible sidebar sections) synthesized from name suffixes, natural sort order, visual distinction between lane threads and OS threads.
- Bookmark and region annotations; region categories with per-category toggles; bookmark marker toggle in the toolbar.
- Filterable Logs tab showing captured `UE_LOG` output.
- Stats tab with per-scope aggregate statistics.
- Memory tab with interactive allocation analysis and an allocation size histogram.
- CsvProfiler event parsing and chart UI.
### Other in-branch supporting changes
- **Cross-platform browser launcher** (`browser_launcher.{h,cpp}`) used by `trace serve`.
- **`ReciprocalU64`** fast 64-bit integer division (zencore/intmath) for trace analyzers.
- **`parallelsort`** cross-platform parallel sort helper (zenutil).
- Frontend zip build rule so the viewer's HTML assets are bundled into `zen.exe`.
- `/Zo` flag for better optimized debug info on Windows release builds.
- `trace-tests.cpp` in the `zen-test` harness (harness itself landed on main via #985).
|
| |
|
|
|
| |
- Introduce `CompactString`: a move-only, heap-allocated, immutable string wrapper that stores its length in a prefix byte for cheap `Size()`/`ToView()` while keeping the object to a single pointer.
- Swap the `ToString()` integer-formatting helpers in `zencore/string.cpp` to `std::to_chars`, which is ~5-10x faster and benefits every `IntNum` / `StringBuilder` / `CbJsonWriter` caller.
- No in-tree users on `main` yet; the type is ready for callers that want owned-string storage with lower per-entry overhead than `std::string` (e.g. long-lived log buffers, session records).
|
| |
|
|
|
|
|
|
| |
Adds infrastructure for reducing short-lived heap allocations, to be applied across the codebase in follow-up PRs.
- **`reduce-allocs` Claude Code skill** — reviews code for unnecessary heap allocations and suggests fixes using stack-friendly patterns (`ExtendableStringBuilder`, `eastl::fixed_vector`, `TRefCounted`, etc.)
- **`TransparentStringHash`** (`zencore/hashutils.h`) — enables `std::string_view` lookups on `std::string`-keyed `unordered_map` without allocating a temporary string (C++20 heterogeneous lookup via `is_transparent`)
- **`AppendPaddedInt()`** and **`AppendFill()`** on `StringBuilderBase` (`zencore/string.h`) — zero-padded integer formatting and repeated-character fills without going through `fmt::format`
- **`StringBuilderAppender`** output iterator adapter — allows `fmt::format_to` to write directly into a `StringBuilderBase`
|
| |
|
|
|
|
|
|
|
|
| |
- Replace per-type fmt::formatter specializations (StringBuilderBase, NiceBase) with a single generic formatter using a HasStringViewConversion concept
- Add ThousandsNum for comma-separated integer formatting (e.g. "1,234,567")
- Thread naming now accepts a sort hint for trace ordering
- Fix main thread trace registration to use actual thread ID and sort first
- Add ExpandEnvironmentVariables() for expanding %VAR% references in strings, with tests
- Add ParseHexBytes() overload with expected byte count validation
- Add Flag_BelowNormalPriority to CreateProcOptions (BELOW_NORMAL_PRIORITY_CLASS on Windows, setpriority on POSIX)
- Add PrettyScroll progress bar mode that pins the status line to the bottom of the terminal using scroll regions, with signal handler cleanup for Ctrl+C/SIGTERM
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Switches the default HTTP client to the libcurl-based backend and follows up with a series of correctness fixes and code quality improvements to `CurlHttpClient`.
**Backend switch & build fixes:**
- Switch default HTTP client to libcurl-based backend
- Suppress `[[nodiscard]]` warning when building fmt
- Miscellaneous bugfixes in HttpClient/libcurl
- Pass `-y` to `xmake config` in `xmake test` task
**Boilerplate reduction:**
- Add `Session::SetHeaders()` for RAII ownership of `curl_slist`, eliminating manual `curl_slist_free_all` calls from every verb method
- Add `Session::PerformWithResponseCallbacks()` to absorb the repeated 12-line write+header callback setup block
- Extract `ParseHeaderLine()` shared helper, replacing 4 duplicate header-parsing implementations
- Extract `BuildHeaderMap()` and `ApplyContentTypeFromHeaders()` helpers to deduplicate header-to-map conversion and Content-Type scanning
- Unify the two `DoWithRetry` overloads (PayloadFile variant now delegates to the Validate variant)
**Correctness fixes:**
- `TransactPackage`: both phases now use `PerformWithResponseCallbacks()`, fixing missing abort support and a dead header collection loop
- `TransactPackage`: error path now routes through `CommonResponse`, preserving curl error codes and messages for the caller
- `ValidatePayload`: merged 3 separate header-scan loops into a single pass
**Performance improvements:**
- Replace `fmt::format` with `ExtendableStringBuilder` in `BuildHeaderList` and `BuildUrlWithParameters`, eliminating heap allocations in the common case
- Replace `curl_easy_escape`/`curl_free` with inline URL percent-encoding using `AsciiSet`
- Remove wasteful `CommonResponse(...)` construction in retry logging, formatting directly from `CurlResult` fields
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Claude config updates
* Bug fixes and hardening across `zencore` and `zenhttp`, identified via static analysis.
### zencore
- **`ZEN_ASSERT` macro** -- extended to accept an optional string message literal; added `ZEN_ASSERT_MSG_` helper for message formatting. Callers needing runtime fmt-style formatting should use `ZEN_ASSERT_FORMAT`.
- **`MpscQueue`** -- fixed `TypeCompatibleStorage` to use a properly-sized `char Storage[sizeof(T)]` array instead of a single `char`; corrected `Data()` to cast `&Storage` rather than `this`; switched cache-line alignment to a fixed constant to avoid GCC's `-Winterference-size` warning. Enabled previously-disabled tests.
- **`StringBuilderImpl`** -- initialized `m_Base`/`m_CurPos`/`m_End` to `nullptr`. Fixed `StringCompare` return type (`bool` -> `int`). Fixed `ParseInt` to reject strings with trailing non-numeric characters. Removed deprecated `<codecvt>` include.
- **`NiceNumGeneral`** -- replaced `powl()` with integer `IntPow()` to avoid floating-point precision issues.
- **`RwLock::ExclusiveLockScope`** -- added move constructor/assignment; initialized `m_Lock` to `nullptr`.
- **`Latch::AddCount`** -- fixed variable type (`std::atomic_ptrdiff_t` -> `std::ptrdiff_t` for the return value of `fetch_add`).
- **`thread.cpp`** -- fixed Linux `pthread_setname_np` 16-byte name truncation; added null check before dereferencing in `Event::Close()`; fixed `NamedEvent::Close()` to call `close(Fd)` outside the lock region; added null guard in `NamedMutex` destructor; `Sleep()` now returns early for non-positive durations.
- **`MD5Stream`** -- was entirely stubbed out (no-op); now correctly calls `MD5Init`/`MD5Update`/`MD5Final`. Fixed `ToHexString` to use the correct string length. Fixed forward declarations. Fixed tests to compare `compare() == 0`.
- **`sentryintegration.cpp`** -- guard against null `filename`/`funcname` in spdlog message handler to prevent a crash in `fmt::format`.
- **`jobqueue.cpp`** -- fixed lost job ID when `IdGenerator` wraps around zero; fixed raw `Job*` in `RunningJobs` map (potential use-after-free) to `RefPtr<Job>`; fixed range-loop copies; fixed format string typo.
- **`trace.cpp`** -- suppress GCC false-positive warnings in third-party `trace.h` include.
### zenhttp
- **WebSocket close race** (`wsasio`, `wshttpsys`, `httpwsclient`) -- `m_CloseSent` promoted from `bool` to `std::atomic<bool>`; close check changed to `exchange(true)` to eliminate the check-then-set data race.
- **`wsframecodec.cpp`** -- reject WebSocket frames with payload > 256 MB to prevent OOM from malformed/malicious frames.
- **`oidc.cpp`** -- URL-encode refresh token and client ID in token requests (`FormUrlEncode`); parse `end_session_endpoint` and `device_authorization_endpoint` from OIDC discovery document.
- **`httpclientcommon.cpp`** -- propagate error code from `AppendData` when flushing the cache buffer.
- **`httpclient.h`** -- initialize all uninitialized members (`ErrorCode`, `UploadedBytes`, `DownloadedBytes`, `ElapsedSeconds`, `MultipartBoundary` fields).
- **`httpserver.h`** -- fix `operator=` return type for `HttpRpcHandler` (missing `&`).
- **`packageformat.h`** -- fix `~0u` (32-bit truncation) to `~uint64_t(0)` for a `uint64_t` field.
- **`httpparser`** -- initialize `m_RequestVerb` in both declaration and `ResetState()`.
- **`httpplugin.cpp`** -- initialize `m_BasePort`; fix format string missing quotes around connection name.
- **`httptracer.h`** -- move `#pragma once` before includes.
- **`websocket.h`** -- initialize `WebSocketMessage::Opcode`.
### zenserver
- **`hubservice.cpp`** -- fix two `ZEN_ASSERT` calls that incorrectly used fmt-style format args; converted to `ZEN_ASSERT_FORMAT`.
|
| |
|
|
|
| |
* remove stray std::unique_ptr<AuthMgr> Auth; causing crashes
* add more feedback during parsing of auth options
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Makes all test cases part of a test suite. Test suites are named after the module and the name of the file containing the implementation of the test.
* This allows for better and more predictable filtering of which test cases to run which should also be able to reduce the time CI spends in tests since it can filter on the tests for that particular module.
Also improves `xmake test` behaviour:
* instead of an explicit list of projects just enumerate the test projects which are available based on build system state
* also introduces logic to avoid running `xmake config` unnecessarily which would invalidate the existing build and do lots of unnecessary work since dependencies were invalidated by the updated config
* also invokes build only for the chosen test targets
As a bonus, also adds `xmake sln --open` which allows opening IDE after generation of solution/xmake project is done.
|
| |
|
|
|
|
| |
* when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing.
* this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance
* also added extended utf8-tests including a fix to `Utf8ToWide()`
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
zencore fixes:
- filesystem.cpp: ReadFile error reporting logic
- compactbinaryvalue.h: CbValue::As*String error reporting logic
zenhttp fixes:
- httpasio BindAcceptor would `return 0;` in a function returning `std::string` (UB)
- httpsys async workpool initialization race
zenstore fixes:
- cas.cpp: GetFileCasResults Results param passed by value instead of reference (large chunk results were silently lost)
- structuredcachestore.cpp: MissCount unconditionally incremented (counted hits as misses)
- cacherpc.cpp: Wrong boolean in Incomplete response array (all entries marked incomplete)
- cachedisklayer.cpp: sizeof(sizeof(...)) in two validation checks computed sizeof(size_t) instead of struct size
- buildstore.cpp: Wrong hash tracked in GC key list (BlobHash pushed twice instead of MetadataHash)
- buildstore.cpp: Removed duplicate m_LastAccessTimeUpdateCount increment in PutBlob
zenserver fixes:
- httpbuildstore.cpp: Reversed subtraction in HTTP range calculation (unsigned underflow)
- hubservice.cpp: Deadlock in Provision() calling Wake() while holding m_Lock (extracted WakeLocked helper)
- zipfs.cpp: Data race in GetFile() lazy initialization (added RwLock with shared/exclusive paths)
|
| |
|
|
|
|
|
|
| |
commands (#706)
* added `--exclude-folders` to `zen upload`, `zen download` and `zen diff`
added `--exclude-extensions` to `zen upload` and `zen diff`
excluded folder names are now matched by folder name in subfolders in addition to root level folders
* allow multiple token separators
|
| |
|
|
|
| |
these changes remove cpr from anything which is not `HttpClient` internals.
The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
|
| |
|
| |
Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
|
| |
|
|
|
|
|
| |
minor clean-up
`zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc
since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This change introduces support for tracing of memory allocation activity. The code is ported from UE5, and Unreal Insights can be used to analyze the output. This is currently only fully supported on Windows, but will be extended to Mac/Linux in the near future.
To activate full memory tracking, pass `--trace=memory` on the commandline alongside `--tracehost=<ip>` or `-tracefile=<path>`. For more control over how much detail is traced you can instead pass some combination of `callstack`, `memtag`, `memalloc` instead. In practice, `--trace=memory` is an alias for `--trace=callstack,memtag,memalloc`). For convenience we also support `--trace=memory_light` which omits call stacks.
This change also introduces multiple memory allocators, which may be selected via command-line option `--malloc=<allocator>`:
* `mimalloc` - mimalloc (default, same as before)
* `rpmalloc` - rpmalloc is another high performance allocator for multithreaded applications which may be a better option than mimalloc (to be evaluated). Due to toolchain limitations this is currently only supported on Windows.
* `stomp` - an allocator intended to be used during development/debugging to help track down memory issues such as use-after-free or out-of-bounds access. Currently only supported on Windows.
* `ansi` - fallback to default system allocator
|
|
|
* moved source directories into `/src`
* updated bundle.lua for new `src` path
* moved some docs, icon
* removed old test trees
|