aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
* Zs/oplog navigation fix (#731)Zousar Shaker2026-01-232-6/+6
| | | * Fix incorrect oplog navigation symbols
* hotfix 5.7.18 (#730)Dan Engelbrecht2026-01-221-2/+5
| | | | * make sure we properly convert command line args for zenserver as well * make sure we *add* wildcards/excludes in addition to defaults
* zen hub (#574)Stefan Boberg2026-01-219-10/+1546
| | | | | Initial implementation of zenserver "hub" mode. This is an experimental feature. zenserver can be started in hub mode by specifying `hub` as the first argument to zenserver
* zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-195-25/+129
| | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
* OTLP/trace improvements (#717)Stefan Boberg2026-01-193-15/+19
| | | | | | | This PR brings over some changes made to avoid performing setup for otel instrumentation if we are not sending otel information anywhere anyway. It also adds the ability to configure an OTLP endpoint on the command line using `--otlp-endpoint=<URI>`. Bear in mind that OTLP support is still not officially supported so this should not be used in production at this stage.
* use current locale when converting wide char command line arguments to ↵Dan Engelbrecht2026-01-151-0/+8
| | | | | | string (#712) * set utf8 locale, only set LC_CTYPE as we don't want to affect how decimal point or sorting orders are handled * set language/region explicitly so we can use LC_ALL
* asio/http optimizations (#449)Stefan Boberg2026-01-141-27/+2
| | | | | | | This change primarily introduces improved logic for dealing with sending data from file references. This is intended to reduce the amount of memory-mapping we end up doing when sending data from files. Windows now uses `TransmitFile` to send file data more efficiently using kernel-side I/O, but Linux/Mac basically behaves as before since they don't offer any true async file I/O support via asio. This should be implemented separately using a background I/O thread pool. This PR also includes improved memory management for http/asio with reduced allocation counts, and a fix for a potential use-after-free in very high load scenarios.
* if we fail to create the server mutex, gracefully report error without ↵Dan Engelbrecht2026-01-121-1/+3
| | | | sending error to Sentry (#705)
* Avoid rendering user text input as HTML (#700)Liam Mitchell2026-01-092-1/+1
| | | * Avoid rendering user text input as HTML
* Ensure upstream put propagation includes overwritezousar2025-12-194-5/+39
| | | | When changing the default limit-overwrite behavior, a unit test surfaced a bug where an put of data with overwrite cache policy would not get propagated via zen's built-in upstream mechanism with a matching overwrite cache policy to the upstream. This change ensures that it does and leaves the unit test configured to exercise this scenario.
* Change default limit-overwrite behavior to truezousar2025-12-172-2/+2
|
* remove error warning in output (#695)Dan Engelbrecht2025-12-172-2/+2
| | | * changed some logging string so they don't get caught in CI logging
* add boost-worker oplog import export options (#693)Dan Engelbrecht2025-12-162-22/+73
| | | | | | | | | | - Feature: `zen oplog-export`, `zen oplog-import` and `zen oplog-download` now has options to boost workers - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive - Improvement: Refactored boost options for `zen builds` operations `upload`, `download`, `diff`, `prime-cache`, `fetch-blob` and `validate-part` - `--boost-worker-count` - Increase the number of worker threads - may cause computer to be less responsive - `--boost-worker-memory` - Increase the limit where we write downloaded data to temporary storage to conserve space - may cause computer to be less responsive due to high memory usage - `--boost-workers` - Enables both 'boost-worker-count' and 'boost-worker-memory' - may cause computer to be less responsive
* add otel instrumentation (#581)Stefan Boberg2025-12-113-0/+19
| | | | | | | | this change adds OTEL tracing to a few places * Top-level application lifecycle (config/init/cleanup, main loop) * http.sys requests it also brings some otlptrace optimizations and dynamic configuration of tracing. OTLP tracing is currently always disabled
* upload to zen cache from oplog import now disabled by default (#680)Dan Engelbrecht2025-12-091-1/+1
|
* add missing patterns to objectstore service http routing (#674)Dan Engelbrecht2025-12-031-3/+7
|
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-273-32/+46
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* logging improvements (#664)Stefan Boberg2025-11-243-21/+24
| | | | | * Eliminate spurious `FlushFileBuffers` calls on Windows by tracking writes * Make cache log file creation optional
* Added actions to drop all projects or all z$ namespaces. (#655)Martin Ridgers2025-11-183-2/+54
| | | | | | | | | | | * Save references to the project and zcache tables * Add an attribute to table rows with the actionable project/namespace id * Drop-all option for projects and cache namespaces * Updated frontend .zip archive * Edited changelog
* Include version string on the dashboard's start page. (#651)Martin Ridgers2025-11-174-2/+26
| | | | | | | | | | | * Method to get plain text from an async request * Include server's version on the dashboard start page * Same paragraph style as the rest of the method * Updated changelog * Update frontend archive
* control cache upload (#646)Dan Engelbrecht2025-11-121-0/+2
| | | * add option to enable/disable upload to builds cache
* switch to xmake for package management (#611)Stefan Boberg2025-11-074-10/+55
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-018-34/+35
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* make sure OpenProcessCache is initialized before use (#618)Stefan Boberg2025-10-291-4/+4
| | | previously, a null reference would be passed into ProjectStore constructor
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-282-2/+4
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* move cpr in-tree (#605)Stefan Boberg2025-10-241-2/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* add host discovery and zen cache support for oplog import (#601)Dan Engelbrecht2025-10-231-13/+29
| | | * add host discovery and zen cache support for oplog import
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-2210-6/+168
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* Mr/dashboard stats summary tweak (#592)Martin Ridgers2025-10-202-9/+14
| | | | | | | * More robust dashboard stats summary * Updated changelog * Updated frontend zip archive
* fix builds storage stats (#590)Dan Engelbrecht2025-10-201-18/+22
| | | * restructure builds storage stats to match web-ui expectations
* restructured zenserver configuration (#575)Stefan Boberg2025-10-1513-210/+317
| | | this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-1432-44/+31
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* split storage config from base config (#570)Stefan Boberg2025-10-136-1208/+1355
| | | | | this change splits the configuration logic into a base part and a storage server part manually tested with shared server config file
* extract storage server into separate source files (#569)Stefan Boberg2025-10-135-1185/+1236
| | | | this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
* move service common code into base class (#567)Stefan Boberg2025-10-135-499/+626
| | | | | Separates storage server code and generic server code into two classes. This is a change to prepare for different services to be implemented using the same framework, into the same executable
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-132-3/+10
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-105-2/+13
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* shorten thread pool names for Linux which has a limit of 15 characters (#563)Stefan Boberg2025-10-081-1/+1
| | | without this change we just get a whole bunch of threads named "main" or "zenserver" depending on timing
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-061-2/+2
|
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-1/+1
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-033-15/+5
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-031-1/+1
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-036-617/+16
| | | * move zen vfs implementation to zenstore
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-8/+26
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* cache RPC replay fixes (minor) (#544)Stefan Boberg2025-10-031-1/+1
| | | | | * fixes misleading server side log message when using `--onhost` * fixed `--onhost` behaviour in zen (no longer replays twice)
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-0313-5436/+6
| | | * move remoteproject code to remotestorelib
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-0210-8831/+5
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-028-1681/+1524
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* Zs/OIDC exe path handling (#538)Zousar Shaker2025-10-021-4/+10
| | | | | * Fix handling of oidc-exe-path The path was only being set if the file DIDN'T exist on the server (along with the warning about it not existing). Now set if it DOES exist, and the warning emitted if it DOESN'T.
* Fix bug in ReplayLogEntries for large op blobs (#537)Matt Peters2025-10-011-0/+1
| | | | | - Bugfix: Fixed issue where large ops in oplog would fail to read properly ReplayLogEntries: Fix bug in the case where an op is too big for OpBl…obsBuffer. The OpBufferView needs to be pointed to the temporary buffer.
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-22 02:26:39 +0000