aboutsummaryrefslogtreecommitdiff
path: root/src/zencore/include
Commit message (Collapse)AuthorAgeFilesLines
* switch to xmake for package management (#611)Stefan Boberg2025-11-073-18/+76
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* get oplog attachments (#622)Dan Engelbrecht2025-11-071-0/+2
| | | * add support for downloading individual attachments from an oplog
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-012-8/+8
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* fix for Latch race (#617)Stefan Boberg2025-10-291-21/+11
| | | | | | Because the counter is decreased in `CountDown()` and subsequently checked against zero to determine if the completion event should be set: `Latch::Wait` checks the counter against zero on entry and would exit early, before waiting on the completion event. This could then lead to the `Latch` instance being torn down before the `CountDown()` does a `Set()` which could then lead to unexpected and unwanted things happening.
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-282-5/+6
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* refactor CasContainerStrategy::IterateOneBlock to make it more readable (#607)Dan Engelbrecht2025-10-241-1/+2
|
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-223-356/+107
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-0/+132
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-0/+3
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-4/+4
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* block reference couting copy support windows (#564)Dan Engelbrecht2025-10-111-0/+23
| | | | - Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations - Enabled by default, disable with `--allow-file-clone=false`
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-2/+4
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* fix link error with operator new (#553)Stefan Boberg2025-10-041-23/+10
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-0/+80
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-0/+3
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-262-2/+14
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-0/+1
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-11/+12
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-052-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-0/+9
| | | * add validation of compact binary payloads before reading them
* improve console output (#476)Dan Engelbrecht2025-08-261-1/+0
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* Merge pull request #139 from ue-foundation/de/zen-service-commandLiam Mitchell2025-08-222-0/+7
|\ | | | | zen service command
| * Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-212-2/+13
| |\
| * \ Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2910-27/+214
| |\ \
| * \ \ Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-03-1413-26/+196
| |\ \ \
| * \ \ \ Merge remote-tracking branch 'origin/de/zen-service-command' into ↵Dan Engelbrecht2025-03-142-0/+6
| |\ \ \ \ | | | | | | | | | | | | | | | | | | de/zen-service-command
| | * | | | Implementation of service commands for Linux.Liam Mitchell2025-02-271-0/+4
| | | | | |
| | * | | | Linux compilation fixesLiam Mitchell2025-02-191-0/+2
| | | | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-231-9/+49
| |\ \ \ \ \ | | |/ / / / | |/| | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-171-0/+185
| |\ \ \ \ \
| * | | | | | remove ServiceLevelDan Engelbrecht2025-01-101-1/+0
| | | | | | |
| * | | | | | get home folderDan Engelbrecht2025-01-101-0/+1
| | | | | | |
| * | | | | | add ServiceLevel for service processes: User, AllUsers and ServiceDan Engelbrecht2025-01-081-0/+1
| | | | | | |
* | | | | | | clean up trace options parsing (#473)Dan Engelbrecht2025-08-222-6/+8
| |_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* | | | | | zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* | | | | | use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-0/+11
| |_|_|_|/ |/| | | | | | | | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* | | | | graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
* | | | | sentry config (#430)Dan Engelbrecht2025-06-121-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* | | | | revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-11/+5
| | | | |
* | | | | add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | add env-variable parsing for sentry option
* | | | | `--sentry-dsn` option for zen command line and zenserver to control Sentry ↵Dan Engelbrecht2025-06-091-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | reporting endpoint (#427) moved sentry database path to temporary directory for zen commandline
* | | | | pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-052-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* | | | | fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-032-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
* | | | | http client streaming upload (#413)Dan Engelbrecht2025-06-021-0/+11
| | | | | | | | | | | | | | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* | | | | made fmt use of format_context more consistent (#409)Stefan Boberg2025-05-262-4/+8
| | | | | | | | | | | | | | | fixes compilation issues on Linux in some cases
* | | | | sparse win file write (#398)Dan Engelbrecht2025-05-161-0/+2
| | | | | | | | | | | | | | | * Added `--use-sparse-files` option to `zen builds` command improving write performance of large files. Enabled by default.
* | | | | optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
* | | | | add sentry for zen command (#373)Dan Engelbrecht2025-05-061-0/+50
| | | | | | | | | | | | | | | | | | | | * refactor sentry integration and add to zen command line tool * move add_ldflags("-framework Security")
* | | | | add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-222-5/+0
| | | | |
* | | | | use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-0/+14
| | | | | | | | | | | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-18 01:57:32 +0000