aboutsummaryrefslogtreecommitdiff
path: root/src/zencore/include
Commit message (Collapse)AuthorAgeFilesLines
* ported optimizations of MeasureVarUInt (#747)Dan Engelbrecht2026-02-061-2/+12
|
* avoid big ioworker backlog (#733)Dan Engelbrecht2026-01-261-2/+14
| | | | | | * add ability to override scheduling mode in ParallelWork * Don't increase buffering size when copying from local cache with --boost-worker-memory enabled * Rework scheduling writes of downloaded data to reduce memory usage
* zenserver API changes, some other minor changes (#720)Stefan Boberg2026-01-191-1/+14
| | | | | | | * add system metrics output to top command * removed unnecessary xmake directives * file system API/comment tweaks * fixed out-of-range access in httpserver test * updated ZenServer base API to allow customization by mode
* remove ZENCORE_API completely (#718)Stefan Boberg2026-01-1917-415/+396
| | | initially we had ZENCORE_API macros to potentially allow for DLL linkage. It turns out that this is not useful and the macros just contribute noise, so this change removes them completely.
* consul package and basic client added (#716)Stefan Boberg2026-01-194-17/+98
| | | | | | | | * this adds a consul package which can be used to fetch a consul binary * it also adds a `ConsulProcess` helper which can be used to spawn and manage a consul service instance * zencore dependencies brought across: - `except_fmt.h` for easer generation of formatted exception messages - `process.h/cpp` changes (adds `Kill` operation and process group support on Windows) - `string.h` changes to allow generic use of `WideToUtf8()`
* added options to configure exclude folders and extensions to zen build ↵Dan Engelbrecht2026-01-132-3/+55
| | | | | | | | commands (#706) * added `--exclude-folders` to `zen upload`, `zen download` and `zen diff` added `--exclude-extensions` to `zen upload` and `zen diff` excluded folder names are now matched by folder name in subfolders in addition to root level folders * allow multiple token separators
* automatic scrub on startup (#667)Dan Engelbrecht2025-11-272-6/+15
| | | | | - Improvement: Deeper validation of data when scrub is activated (cas/cache/project) - Improvement: Enabled more multi threading when running scrub operations - Improvement: Added means to force a scrub operation at startup with a new release using ZEN_DATA_FORCE_SCRUB_VERSION variable in xmake.lua
* GetSystemMetrics for Mac,Linux (#660)Stefan Boberg2025-11-241-3/+3
| | | Implemented GetSystemMetrics function for Mac and Linux
* loose chunk filtering bug when using wildcards (#654)Dan Engelbrecht2025-11-181-0/+15
| | | | | | | | | | * fix filtering of loose chunks when downloading with a filter add tests * changelog * move InlineRemoveUnusedHashes * remove extra braces
* switch to xmake for package management (#611)Stefan Boberg2025-11-073-18/+76
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* get oplog attachments (#622)Dan Engelbrecht2025-11-071-0/+2
| | | * add support for downloading individual attachments from an oplog
* Various fixes to address issues flagged by gcc / non-UE toolchain build (#621)Stefan Boberg2025-11-012-8/+8
| | | | | | | | | | | | | | | | | | | | * gcc: avoid using memset on nontrivial struct * redundant `return std::move` * fixed various compilation issues flagged by gcc * fix issue in xmake.lua detecting whether we are building with the UE toolchain or not * add GCC ignore -Wundef (comment is inaccurate) * remove redundant std::move * don't catch exceptions by value * unreferenced variables * initialize "by the book" instead of memset * remove unused exception reference * add #include <cstring> to fix gcc build * explicitly poulate KeyValueMap by traversing input spans fixes gcc compilation * remove unreferenced variable * eliminate redundant `std::move` which gcc complains about * fix gcc compilation by including <cstring> * tag unreferenced variable to fix gcc compilation * fixes for various cases of naming members the same as their type
* fix for Latch race (#617)Stefan Boberg2025-10-291-21/+11
| | | | | | Because the counter is decreased in `CountDown()` and subsequently checked against zero to determine if the completion event should be set: `Latch::Wait` checks the counter against zero on entry and would exit early, before waiting on the completion event. This could then lead to the `Latch` instance being torn down before the `CountDown()` does a `Set()` which could then lead to unexpected and unwanted things happening.
* fix --zentrace=no compile errors (#616)Stefan Boberg2025-10-282-5/+6
| | | | | | * make sure the correct `UE_WITH_TRACE` conditional is used to enable/disable support code as appropriate * fixed some accidental `int32`, `int64` et al usage, due to typedefs leaking through from trace header with this fix, it is now possible to build with `--zentrace=no` again
* refactor CasContainerStrategy::IterateOneBlock to make it more readable (#607)Dan Engelbrecht2025-10-241-1/+2
|
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-223-356/+107
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-0/+132
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-0/+3
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-4/+4
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* block reference couting copy support windows (#564)Dan Engelbrecht2025-10-111-0/+23
| | | | - Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations - Enabled by default, disable with `--allow-file-clone=false`
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-2/+4
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* fix link error with operator new (#553)Stefan Boberg2025-10-041-23/+10
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-0/+80
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-0/+3
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-262-2/+14
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-0/+1
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-11/+12
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-052-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-0/+9
| | | * add validation of compact binary payloads before reading them
* improve console output (#476)Dan Engelbrecht2025-08-261-1/+0
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* Merge pull request #139 from ue-foundation/de/zen-service-commandLiam Mitchell2025-08-222-0/+7
|\ | | | | zen service command
| * Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-212-2/+13
| |\
| * \ Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2910-27/+214
| |\ \
| * \ \ Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-03-1413-26/+196
| |\ \ \
| * \ \ \ Merge remote-tracking branch 'origin/de/zen-service-command' into ↵Dan Engelbrecht2025-03-142-0/+6
| |\ \ \ \ | | | | | | | | | | | | | | | | | | de/zen-service-command
| | * | | | Implementation of service commands for Linux.Liam Mitchell2025-02-271-0/+4
| | | | | |
| | * | | | Linux compilation fixesLiam Mitchell2025-02-191-0/+2
| | | | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-231-9/+49
| |\ \ \ \ \ | | |/ / / / | |/| | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-171-0/+185
| |\ \ \ \ \
| * | | | | | remove ServiceLevelDan Engelbrecht2025-01-101-1/+0
| | | | | | |
| * | | | | | get home folderDan Engelbrecht2025-01-101-0/+1
| | | | | | |
| * | | | | | add ServiceLevel for service processes: User, AllUsers and ServiceDan Engelbrecht2025-01-081-0/+1
| | | | | | |
* | | | | | | clean up trace options parsing (#473)Dan Engelbrecht2025-08-222-6/+8
| |_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* | | | | | zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* | | | | | use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-0/+11
| |_|_|_|/ |/| | | | | | | | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* | | | | graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
* | | | | sentry config (#430)Dan Engelbrecht2025-06-121-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* | | | | revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-11/+5
| | | | |
* | | | | add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | add env-variable parsing for sentry option
* | | | | `--sentry-dsn` option for zen command line and zenserver to control Sentry ↵Dan Engelbrecht2025-06-091-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | reporting endpoint (#427) moved sentry database path to temporary directory for zen commandline
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-18 23:24:25 +0000