aboutsummaryrefslogtreecommitdiff
path: root/src/zencore/include
Commit message (Collapse)AuthorAgeFilesLines
* updated chunking strategy (#589)Dan Engelbrecht2025-10-201-0/+132
| | | | | | | | | | - Improvement: `zen builds`now split large files that are compress only into 64 MB chunks to avoiding very large files in Cloud Storage - Improvement: `zen builds` now treats `.msixvc` files as non-compressable Moved and cleaned up compactbinary_helpers functions Tweaked fixed chunking implementation for better performance Refactored so we have one list of "non-compressable" extensions Implemented new `StandardChunkingStrategy` and move the two existing to hidden legacy namespace Added `FilteredDownloadedBytesPerSecond.Start();` call that got lost during previous refactoring
* refactor builds cmd part2 (#572)Dan Engelbrecht2025-10-141-0/+3
| | | | | * fix metadata info in filebuildstorage GetBuild * move MakeSafeAbsolutePathÍnPlace to filesystem.h/cpp * add BuildsOperationUploadFolder op moving code from builds_cmd.cpp
* refactor builds cmd (#566)Dan Engelbrecht2025-10-131-4/+4
| | | Move builds download code from builds_cmd.cpp to remotestorelib
* block reference couting copy support windows (#564)Dan Engelbrecht2025-10-111-0/+23
| | | | - Improvement: On Windows file systems that allow block reference counting we use it where possible to speed up copy of data during `zen builds download` operations - Enabled by default, disable with `--allow-file-clone=false`
* add ability to limit concurrency (#565)Stefan Boberg2025-10-101-2/+4
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* fix link error with operator new (#553)Stefan Boberg2025-10-041-23/+10
|
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-0/+80
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-291-0/+3
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* Make sure we call the previous terminate handle if present when we intercept ↵Dan Engelbrecht2025-09-262-2/+14
| | | | | | terminate calls (#514) Improvement: Make sure we call the previous terminate handle if present when we intercept terminate calls Improvement: Avoid allocating memory for call stack in terminate handle and assert callback
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-0/+1
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-101-11/+12
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-052-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-0/+9
| | | * add validation of compact binary payloads before reading them
* improve console output (#476)Dan Engelbrecht2025-08-261-1/+0
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
* Merge pull request #139 from ue-foundation/de/zen-service-commandLiam Mitchell2025-08-222-0/+7
|\ | | | | zen service command
| * Merge remote-tracking branch 'origin/main' into de/zen-service-commandLiam Mitchell2025-08-212-2/+13
| |\
| * \ Merge branch 'main' into de/zen-service-commandLiam Mitchell2025-07-2910-27/+214
| |\ \
| * \ \ Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-03-1413-26/+196
| |\ \ \
| * \ \ \ Merge remote-tracking branch 'origin/de/zen-service-command' into ↵Dan Engelbrecht2025-03-142-0/+6
| |\ \ \ \ | | | | | | | | | | | | | | | | | | de/zen-service-command
| | * | | | Implementation of service commands for Linux.Liam Mitchell2025-02-271-0/+4
| | | | | |
| | * | | | Linux compilation fixesLiam Mitchell2025-02-191-0/+2
| | | | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-231-9/+49
| |\ \ \ \ \ | | |/ / / / | |/| | | |
| * | | | | Merge remote-tracking branch 'origin/main' into de/zen-service-commandDan Engelbrecht2025-01-171-0/+185
| |\ \ \ \ \
| * | | | | | remove ServiceLevelDan Engelbrecht2025-01-101-1/+0
| | | | | | |
| * | | | | | get home folderDan Engelbrecht2025-01-101-0/+1
| | | | | | |
| * | | | | | add ServiceLevel for service processes: User, AllUsers and ServiceDan Engelbrecht2025-01-081-0/+1
| | | | | | |
* | | | | | | clean up trace options parsing (#473)Dan Engelbrecht2025-08-222-6/+8
| |_|_|_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | * clean up trace command line options explicitly shut down worker pools * some additional startup trace scopes
* | | | | | zen print fixes/improvements (#469)Dan Engelbrecht2025-08-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: `zen print` now allows output of compact binary content even if they are in non-optimal format (Unifom vs Non-Uniform arrays and objects) - Feature: `zen print` now has a `--show-type-info` option to add type information to output of compact binary content - Bugfix: Stats information for Build Store (Zen Store Cache) no longer throws exception and outputs invalid state information
* | | | | | use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-0/+11
| |_|_|_|/ |/| | | | | | | | | | | | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* | | | | graceful wait in parallelwork destructor (#438)Dan Engelbrecht2025-06-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | * exception safety when issuing ParallelWork * add asserts to Latch usage to catch usage errors * extended error messaging and recovery handling in ParallelWork destructor to help find issues
* | | | | sentry config (#430)Dan Engelbrecht2025-06-121-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: Added `--sentry-environment` to `zen` and `zenserver` - Feature: Added `--sentry-debug` to `zen` and `zenserver` - Feature: Added environment variable parsing for the following options: - `UE_ZEN_SENTRY_ENABLED`: `--no-sentry` (inverted) - `UE_ZEN_SENTRY_DEBUG`: `--sentry-debug` - `UE_ZEN_SENTRY_ALLOWPERSONALINFO`: `--sentry-allow-personal-info` - `UE_ZEN_SENTRY_DSN`: `--sentry-dsn` - `UE_ZEN_SENTRY_ENVIRONMENT`: `--sentry-environment`
* | | | | revert 61b4a88f and cadaad63Dan Engelbrecht2025-06-101-11/+5
| | | | |
* | | | | add sentry configurations options for debug/environmentDan Engelbrecht2025-06-101-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | add env-variable parsing for sentry option
* | | | | `--sentry-dsn` option for zen command line and zenserver to control Sentry ↵Dan Engelbrecht2025-06-091-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | reporting endpoint (#427) moved sentry database path to temporary directory for zen commandline
* | | | | pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-052-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* | | | | fixed size chunking for encrypted files (#410)Dan Engelbrecht2025-06-032-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use fixed size block chunking for know encrypted/compressed file types - Improvement: Skip trying to compress chunks that are sourced from files that are known to be encrypted/compressed - Improvement: Add global open file cache for written files increasing throughput during download by reducing overhead of open/close of file by 80%
* | | | | http client streaming upload (#413)Dan Engelbrecht2025-06-021-0/+11
| | | | | | | | | | | | | | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* | | | | made fmt use of format_context more consistent (#409)Stefan Boberg2025-05-262-4/+8
| | | | | | | | | | | | | | | fixes compilation issues on Linux in some cases
* | | | | sparse win file write (#398)Dan Engelbrecht2025-05-161-0/+2
| | | | | | | | | | | | | | | * Added `--use-sparse-files` option to `zen builds` command improving write performance of large files. Enabled by default.
* | | | | optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
* | | | | add sentry for zen command (#373)Dan Engelbrecht2025-05-061-0/+50
| | | | | | | | | | | | | | | | | | | | * refactor sentry integration and add to zen command line tool * move add_ldflags("-framework Security")
* | | | | add cxxopts overload for parsing file paths from command line (#362)Dan Engelbrecht2025-04-222-5/+0
| | | | |
* | | | | use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-0/+14
| | | | | | | | | | | | | | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* | | | | multithreaded clean (#331)Dan Engelbrecht2025-03-311-0/+5
| | | | | | | | | | | | | | | | | | | | - Improvement: Faster cleaning of directories - Improvement: Faster initial scanning of local state
* | | | | long filename support (#330)Dan Engelbrecht2025-03-311-5/+80
| | | | | | | | | | | | | | | - Bugfix: Long file paths now works correctly on Windows
* | | | | improved post upload/download summary (#308)Dan Engelbrecht2025-03-181-8/+10
| |_|_|/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * added ValidateStatistics and improved post upload summary * improved download statistics * smoother stats update when compressing * better feedback during stream compresss/decompress * don't capture TotalPartWriteCount by reference * disk stats cleanup * multi-test-download overall timer
* | | | fix quoted command lines arguments (#306)Dan Engelbrecht2025-03-122-0/+5
| | | | | | | | | | | | Handling of quotes and quotes with leading backslash for command line parsing - UE-231677
* | | | fixes for log timestamps (#304)Stefan Boberg2025-03-121-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | * add GetTimeSinceProcessStart returning time since process start. implemented using https://github.com/maxliani/GetTimeSinceProcessStart/tree/main * fix fractions when using epoch mode. Previously it would show the fraction from the absolute time stamp and not relative to epoch * used GetTimeSinceProcessStart to offset the epoch so that it represents the process spawn time
* | | | Build command tweaks (#301)Stefan Boberg2025-03-111-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Don't chunk up .mp4 files as they generally won't benefit from deduplication or partial in-place-updates - Improvement: Emit build name to console output when downloading a build - Improvement: Added some debug logging - Bugfix: Logging setup would previously not function correctly when not logging to file
* | | | pick up existing cache (#299)Dan Engelbrecht2025-03-102-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Scavenge .zen temp folders for existing data (downloaded, decompressed or written) from previous failed run - Improvement: Faster abort during stream compression - Improvement: Try to move downloaded blobs with rename if possible avoiding an extra disk write - Improvement: Only clean temp folders on successful or cancelled build - keep it if download fails
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-18 06:19:06 +0000