aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
* move cpr in-tree (#605)Stefan Boberg2025-10-241-2/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* add host discovery and zen cache support for oplog import (#601)Dan Engelbrecht2025-10-231-13/+29
| | | * add host discovery and zen cache support for oplog import
* add support for OTLP logging/tracing (#599)Stefan Boberg2025-10-2210-6/+168
| | | | | | | | - adds `zentelemetry` project which houses new functionality for serializing logs and traces in OpenTelemetry Protocol format (OTLP) - moved existing stats functionality from `zencore` to `zentelemetry` - adds `TRefCounted<T>` for vtable-less refcounting - adds `MemoryArena` class which allows for linear allocation of memory from chunks - adds `protozero` which is used to encode OTLP protobuf messages
* Mr/dashboard stats summary tweak (#592)Martin Ridgers2025-10-202-9/+14
| | | | | | | * More robust dashboard stats summary * Updated changelog * Updated frontend zip archive
* fix builds storage stats (#590)Dan Engelbrecht2025-10-201-18/+22
| | | * restructure builds storage stats to match web-ui expectations
* restructured zenserver configuration (#575)Stefan Boberg2025-10-1513-210/+317
| | | this breaks out the configuration logic to allow multiple applications to share common configuration and initialization logic whilst customizing chosen aspects of the process
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-1432-44/+31
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* split storage config from base config (#570)Stefan Boberg2025-10-136-1208/+1355
| | | | | this change splits the configuration logic into a base part and a storage server part manually tested with shared server config file
* extract storage server into separate source files (#569)Stefan Boberg2025-10-135-1185/+1236
| | | | this change moves common base service code into `zenserver.(cpp|h)` and storage server code into `zenstorageserver.(cpp|h)` to more clearly separate concerns but also to make way for another service
* move service common code into base class (#567)Stefan Boberg2025-10-135-499/+626
| | | | | Separates storage server code and generic server code into two classes. This is a change to prepare for different services to be implemented using the same framework, into the same executable
* hide http.sys options when unavailable (#568)Stefan Boberg2025-10-132-3/+10
|
* add ability to limit concurrency (#565)Stefan Boberg2025-10-105-2/+13
| | | | | | | | | | | | effective concurrency in zenserver can be limited via the `--corelimit=<N>` option on the command line. Any value passed in here will be used instead of the return value from `std::thread::hardware_concurrency()` if it is lower. * added --corelimit option to zenserver * made sure thread pools are configured lazily and not during global init * added log output indicating effective and HW concurrency * added change log entry * removed debug logging from ZenEntryPoint::Run() also removed main thread naming on Linux since it makes the output from `top` and similar tools confusing (it shows `main` instead of `zenserver`)
* shorten thread pool names for Linux which has a limit of 15 characters (#563)Stefan Boberg2025-10-081-1/+1
| | | without this change we just get a whole bunch of threads named "main" or "zenserver" depending on timing
* added Hidden option to oidctoken creation with oidc token exe (#556)Dan Engelbrecht2025-10-061-2/+2
|
* move rpcrecorder out from cache subfolder (#552)Dan Engelbrecht2025-10-031-1/+1
|
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-033-15/+5
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* zenutil cleanup (#550)Dan Engelbrecht2025-10-031-1/+1
| | | | * move referencemetadata to zenstore * rename zenutil/windows/service to windowsservice
* move zen vfs implementation to zenstore (#549)Dan Engelbrecht2025-10-036-617/+16
| | | * move zen vfs implementation to zenstore
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-034-8/+26
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* cache RPC replay fixes (minor) (#544)Stefan Boberg2025-10-031-1/+1
| | | | | * fixes misleading server side log message when using `--onhost` * fixed `--onhost` behaviour in zen (no longer replays twice)
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-0313-5436/+6
| | | * move remoteproject code to remotestorelib
* move projectstore to zenstore (#541)Dan Engelbrecht2025-10-0210-8831/+5
|
* projectstore refactor phase 2 (#539)Dan Engelbrecht2025-10-028-1681/+1524
| | | Refactor projectstore/httpprojectservice to prepare for move of projectstore to zenstore
* Zs/OIDC exe path handling (#538)Zousar Shaker2025-10-021-4/+10
| | | | | * Fix handling of oidc-exe-path The path was only being set if the file DIDN'T exist on the server (along with the warning about it not existing). Now set if it DOES exist, and the warning emitted if it DOESN'T.
* Fix bug in ReplayLogEntries for large op blobs (#537)Matt Peters2025-10-011-0/+1
| | | | | - Bugfix: Fixed issue where large ops in oplog would fail to read properly ReplayLogEntries: Fix bug in the case where an op is too big for OpBl…obsBuffer. The OpBufferView needs to be pointed to the temporary buffer.
* skip dropped oplogs in ScanForOplogs() (#534)Dan Engelbrecht2025-09-301-0/+5
|
* projectstore refactor (#531)Dan Engelbrecht2025-09-304-721/+667
| | | | | | | | | | | | | | | | | * convert ProjectStore::GetProjectFiles to not use http return codes * convert ProjectStore::GetProjectChunkInfos to not use http return codes * convert ProjectStore::GetChunkInfo to not use http return codes * convert ProjectStore::GetChunkRange to not use http return codes * convert ProjectStore::GetChunk to not use http return codes * convert ProjectStore::PutChunk to not use http return codes * convert ProjectStore::WriteOplog to not use http return codes * convert ProjectStore::ReadOplog to not use http return codes
* gracefully handle missing chunks when exporting an oplog (#526)Dan Engelbrecht2025-09-292-28/+66
| | | | - Improvement: Gracefully handle missing chunks when exporting an oplog - Bugfix: Add object validation when replaying oplog to avoid crash on malformed data
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-295-185/+92
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* some bug fixes (#522)Stefan Boberg2025-09-291-2/+2
| | | | | | * fix for invalid regex in HttpBuildStoreService - triggers with most recent MSVC version * in GcScheduler don't wait for exit signal if exit has already been requested. this caused extended waits for shutdown in some automated tests on very fast machines, possibly also due to some behaviour change in condition_variable * speculative fix/workaround for issue with TLS teardown on secondary thread while main was tearing down trace
* new append op rpc method (#511)Dan Engelbrecht2025-09-261-227/+375
| | | | | | - Feature: New `/prj/{project}/{oplog}/rpc` endpoint method `appendops` to send an array of oplog ops and receiving a list of `need` for attachments not present - Feature: Added `usingtmpfiles` boolean field to `/prj/{project}/{oplog}/rpc` method `putchunks` to be explicit about allowing move of temp attachment files - Improvement: Added additional validation of compact binary objects when reading from disk/receiving from client - Improvement: Windows: Added fallback code to use standard `MoveFile` api when rename via `SetFileInformationByHandle` fails in `MoveToFile` (used by filecas)
* Merge pull request #509 from ue-foundation/zs/put-overwrite-policy-responseZousar Shaker2025-09-251-5/+10
|\ | | | | Zs/put overwrite policy response
| * Report Incomplete Records To Clientzousar2025-09-241-3/+5
| | | | | | | | When requesting partial records, report back when a record is incomplete via an "Incomplete" array of bools that is a sibling to the "Result" array for batch/rpc operations, or via the HttpResponseCode::PartialContent status code for individual record requests.
| * Merge branch 'main' into zs/put-overwrite-policy-responsezousar2025-09-226-10/+40
| |\
| * | Change batch put responses for client reportingzousar2025-09-191-2/+5
| | | | | | | | | | | | Conflicts are now treated as successes, and we optionally return a Details array instead of an ErrorMessages array. Details are returned for all requests in a batch, or no requests in a batch depending on whether there are any details to be shared about any of the put requests. The details for a conflict include the raw hash and raw size of the item. If the item is a record, we also include the record as an object.
* | | limit number of chunks per block (#508)Dan Engelbrecht2025-09-233-49/+77
| |/ |/| | | | | | | - Improvement: Chunk block generation for `zen oplog-export` and `zen builds upload` command are now limited to max 4000 chunks per block zen builds list-block testing function to check results for recently used blocks in a context
* | dont set Size or RawSize when fetching chunkinfos if the payload can't be ↵Dan Engelbrecht2025-09-221-2/+7
| | | | | | | | found/validated (#507)
* | more responsive cancel during oplog import (#505)Dan Engelbrecht2025-09-221-0/+18
| | | | | | | | - Improvement: Faster oplog import due to chunk existance check improvement - Improvement: Cancelling oplog import is now more responsive during initial phase
* | Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-225-8/+15
|/ | | | command to use unattended mode when launching oidc-token.exe (#506)
* Sorting oplog tree view by size would raise an error. (#497)Martin Ridgers2025-09-162-1/+1
| | | | | | | * Sorting oplog tree view by size would raise an error * Updated frontend .zip archive * Updated changelog
* new in-memory storage strategy for oplogs (#490)Dan Engelbrecht2025-09-154-518/+748
| | | - Improvement: Revised project oplog in-memory representation which reduces load times and memory usage
* initialize all services before healthy (#493)Dan Engelbrecht2025-09-121-35/+58
| | | * make sure all services are initialized and registered before we respond OK to health check
* add EMode to WorkerTheadPool to avoid thread starvation (#492)Dan Engelbrecht2025-09-104-771/+805
| | | - Improvement: Add a new mode to worker thread pools to avoid starvation of workers which could cause long stalls due to other work begin queued up. UE-305498
* faster oplog entries with referenceset (#488)Dan Engelbrecht2025-09-085-56/+86
| | | | - Improvement: Faster project store `/entries` endpoint, 10-15% faster when using a reference set to limit entries fix missing space after task name in pretty progressbar
* refactor zen command return value handling (#487)Dan Engelbrecht2025-09-051-37/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Improvement: Use consistent language for command line argument parsing errors - Improvement: Changed zen command parsing errors to output help first and error last to make it easier to spot the error - Improvement: Refactor zen command return codes to conform to valid Linux range (0-255) kSuccess = 0, kOtherError = 1, kBadInput = 2, kOutOfMemory = 16, kOutOfDisk = 17, kAssertError = 70, kHttpOtherClientError = 80, kHttpCantConnectError = 81, kHttpNotFound = 66, // NotFound(404) kHttpUnauthorized = 77, // Unauthorized(401), kHttpSLLError = 82, kHttpForbidden = 83, // Forbidden(403) kHttpTimeout = 84, // RequestTimeout(408) kHttpConflict = 85, // Conflict(409) kHttpNoHost = 86, kHttpOtherServerError = 90, kHttpInternalServerError = 91, // InternalServerError(500) kHttpServiceUnavailable = 69, // ServiceUnavailable(503) kHttpBadGateway = 92, // BadGateway(502) kHttpGatewayTimeout = 93, // GatewayTimeout(504)
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-047-199/+266
| | | * add validation of compact binary payloads before reading them
* oplog memory usage reduction (#482)Dan Engelbrecht2025-09-043-99/+195
| | | | | | - Improvement: For projectstore oplog GET operation, only read basic information and release it if the oplog is not already open to reduce memory usage when listing oplogs in web UI - Improvement: Reduce memory usage for oplog Op address lookup Refactored Oplog::EState -> Oplog ::EMode and make sure we open data files in read-only mode when EMode::kBasicReadOnly is used.
* projectstore lifetime improvements (#481)Dan Engelbrecht2025-08-294-368/+552
| | | | | - Improvement: Oplogs that have not been touched for 15 min are unloaded from memory during GC pass, oplogs are reloaded on demand - Improvement: Oplogs read for GC/Validation operations are loaded using a lightweight mode reducing memory usage and load times - Improvement: Cleaned up logging for oplogs/projects to reduce clutter
* rework `--quiet` zenserver option add `--noconsole` option (#477)Dan Engelbrecht2025-08-265-38/+87
| | | | - Improvement: Changed zenserver `--quiet` option to suppress INFO level messages and below to surface startup and runtime errors - Feature: Added `--noconsole` option that suppresses all output to standard out, this works as the `--quiet` option used to work
* improve console output (#476)Dan Engelbrecht2025-08-262-3/+3
| | | | * add color coded logging level to console output (for warn/err/crit levels) * clean up console output
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-21 08:37:56 +0000