aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* missing chunks bugfix (#424)Dan Engelbrecht2025-06-097-84/+479
| | | | | | | | | | | * make sure to close log file when resetting log * drop entries that refers to missing blocks * Don't scrub keys that has been rewritten * currectly count added bytes / m_TotalSize * fix negative sleep time in BlockStoreFile::Open() * be defensive when fetching log position * append to log files *after* we updated all state successfully * explicitly close stuff in destructors with exception catching * clean up empty size block store files
* pause, resume and abort running builds cmd (#421)Dan Engelbrecht2025-06-054-7/+14
| | | | | - Feature: `zen builds pause`, `zen builds resume` and `zen builds abort` commands to control a running `zen builds` command - `--process-id` the process id to control, if omitted it tries to find a running process using the same executable as itself - Improvement: Process report now indicates if it is pausing or aborting
* fix cachbucket mem hit count (#415)Dan Engelbrecht2025-06-021-4/+7
| | | | | * Don't count a miss twice for memory stats if the entry can't be found * changelog
* add missing flush inblockstore compact (#411)Dan Engelbrecht2025-05-305-17/+130
| | | | - Bugfix: Flush the last block before closing the last new block written to during blockstore compact. UE-291196 - Feature: Drop unreachable CAS data during GC pass. UE-291196
* faster oplog validate (#408)Dan Engelbrecht2025-05-304-10/+69
| | | Improvement: Faster oplog validate to reduce GC wall time and disk I/O pressure
* frequent disk space check (#407)Dan Engelbrecht2025-05-271-25/+84
| | | | * check low disk space condition more frequently and trigger GC when low water mark is reached * show waited time when waiting for zenserver instance to exit
* unblock cache bucket drop (#406)Dan Engelbrecht2025-05-264-54/+124
| | | | * don't hold exclusive locks while deleting files from a dropped bucket/namespace * cleaner detection of missing namespace when issuing a drop
* handle exception with batch work (#401)Dan Engelbrecht2025-05-194-134/+147
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* make sure tests initialize trace so we don't end up allocating tons of ↵Stefan Boberg2025-05-151-0/+2
| | | | memory for no reason (#397)
* keep snapshot on log delete fail (#391)Dan Engelbrecht2025-05-126-104/+81
| | | | | - Improvement: Cleaned up snapshot writing for CompactCAS/FileCas/Cache/Project stores - Improvement: Safer recovery when failing to delete log for CompactCAS/FileCas/Cache/Project stores - Improvement: Added log file reset when writing snapshot at startup for FileCas
* enable per bucket config (#388)Dan Engelbrecht2025-05-122-2/+22
| | | | Feature: Add per bucket cache configuration (Lua options file only) Improvement: --cache-memlayer-sizethreshold is now deprecated and has a new name: --cache-bucket-memlayer-sizethreshold to line up with per cache bucket configuration
* tweak iterate block parameters (#390)Dan Engelbrecht2025-05-122-34/+50
| | | * tweak block iteration chunk sizes
* flush cas log file (#387)Dan Engelbrecht2025-05-092-46/+31
| | | * make sure we remove the cas log file when writing full index at startup
* make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock ↵Dan Engelbrecht2025-05-071-85/+61
| | | | | contention (#385) * make RemoveExpiredData and PreCache serial to reduce CPU overhead / lock contention
* optimize block store CompactBlocks (#384)Dan Engelbrecht2025-05-071-14/+19
| | | | | - Improvement: Optimize block compact reducing memcpy operations - Improvement: Handle padding of block store blocks when compacting to avoid excessive flusing of write buffer - Improvement: Handle padding when writing oplog index snapshot to avoid unnecessary flushing of write buffer
* optimize cache bucket state writing (#382)Dan Engelbrecht2025-05-061-42/+59
| | | * optimize cache bucket snapshot and sidecar writing
* make OOD and OOM in gc non critical (#381)Dan Engelbrecht2025-05-051-27/+218
| | | | * oom and ood exceptions in GC are now treated as warnings instead of errors
* silence Out Of Disk errors to sentry (#378)Dan Engelbrecht2025-05-051-48/+56
| | | | | * block writing GC state/info if disk is full * fix if/else on error while writing gc state
* iterate chunks crash fix (#376)Dan Engelbrecht2025-05-024-76/+314
| | | * Bugfix: Add explicit lambda capture in CasContainer::IterateChunks to avoid accessing state data references
* cbobject validation (#377)Dan Engelbrecht2025-05-021-10/+31
| | | | * validate incoming CbObject to cache when receiving a package * validate records when fetched from store in cache before parsing them
* replace local equal_to_2 with eastl impl (#368)Stefan Boberg2025-04-251-16/+2
|
* predicate to enable compiling with later EASTL version (#367)Stefan Boberg2025-04-241-1/+1
|
* blobstore size limit (#342)Dan Engelbrecht2025-04-042-13/+255
| | | - Feature: zenserver option `--buildstore-disksizelimit` to set an soft upper limit for build storage data. Defaults to 1TB.
* build store save access times (#341)v5.6.3-pre0Dan Engelbrecht2025-04-033-30/+304
| | | | | * save payload size in log for buildstore * read/write access times and manifest for buldstore * use retry when removing temporary files
* use oidctoken executable to generate auth (#336)Dan Engelbrecht2025-04-021-13/+0
| | | - Feature: `zen builds` auth option `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token
* reduce disk io during gc (#335)Dan Engelbrecht2025-04-011-20/+8
| | | * do cache bucket flush/write snapshot as part of compact to reduce disk I/O
* long filename support (#330)Dan Engelbrecht2025-03-319-117/+117
| | | - Bugfix: Long file paths now works correctly on Windows
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-5/+26
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-2611-46/+1738
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* reduced memory churn using fixed_xxx containers (#236)Stefan Boberg2025-03-067-108/+143
| | | | | | * Added EASTL to help with eliminating memory allocations * Applied EASTL to eliminate memory allocations, primarily by using `fixed_vector` et al to use stack allocations / inline struct allocations Reduces memory events in traces by close to a factor of 10 in test scenario (starting editor for project F)
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-2/+2
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-124-997/+0
|
* Add multithreading directory scanning in core/filesystem (#277)Dan Engelbrecht2025-01-226-71/+51
| | | | | | add DirectoryContent::IncludeFileSizes add DirectoryContent::IncludeAttributes add multithreaded GetDirectoryContent use multithreaded GetDirectoryContent in workspace folder scanning
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-166-6/+6
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-6/+16
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* Miscellaneous minor LLM fixes (#268)v5.5.17-pre0Stefan Boberg2024-12-172-1/+4
| | | | | | | With this change, LLM tags are assigned using the name,parent tuple rather than just by name only. This allows tag hierarchies like `cache/store` and `project/store` which would previously get collapsed into the first pair seen when registering the `store` tag. This PR also adds some more LLM tag annotations to more accurately associate memory allocations with subsystems In addition, this PR also tweaks the frequency of timer marker events to increase the resolution in Insights and avoid some cases of Insights deciding that marker events are too far apart since we don't allocate as frequently as UE tends to.
* batch fetch record cache values (#266)Dan Engelbrecht2024-12-173-21/+90
| | | | | | - Improvement: Batch fetch record attachments when appropriate - Improvement: Reduce memory buffer allocation in BlockStore::IterateBlock - Improvement: Tweaked BlockStore::IterateBlock logic when to use threaded work (at least 4 chunks requested) - Bugfix: CasContainerStrategy::IterateChunks could give wrong payload/index when requesting 1 or 2 chunks
* more memory tagging and fixes (#263)Stefan Boberg2024-12-161-0/+16
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* Builds API remote project store (#258)Dan Engelbrecht2024-12-121-2/+2
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* Memory tracking improvements (#262)Stefan Boberg2024-12-111-1/+0
| | | | | * added LLM tag to properly tag RPC allocations * annotated some more httpsys functions with memory tags * only emit memory scope events if the active tag is different from the new tag
* projectstore getchunks rpc with modtag (#244)Dan Engelbrecht2024-12-051-15/+17
| | | Feature: Project store "getchunks" rpc call /prj/{project}/oplog/{log}/rpc extended to accept both CAS (RawHash) and Id (Oid) identifiers as well as partial ranges
* Unity build fixes (#253)Stefan Boberg2024-12-054-63/+76
| | | some fixes to make everything build using unity build mode. Mostly moved code from anonymous namespaces into local impl namespace to avoid ambiguity in name resolution.
* ODR violation fixStefan Boberg2024-12-031-2/+2
|
* reduce memory churn (#248)Stefan Boberg2024-12-021-5/+5
| | | | | * eliminated allocation in SetCurrentThreadName * reduced memory allocator activity in cache RPC response building * reduced allocations in compact binary building
* validate that root path exists for workspace before adding it (#250)Dan Engelbrecht2024-12-021-0/+5
|
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-028-0/+191
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-3/+17
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* use plain sorted array instead of map of vectors (#237)Dan Engelbrecht2024-11-272-22/+33
| | | | | * use plain sorted array instead of map of vectors * reserve vectors up front = 5% perf increase * don't do batch read of chunks if we have a single chunk -> 1% perf gain
* add missing shard lock in filecas compact (#239)Dan Engelbrecht2024-11-271-1/+6
| | | | * add missing shardlock during compact in filecas * add warning log when filecas fails to open a file it expects to be present
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-259-18/+31
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-23 09:31:37 +0000