aboutsummaryrefslogtreecommitdiff
path: root/src/zenstore/gc.cpp
Commit message (Collapse)AuthorAgeFilesLines
* long filename support (#330)Dan Engelbrecht2025-03-311-10/+10
| | | - Bugfix: Long file paths now works correctly on Windows
* zen build cache service (#318)Dan Engelbrecht2025-03-261-2/+19
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* Suppress progress report callback if oplog import detects zero op oplog (#271)Dan Engelbrecht2025-01-131-6/+16
| | | | * Suppress progress report callback if oplog import detects oplog with zero ops * output error code when catching system errors
* more memory tagging and fixes (#263)Stefan Boberg2024-12-161-0/+16
| | | This change adds more instrumentation for memory tracking, so that as little as possible comes through as Unknown in Insights analysis.
* ODR violation fixStefan Boberg2024-12-031-2/+2
|
* added support for dynamic LLM tags (#245)Stefan Boberg2024-12-021-0/+30
| | | | | * added FLLMTag which can be used to register memory tags outside of core * changed `UE_MEMSCOPE` -> `ZEN_MEMSCOPE` for consistency * instrumented some subsystems with dynamic tags
* add missing projectstore expire time in gc log (#227)Dan Engelbrecht2024-11-251-0/+1
|
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-151-185/+415
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* Use a smaller thread pool during pre-cache phase of GC to reduce memory ↵Dan Engelbrecht2024-10-221-7/+11
| | | | pressure (#205)
* remove gc v1 (#121)Dan Engelbrecht2024-10-031-506/+19
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* Porject -> ProjectStefan Boberg2024-10-021-1/+1
|
* optimize gc reference sort (#179)Dan Engelbrecht2024-10-011-33/+37
| | | | | - Do a single call to mempcy when fetching attachments from the meta store in GC - Use small lambda when calling std::sort in FilterReferences (enables inlining of the comparision function) - Use a single function for < and == comparision in KeepUnusedReferences
* use alternate IoHash comparision function (#177)v5.5.8-pre5Dan Engelbrecht2024-09-301-4/+23
| | | * Use alternate IoHash comparision function - reduces KeepUnusedReferences execution time by ~20%
* gc command attachment options (#176)Dan Engelbrecht2024-09-301-7/+30
| | | * zen command - add options to control meta data cache when triggering gc
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-36/+150
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* gc unused refactor (#165)Dan Engelbrecht2024-09-231-16/+242
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-1/+1
| | | - Improvement: Move GC logging in callback functions into "gc" context
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* if disk space is low, set the last gc time to avoid spamming retries (#124)Dan Engelbrecht2024-08-191-0/+2
| | | * if disk space is low, set the last gc time to avoid spamming retries
* improved logging removing unimportant information (#116)Dan Engelbrecht2024-08-141-24/+33
|
* hardening and reduced spam from GC on failure (#112)Dan Engelbrecht2024-08-141-141/+229
| | | | * Retry writing GC state if it fails to handle transient problems * If GC operation fails demote errors to warnings on consecutive fails
* add gc single threaded option (#104)Dan Engelbrecht2024-08-071-4/+9
| | | * add option to force gcv2 to run single threaded
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-131-39/+85
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-1/+1
| | | * use write and move in place for safer writing of files
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-1/+19
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* improved assert (#37)Dan Engelbrecht2024-04-041-16/+16
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* Make sure we wait for all scheduled tasks to complete before throwing ↵Dan Engelbrecht2024-02-281-48/+88
| | | | | exceptions further (#662) Bugfix: We must not throw exceptions to calling function until all async work we spawned has returned
* Don't capture local variables in loop by reference (#623)Dan Engelbrecht2023-12-191-27/+27
| | | * Don't capture local variables in loop by reference
* fix peak disk load in gc status (#608)Dan Engelbrecht2023-12-131-12/+11
| | | * MaxLoad is max load per monitor slot, not the MaxLoad for the entire graph
* improved scrubbing of oplogs and filecas (#596)Stefan Boberg2023-12-111-3/+10
| | | | | | - Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks) - Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command - Improvement: Implemented oplog scrubbing (previously was a no-op) - Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
* add separate PreCache step for GcReferenceChecker (#578)Dan Engelbrecht2023-12-011-6/+54
| | | | | | - Improvement: GCv2: Use separate PreCache step to improve concurrency when checking references - Improvement: GCv2: Improved verbose logging - Improvement: GCv2: Sort chunks to read by block/offset when finding references - Improvement: GCv2: Exit as soon as no more unreferenced items are left
* global thread worker pools (#577)Dan Engelbrecht2023-11-291-10/+4
| | | - Improvement: Use two global worker thread pools instead of ad-hoc creation of worker pools
* tracing for gcv2 (#574)Dan Engelbrecht2023-11-281-1/+11
| | | | | | - Improvement: Added more trace scopes for GCv2 - Bugfix: Make sure we can override flags to "false" when running `zen gc` commmand - `smallobjects`, `skipcid`, `skipdelete`, `verbose`
* gc stop command (#569)v0.2.36-pre2Dan Engelbrecht2023-11-271-1/+16
| | | | | - Feature: New endpoint `/admin/gc-stop` to cancel a running garbage collect operation - Feature: Added `zen gc-stop` command to cancel a running garbage collect operation - Bugfix: GCv2 - make sure to discover all projects and oplogs before checking for expired data
* Add GC Cancel/Stop (#568)Dan Engelbrecht2023-11-241-6/+97
| | | | - GcScheduler will now cancel any running GC when it shuts down. - Old GC is rather limited in *when* it reacts to cancel of GC. GCv2 is more responsive.
* add command line options for compact block threshold and gc verbose (#557)Dan Engelbrecht2023-11-211-146/+210
| | | | | | | | | | | - Feature: Added new options to zenserver for GC V2 - `--gc-compactblock-threshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `--gc-verbose` GCV2 - enable more verbose output when running a GC pass - Feature: Added new options to `zen gc` command for GC V2 - `--compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `--verbose` GCV2 - enable more verbose output when running a GC pass - Feature: Added new parameters for endpoint `admin/gc` (PUT) - `compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90% - `verbose` GCV2 - enable more verbose output when running a GC pass
* compact separate for gc referencer (#533)Dan Engelbrecht2023-11-211-396/+425
| | | | | - Refactor GCV2 so GcReferencer::RemoveExpiredData returns a store compactor, moving out the actual disk work from deleting items in the index. - Refactor GCV2 GcResult to reuse GcCompactStoreStats and GcStats - Make Compacting of stores non-parallell to not eat all the disk I/O when running GC
* blocking queue fix (#550)Dan Engelbrecht2023-11-161-1/+1
| | | | | | | | | * make BlockingQueue::m_CompleteAdding non-atomic * ZenCacheDiskLayer::Flush logging * name worker threads in ZenCacheDiskLayer::DiscoverBuckets * name worker threads in gcv2 * improved logging in ZenServerInstance * scrub threadpool naming * remove waitpid handling, we should just call wait to kill zombie processes
* gc history log (#519)Dan Engelbrecht2023-11-131-8/+279
| | | | | - Feature: Writes a `gc.log` with settings and detailed result after each GC execution (version 2 only) - Break out file name rotate to allow access for gclog - CompactBinaryToJson(MemoryView Data, StringBuilderBase& InBuilder)
* gc v2 tests (#512)Dan Engelbrecht2023-11-061-27/+11
| | | | | | | | | | * set MaxBlockCount at init * properly calculate total size * basic blockstore compact blocks test * correct detection of block swap * Use one implementation for CreateRandomBlob * reduce some data sets to increase speed of tests * reduce test time * rename BlockStoreCompactState::AddBlock -> BlockStoreCompactState::IncludeBlock
* multithread cache bucket (#508)Dan Engelbrecht2023-11-061-252/+252
| | | | * Multithread init and flush of cache bucket * tweaked threading cound for bucket discovery, disklayer flush and gc v2
* individual gc stats (#506)Dan Engelbrecht2023-10-301-265/+366
| | | | | - Feature: New parameter for endpoint `admin/gc` (GET) `details=true` which gives details stats on GC operation when using GC V2 - Feature: New options for zen command `gc-status` - `--details` that enables the detailed output from the last GC operation when using GC V2
* New GC implementation (#459)Dan Engelbrecht2023-10-301-18/+308
| | | - Feature: New garbage collection implementation, still in evaluation mode. Enabled by `--gc-v2` command line option
* added missing includes (#504)Stefan Boberg2023-10-271-0/+1
| | | | | this change adds some includes to files which "inherit" includes from elsewhere this was exposed on another branch when removing some heavy dependencies from central headers
* fix m_LastFullGcDuration, m_LastFullGCDiff, m_LastFullGcDuration and ↵Dan Engelbrecht2023-10-231-18/+13
| | | | m_LastLightweightGcDuration stats (#494)
* clean up GcContributor and GcStorage to be pure interfaces (#485)Dan Engelbrecht2023-10-201-24/+0
|
* Add --skip-delete option to gc command (#484)Dan Engelbrecht2023-10-201-0/+4
| | | | - Feature: Add `--skip-delete` option to gc command - Bugfix: Fix implementation when claiming GC reserve during GC
* add `flush` command and more gc status info (#483)Dan Engelbrecht2023-10-181-23/+89
| | | | | | - Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore - Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore - Improved: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc - Changed: Cache access and write log are disabled by default
* skip lightweight GC if full GC is due soon (#467)Stefan Boberg2023-10-121-20/+30
| | | | | | GC will now skip a lightweight GC if a full GC is due to run within the next lightweight GC interval also fixed some minor typos
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 03:48:23 +0000