| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
| |
- Improvement: Refactor memory cache for faster trimming and correct trim reporting
- Improvement: Added trace scopes for memory cache trimming
Adding a link back to the cache item payload on the memory cache item allows us to iterate over only the items cached in memory instead of over the entire index.
This also allows us to do efficient compact of the memory cache array when trimming.
It adds 4 bytes of overhead to each item cached in memory.
|
| |
|
|
|
|
| |
- Make sure we don't hold the namespace bucket lock when we create buckets to avoid deadlock
- Pass lock scope to helper functions to clarify locking rules
- Block flush and gc operations for a bucket that is not yet initialized
- Add ZenCacheDiskLayer::GetOrCreateBucket to avoid code duplication
|
| |
|
| |
- Improvement: Reduce memory usage in GC and diskbucket flush
|
| |
|
|
|
|
| |
- Improvement: GCv2: Use separate PreCache step to improve concurrency when checking references
- Improvement: GCv2: Improved verbose logging
- Improvement: GCv2: Sort chunks to read by block/offset when finding references
- Improvement: GCv2: Exit as soon as no more unreferenced items are left
|
| |
|
|
|
| |
the previous implementation of in-memory index snapshots serialise data to memory before writing to disk and vice versa when reading. This leads to some memory spikes which end up pushing useful data out of system cache and also cause stalls on I/O operations.
this change moves more code to a streaming serialisation approach which scales better from a memory usage perspective and also performs much better
|
| |
|
|
|
| |
- Refactor GCV2 so GcReferencer::RemoveExpiredData returns a store compactor, moving out the actual disk work from deleting items in the index.
- Refactor GCV2 GcResult to reuse GcCompactStoreStats and GcStats
- Make Compacting of stores non-parallell to not eat all the disk I/O when running GC
|
| |
|
| |
* proper != operator for DiskLocation
|
| |
|
|
| |
* don't touch non-locked data when creating manifest
* safety assert for test dir
|
| |
|
| |
reduces memory footprint of cache index by 10% or so by limiting the maximum number of entries in a bucket to 2^32 (was 2^64)
|
| |
|
|
|
|
|
| |
- Improvement: Use GC reserve when writing index/manifest for a disk cache bucket when disk is low when available
- Improvement: Demote errors to warning for issues that are not critical and we handle gracefully
- Improvement: Treat more out of memory errors from windows as Out Of Memory errors
Fixed wrong sizeof() statement for compactcas index (luckily the two structs are of same size)
|
| |
|
|
|
|
|
| |
* reduce memory footprint for disk cache separate dense arrays for rawhash+rawsize and memcache buffer
* don't write RawHash/RawSize for buckets with no such metadata
* helper functions
* make index into metadata and cached payload type safe
* helper functions for memcached
|
| |
|
|
|
| |
- Feature: New parameter for endpoint `admin/gc` (GET) `details=true` which gives details stats on GC operation when using GC V2
- Feature: New options for zen command `gc-status`
- `--details` that enables the detailed output from the last GC operation when using GC V2
|
| |
|
| |
- Feature: New garbage collection implementation, still in evaluation mode. Enabled by `--gc-v2` command line option
|
| |
|
|
|
| |
this change adds some includes to files which "inherit" includes from elsewhere
this was exposed on another branch when removing some heavy dependencies from central headers
|
| |
|
|
| |
- Feature: Added `--cache-memlayer-sizethreshold` option to zenserver to control at which size cache entries get cached in memory
- Changed: Merged cache memory layer with cache disk layer to reduce memory and cpu overhead
|
| |
|
|
|
| |
- Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC
- disabled by default, enable with `--cache-reference-cache-enabled`
- Improvement: Faster collection of referenced CId content in project store
|
| |
|
|
|
|
|
|
|
| |
* make sure we hold the index lock when reading payload data in reclaim space
* don't use index snapshot when updating index in reclaim space
* check that things have not moved under our feet
* don't touch m_Payloads without a lock
* start write block index on the highest block index
* we don't need to bump writeblockindex when stopping write to a block, we will bump appropriately when we start a new block
* changelog
|
| |
|
|
|
|
|
|
|
|
| |
- Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting
- Change format for faster read/write and reduced size on disk
- Don't lock index while writing manifest to disk
- Skip garbage collect if we are currently in a Flush operation
- BlockStore::Flush no longer terminates currently writing block
- Garbage collect references to currently writing block but keep the block as new data may be added
- Fix BlockStore::Prune used disk space calculation
- Don't materialize data in filecas when we just need the size
|
| |
|
|
|
| |
- Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable
- Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable
- Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
|
| |
|
| |
currently, only Windows (using Projected File System) is supported
|
|
|
This change separates out the disk and memory storage strategies into separate cpp/h files to improve maintainability.
|