aboutsummaryrefslogtreecommitdiff
path: root/zenserver/cache/structuredcachestore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-3648/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* save cache rawsize and rawhash in manifest file instead of log file (#251)Dan Engelbrecht2023-04-211-230/+90
| | | | * save cache rawsize and rawhash in manifest file instead of log file * don't use # as prefix for counts in log
* oplog and cache stats (#244)Dan Engelbrecht2023-04-211-5/+95
| | | | | | | | | | | | | | | | * basic oplog stats * add GetValueStats to cache store * RwLock::ExclusiveLockScope -> RwLock::SharedLockScope * add rawhash and attachment count to CacheValueStats * added cache-stats and project-stats commands * add cast to make Mac overload detection happy * fix accept type in cache-stats command * Add options to project-stats command * use resource paths for stats in project store * use resource paths for stats in cache * fix cache-info and project-info url discriminator * more control over details$ output * cleanup * changelog
* store cache rawhash and rawsize for unstructured cache values (#234)Dan Engelbrecht2023-02-231-132/+366
| | | | | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize. * Use CompressedBuffer::ValidateCompressedHeader when applicable * Make sure we rewrite the snapshot if we read an legacy existing index/log * changelog
* Refactor CacheBuckets to allow for storing RawHash/RawSize (#232)v0.2.4-pre0v0.2.4Dan Engelbrecht2023-02-201-56/+193
| | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize.
* Add info (GET) endpoints for structured cache (#211)Dan Engelbrecht2023-01-111-5/+154
| | | | | | | * Add GET requests on cache/namespace/bucket level * Add root route for project store requests (same as /list) * Add markerpath to oplog info object * Add totalsize, opcount and expired to oplog info * Changelog
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-161-12/+17
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-121-3/+2
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* Fix logging of number of read entries from log/index file for stores (#204)Dan Engelbrecht2022-12-081-5/+4
| | | | * Fix logging of number of read entries from log/index file for stores * changelog
* optimizations (#200)Dan Engelbrecht2022-12-071-22/+25
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Cache request record/replay (#198)Dan Engelbrecht2022-12-071-8/+25
| | | | | | | | | This adds recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request. `<host>/z$/exec$/start-recording?<disk-storage-path>` `<host>/z$/exec$/stop-recording` `<host>/z$/exec$/replay-recording?<thread-count>&<disk-storage-path>`
* Size based gc trigger (#197)Dan Engelbrecht2022-12-021-18/+10
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-021-17/+17
|
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-241-45/+49
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* Support file reference in package message (#184)Dan Engelbrecht2022-11-071-1/+1
| | | | | | | | * Fix packed message parsing for absolute path * Always enable are sharing when opening files as IoBuffers. * Allow control over sending partial files as localfile ref * Check "AcceptFlags" field in RPC message for allowing localfile ref in reply * make oplog entry add operations ZEN_DEBUG level logs * changelog
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-081-1/+1
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Use "\\?\" prefixed paths and fix hardcoded path delimiters (#149)Dan Engelbrecht2022-08-261-1/+3
| | | | | | * use "\\?\" prefix for windows paths * fix path delimiters * disable vcpkg caching * Workaround for spdlog not being able to create directories prefixed with `\\?\`
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-171-549/+179
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* merged from mainStefan Boberg2022-06-161-24/+116
|
* fixed mac build ("unused" variable)Stefan Boberg2022-06-111-0/+2
|
* Use m_BucketDir for cache key contextDan Engelbrecht2022-06-081-2/+2
| | | | Easier to debug and no conflicts
* We need to make each Cache Key contribution per bucket unique even across ↵Dan Engelbrecht2022-06-071-2/+2
| | | | namespaces
* Don't hold index lock while reading standalone valuesDan Engelbrecht2022-06-031-0/+2
| | | | Can cause deadlock
* hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValueDan Engelbrecht2022-06-031-13/+36
|
* add comment about removing filesDan Engelbrecht2022-05-301-0/+1
|
* Enable FILE_SHARE_DELETE on standalone files in disk bucketsDan Engelbrecht2022-05-281-107/+77
| | | | | | | | This allows us to delete the file even if it is open for read. We do a delete, the rename since we are not allowed to do a rename-overwrite, only delete. As we have the shard lock for the file we want to replace we can safely do a delete+rename. In the rare case that we fail to rename the file into place the old data is lost. As this is a *cache* and it should be very rare this is OK.
* bugfixes and test for namespace dropDan Engelbrecht2022-05-251-15/+92
|
* clean up namespace foldersDan Engelbrecht2022-05-251-36/+42
|
* namespace dropDan Engelbrecht2022-05-251-1/+24
|
* Make sure ZenCacheMemoryLayer handles dropped buckets correctly (just like ↵Dan Engelbrecht2022-05-251-32/+87
| | | | ZenCacheDiskLayer)
* dropIndex -> DropIndexDan Engelbrecht2022-05-251-3/+3
|
* If a bucket is in m_BucketMap it is OK, no need for separate flagDan Engelbrecht2022-05-251-56/+21
|
* drop bucket testDan Engelbrecht2022-05-241-0/+73
|
* Use rename/delete and keep pointer for dropped bucketsDan Engelbrecht2022-05-241-53/+104
|
* Make sure to hold exclusive lock over index and all shard locks.Dan Engelbrecht2022-05-241-1/+9
| | | | Clear index on drop.
* Merge pull request #102 from EpicGames/de/auto-create-namespacesDan Engelbrecht2022-05-241-19/+52
|\ | | | | Automatically create namespaces on requests (if enabled via configuration)
| * Automatically create namespaces on requests (if enabled via configuration)Dan Engelbrecht2022-05-201-19/+52
| |
* | De/fix namespace folder scanning (#103)Dan Engelbrecht2022-05-231-1/+1
|/
* Merge pull request #99 from EpicGames/de/move-namespace-fieldDan Engelbrecht2022-05-191-5/+19
|\ | | | | Keep Namespace out of CacheKey and store it on request level
| * migrate legacy cache folders to ue4.ddc namespaceDan Engelbrecht2022-05-191-5/+19
| | | | | | | | | | map default namespace to at runtime ue4.ddc use a non-valid name for the default namespace so we avoid any collision or accidental creation of folder for that
* | Fix and retry count and add an extra iteration to give more time for successDan Engelbrecht2022-05-191-5/+5
|/
* fix release build, misplaced namespace bracketsDan Engelbrecht2022-05-171-4/+0
|
* use ns_ prefix on disk onlyDan Engelbrecht2022-05-161-4/+6
|
* Tests for HttpRequestParseRelativeUriDan Engelbrecht2022-05-121-6/+5
|
* Add namespace testDan Engelbrecht2022-05-111-0/+53
|
* Make sure we clean up temp file in all scenariosDan Engelbrecht2022-05-101-8/+20
|
* happy path should be minimal workDan Engelbrecht2022-05-091-23/+23
|
* make test run on more platformsDan Engelbrecht2022-05-091-7/+7
|
* Restore logic where we accept failed overwrite if resulting size is the sameDan Engelbrecht2022-05-091-3/+44
| | | | Correctly calculate the m_TotalSize difference when overwriting file
* Make sure CacheBucket::PutStandaloneCacheValue cleans up the temp file if we ↵Dan Engelbrecht2022-05-091-0/+48
| | | | fail to move the it into place
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 02:06:29 +0000