aboutsummaryrefslogtreecommitdiff
path: root/zenserver/cache
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-026-7946/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* save cache rawsize and rawhash in manifest file instead of log file (#251)Dan Engelbrecht2023-04-212-235/+92
| | | | * save cache rawsize and rawhash in manifest file instead of log file * don't use # as prefix for counts in log
* oplog and cache stats (#244)Dan Engelbrecht2023-04-214-35/+397
| | | | | | | | | | | | | | | | * basic oplog stats * add GetValueStats to cache store * RwLock::ExclusiveLockScope -> RwLock::SharedLockScope * add rawhash and attachment count to CacheValueStats * added cache-stats and project-stats commands * add cast to make Mac overload detection happy * fix accept type in cache-stats command * Add options to project-stats command * use resource paths for stats in project store * use resource paths for stats in cache * fix cache-info and project-info url discriminator * more control over details$ output * cleanup * changelog
* send payloads as duplicated handles (#240)Dan Engelbrecht2023-03-142-18/+33
| | | | | | | | | * send payloads as duplicated handles if requestor provides process id and allows local file references. * linux/macos fixes * tests * fix access rights when duplicating handle * fix closing of duplicated handles on error * cleanup * changelog
* store cache rawhash and rawsize for unstructured cache values (#234)Dan Engelbrecht2023-02-233-170/+436
| | | | | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize. * Use CompressedBuffer::ValidateCompressedHeader when applicable * Make sure we rewrite the snapshot if we read an legacy existing index/log * changelog
* Refactor CacheBuckets to allow for storing RawHash/RawSize (#232)v0.2.4-pre0v0.2.4Dan Engelbrecht2023-02-202-98/+258
| | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize.
* Enhanced rpc request recording (#229)Dan Engelbrecht2023-02-172-262/+61
| | | | | | | | | | | | | * rpc replay zen command * fix replay sessions for thread * recording start/stop as zen commands * move rpcrecording code to zenutil to remove code duplication * simplify recording http request threading * added more data logging to rpc replay * NotFound is an acceptable response for an rpc request * fix rpc replay command line parsing * rpc replay stats * Allow spawning of sub-process workers when replaying rpc recording * changelog
* Add info (GET) endpoints for structured cache (#211)Dan Engelbrecht2023-01-114-57/+406
| | | | | | | * Add GET requests on cache/namespace/bucket level * Add root route for project store requests (same as /list) * Add markerpath to oplog info object * Add totalsize, opcount and expired to oplog info * Changelog
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-163-12/+19
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-121-3/+2
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* Path from handle perf improvement (#206)Dan Engelbrecht2022-12-081-8/+12
| | | | | | * Read recorded requests to memory before parsing This will more accurately simulate how requests comes in from a client * Make a fast path for GetFinalPathNameByHandleW Try to get the path with a fixes size buffer first to avoid always doing two calls to GetFinalPathNameByHandleW
* Fix logging of number of read entries from log/index file for stores (#204)Dan Engelbrecht2022-12-081-5/+4
| | | | * Fix logging of number of read entries from log/index file for stores * changelog
* optimizations (#200)Dan Engelbrecht2022-12-072-77/+91
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Cache request record/replay (#198)Dan Engelbrecht2022-12-073-213/+457
| | | | | | | | | This adds recording and playback of cache request with full data - both get and put operations can be replayed. Invoke via web request. `<host>/z$/exec$/start-recording?<disk-storage-path>` `<host>/z$/exec$/stop-recording` `<host>/z$/exec$/replay-recording?<thread-count>&<disk-storage-path>`
* Size based gc trigger (#197)Dan Engelbrecht2022-12-021-18/+10
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-021-17/+17
|
* Make sure we always store record/op before attachments (#195)Dan Engelbrecht2022-12-011-94/+137
| | | | | | | | * Make sure we always store record/op before attachments We don't want to store attachments first - a GC operation could then remove attachments if triggered before storing record/op * zen::Latch * Use latch to wait for attachments to be stored * use zen::latch when adding attachments from project oplog import * changelog
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-242-51/+53
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* bump cache large object size limit to 128Kb (#191)Dan Engelbrecht2022-11-211-1/+1
| | | | * bump cache large object size limit to 128Kb * changelog
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-181-6/+9
| | | * Add `import-project` and `export-project` command line parsing
* move BasicFile to zenutil to remove zenstore dependency from zen command (#190)Dan Engelbrecht2022-11-171-1/+0
|
* Support file reference in package message (#184)Dan Engelbrecht2022-11-073-9/+29
| | | | | | | | * Fix packed message parsing for absolute path * Always enable are sharing when opening files as IoBuffers. * Allow control over sending partial files as localfile ref * Check "AcceptFlags" field in RPC message for allowing localfile ref in reply * make oplog entry add operations ZEN_DEBUG level logs * changelog
* Add "Accept" field in RPC request to gracefully handle requests from older ↵Dan Engelbrecht2022-10-132-23/+101
| | | | instances (#180)
* Format all rpc package responses using `FormatPackageMessageBuffer` to avoid ↵Dan Engelbrecht2022-09-291-26/+10
| | | | | memory copy (#174) When reading upstream, fall back to old rpc response to handle older instances.
* De/more upstream details (#168)v0.1.6-pre9v0.1.6-pre10Dan Engelbrecht2022-09-221-93/+166
| | | | | | * Pass along endpoint info for each upstream request * Add more timing details in log * more log details for single item upstream fetch * DISABLEDQUERY over SKIP
* rename URI chunk requests from value -> chunk (#166)Dan Engelbrecht2022-09-192-10/+10
|
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-081-1/+1
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Remove legacy code (#161)Dan Engelbrecht2022-09-081-12/+10
| | | | * changelog * remove obsolete legacy code
* Implement proper GetCacheValues upstream (#155)Dan Engelbrecht2022-09-061-17/+34
| | | | * Implement proper GetCacheValues upstream * changelog
* Use "\\?\" prefixed paths and fix hardcoded path delimiters (#149)Dan Engelbrecht2022-08-261-1/+3
| | | | | | * use "\\?\" prefix for windows paths * fix path delimiters * disable vcpkg caching * Workaround for spdlog not being able to create directories prefixed with `\\?\`
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-173-563/+193
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* merged from mainStefan Boberg2022-06-161-24/+116
|
* removed unnecessary CasStore forward declarationsStefan Boberg2022-06-132-2/+0
|
* Merge branch 'main' of https://github.com/EpicGames/zenStefan Boberg2022-06-111-5/+7
|\
| * namespace/bucket validation now uses AsciiSet for more efficient validation ↵Stefan Boberg2022-06-101-5/+7
| | | | | | | | (#127)
* | fixed mac build ("unused" variable)Stefan Boberg2022-06-111-0/+2
|/
* Corrected some typos in log messagesStefan Boberg2022-06-081-3/+5
|
* Merge branch 'main' into de/add-log-for-invalid-namesDan Engelbrecht2022-06-081-2/+2
|\
| * Use m_BucketDir for cache key contextDan Engelbrecht2022-06-081-2/+2
| | | | | | | | Easier to debug and no conflicts
| * We need to make each Cache Key contribution per bucket unique even across ↵Dan Engelbrecht2022-06-071-2/+2
| | | | | | | | namespaces
* | add warnings on illegal namespace/bucket namesDan Engelbrecht2022-06-081-0/+5
|/
* Don't hold index lock while reading standalone valuesDan Engelbrecht2022-06-031-0/+2
| | | | Can cause deadlock
* hardening of ZenCacheDiskLayer::CacheBucket::PutStandaloneCacheValueDan Engelbrecht2022-06-031-13/+36
|
* add comment about removing filesDan Engelbrecht2022-05-301-0/+1
|
* Enable FILE_SHARE_DELETE on standalone files in disk bucketsDan Engelbrecht2022-05-281-107/+77
| | | | | | | | This allows us to delete the file even if it is open for read. We do a delete, the rename since we are not allowed to do a rename-overwrite, only delete. As we have the shard lock for the file we want to replace we can safely do a delete+rename. In the rare case that we fail to rename the file into place the old data is lost. As this is a *cache* and it should be very rare this is OK.
* NameSpaceMap -> NamespaceMapDan Engelbrecht2022-05-251-2/+2
|
* bugfixes and test for namespace dropDan Engelbrecht2022-05-251-15/+92
|
* clean up namespace foldersDan Engelbrecht2022-05-251-36/+42
|
* namespace dropDan Engelbrecht2022-05-253-12/+38
|
* Make sure ZenCacheMemoryLayer handles dropped buckets correctly (just like ↵Dan Engelbrecht2022-05-252-36/+95
| | | | ZenCacheDiskLayer)
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 00:57:12 +0000