aboutsummaryrefslogtreecommitdiff
path: root/zenstore
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-0220-7371/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* made Ref<> constructor explicit (#262)Stefan Boberg2023-04-271-4/+4
| | | This change makes the Ref<> constructor explicit, which can help avoid unnecessary overheads and other accidents
* bugfixes (#261)Dan Engelbrecht2023-04-271-0/+5
| | | | | * Don't try to GC if now blocks to GC was found, regardless if we have locations in memory * Don't let exception leak from scope guard as throw in destructor will abort application * changelog
* Safely handle missing blocks when doing garbage collection in block store ↵Dan Engelbrecht2023-04-261-5/+18
| | | | data (#259)
* save cache rawsize and rawhash in manifest file instead of log file (#251)Dan Engelbrecht2023-04-213-12/+12
| | | | * save cache rawsize and rawhash in manifest file instead of log file * don't use # as prefix for counts in log
* oplog and cache stats (#244)Dan Engelbrecht2023-04-212-3/+3
| | | | | | | | | | | | | | | | * basic oplog stats * add GetValueStats to cache store * RwLock::ExclusiveLockScope -> RwLock::SharedLockScope * add rawhash and attachment count to CacheValueStats * added cache-stats and project-stats commands * add cast to make Mac overload detection happy * fix accept type in cache-stats command * Add options to project-stats command * use resource paths for stats in project store * use resource paths for stats in cache * fix cache-info and project-info url discriminator * more control over details$ output * cleanup * changelog
* oops: clang-formatStefan Boberg2023-04-191-8/+8
|
* put forcelink calls into ZEN_WITH_TESTS blocksStefan Boberg2023-04-191-0/+4
|
* tweaks for enabling unity buildsStefan Boberg2023-04-194-19/+47
| | | | | | | | | | mostly changes to make sure anonymous namespaces don't clash and a change to avoid windows headers from leaking into other compilation units unity builds are not yet enabled by default, but can be enabled by uncommenting this line in the root `xmake.lua` ``` --add_rules("c++.unity_build") ```
* make sure initialization of a new filecas dont remove the cas manifest file ↵Dan Engelbrecht2023-04-191-1/+37
| | | | | | or tiny/small cas store folders (#246) * make sure initialization of a new filecas doesnt remove the cas manifest file or tiny/small cas store folders * changelog
* store cache rawhash and rawsize for unstructured cache values (#234)Dan Engelbrecht2023-02-232-9/+53
| | | | | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize. * Use CompressedBuffer::ValidateCompressedHeader when applicable * Make sure we rewrite the snapshot if we read an legacy existing index/log * changelog
* Make sure to add any scanned files to the cas registry (#230)Dan Engelbrecht2023-02-151-0/+1
|
* FileCas (#226)Dan Engelbrecht2023-02-132-133/+449
| | | | | * maintain snapshot of disk state in file cas * Add folder scanning to establish initial state for filecas and pre-scrubbing * changelog
* oplog upload/download (#214)Dan Engelbrecht2023-02-091-0/+1
| | | | | | | | | | | | - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint.
* fix gc logging (#213)Dan Engelbrecht2023-01-131-22/+31
| | | | | | * Don't output time to next GC if time is "infinite". * Do immediate check of GC status on thread startup instead of waiting montior intervall first. * set up reasonable gc defaults * changelog
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-162-14/+18
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-121-3/+2
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* Fix logging of number of read entries from log/index file for stores (#204)Dan Engelbrecht2022-12-081-2/+2
| | | | * Fix logging of number of read entries from log/index file for stores * changelog
* optimizations (#200)Dan Engelbrecht2022-12-075-19/+28
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Size based gc trigger (#197)Dan Engelbrecht2022-12-024-111/+511
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-023-23/+23
|
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-247-117/+255
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* Don't resize block store block file to max size at creation (#193)Dan Engelbrecht2022-11-231-6/+7
| | | | * Don't resize block store block file to max size at creation * changelog
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-186-19/+45
| | | * Add `import-project` and `export-project` command line parsing
* move BasicFile to zenutil to remove zenstore dependency from zen command (#190)Dan Engelbrecht2022-11-177-695/+5
|
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-083-3/+3
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Remove legacy code (#161)Dan Engelbrecht2022-09-082-221/+0
| | | | * changelog * remove obsolete legacy code
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-1717-1269/+620
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* merged from mainStefan Boberg2022-06-163-111/+195
|
* removed more unnecessary CasStore related codeStefan Boberg2022-06-133-42/+0
|
* cidstore: propagate the correct content typeStefan Boberg2022-06-101-0/+2
|
* gc: improved message indicating no GC is scheduledStefan Boberg2022-06-101-1/+1
|
* We need to make each Cache Key contribution per bucket unique even across ↵Dan Engelbrecht2022-06-072-14/+8
| | | | namespaces
* Always block GC of current write blockDan Engelbrecht2022-05-311-0/+4
|
* Make sure we can properly create the block file before assigning it for useDan Engelbrecht2022-05-271-14/+18
|
* De/fix namespace folder scanning (#103)Dan Engelbrecht2022-05-231-1/+2
|
* Add catch2 support (#101)Stefan Boberg2022-05-203-7/+7
| | | | | | | Added option to use catch2 for unit tests Currently both doctest and catch2 are supported via some compatibility macros. doctest is the default, and ZEN_USE_CATCH2 needs to be defined to switch to catch2. Our goal is to evaluate how well catch2 works and switch to catch2 if everything pans out since UE5 now supports using catch2 for unit tests.
* Merge pull request #89 from EpicGames/de/namespacesv1.0.1.5Dan Engelbrecht2022-05-091-30/+10
|\ | | | | Add namespacecachestore layer to allow multiple structured cache namespaces
| * Added GetDirectoryContent utilityDan Engelbrecht2022-05-061-30/+10
| |
* | Make sure blockstore owner and block store state does not get out of sync ↵Dan Engelbrecht2022-05-085-94/+38
|/ | | | | | when fetching a chunk Move MarkAsDeleteOnClose() to IoBuffer(ExtendedCore) and set it on close, SetFileInformationByHandle sometimes fails if done in parallel with FileMapping
* handle that more than one block can be written to in parallelDan Engelbrecht2022-05-033-36/+45
|
* unused variable in test fixDan Engelbrecht2022-05-031-4/+4
|
* threading test for blockstoreDan Engelbrecht2022-05-031-1/+91
|
* more tests for block storeDan Engelbrecht2022-05-033-17/+334
|
* cleanupDan Engelbrecht2022-05-032-5/+1
|
* switched back memory_order for m_TotalSize to relaxedDan Engelbrecht2022-05-021-2/+2
|
* clean up any incomplete blocks if ReclaimSpace failsDan Engelbrecht2022-05-021-136/+153
|
* Make sure we close all block files when dropping a cache bucketDan Engelbrecht2022-05-022-0/+12
|
* simplify lambda capturesDan Engelbrecht2022-05-024-37/+23
|
* Refactor WriteChunk to not need callbackDan Engelbrecht2022-05-023-36/+22
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 20:57:49 +0000