aboutsummaryrefslogtreecommitdiff
path: root/zenstore/include
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-027-703/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* oplog and cache stats (#244)Dan Engelbrecht2023-04-211-2/+2
| | | | | | | | | | | | | | | | * basic oplog stats * add GetValueStats to cache store * RwLock::ExclusiveLockScope -> RwLock::SharedLockScope * add rawhash and attachment count to CacheValueStats * added cache-stats and project-stats commands * add cast to make Mac overload detection happy * fix accept type in cache-stats command * Add options to project-stats command * use resource paths for stats in project store * use resource paths for stats in cache * fix cache-info and project-info url discriminator * more control over details$ output * cleanup * changelog
* store cache rawhash and rawsize for unstructured cache values (#234)Dan Engelbrecht2023-02-231-8/+10
| | | | | | | | | * refactored MemoryCacheBucket to allow for storing RawHash/RawSize. * remove redundant conversions in AccessTime * reduce max count for memory cache bucket to 32-bit value * refactored DiskCacheBucket to allow for storing RawHash/RawSize. * Use CompressedBuffer::ValidateCompressedHeader when applicable * Make sure we rewrite the snapshot if we read an legacy existing index/log * changelog
* oplog upload/download (#214)Dan Engelbrecht2023-02-091-0/+1
| | | | | | | | | | | | - Feature: Zen server endpoint `prj/{project}/oplog/{log}/chunks` to post multiple attachments in one request. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/save` to save an oplog container. Accepts `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `prj/{project}/oplog/{log}/load` to request an oplog container. Responds with an `CbObject` containing a compressed oplog and attachment references organized in blocks. - Feature: Zen server endpoint `{project}/oplog/{log}/rpc` to initiate an import to or export from an external location and other operations. Use either JSon or CbPackage as payload. - CbObject/JSon RPC format for `import` and `export` methods: - CbObject RPC format for `getchunks` method, returns CbPackage with the found chunks, if all chunks are found the number of attachments matches number of chunks requested. - Feature: Zen server `{project}/oplog/{log}/{hash}` now accepts `HttpVerb::kPost` as well as `HttpVerb::kGet`. - Feature: Zen command line tool `oplog-export` to export an oplog to an external target using the zenserver oplog export endpoint. - Feature: Zen command line tool `oplog-import` to import an oplog from an external source using the zenserver oplog import endpoint.
* optimizations (#200)Dan Engelbrecht2022-12-071-1/+1
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Size based gc trigger (#197)Dan Engelbrecht2022-12-021-7/+32
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-241-1/+4
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-181-1/+6
| | | * Add `import-project` and `export-project` command line parsing
* move BasicFile to zenutil to remove zenstore dependency from zen command (#190)Dan Engelbrecht2022-11-173-117/+2
|
* Remove legacy code (#161)Dan Engelbrecht2022-09-081-10/+0
| | | | * changelog * remove obsolete legacy code
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-176-189/+152
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* merged from mainStefan Boberg2022-06-161-7/+6
|
* removed more unnecessary CasStore related codeStefan Boberg2022-06-132-27/+0
|
* We need to make each Cache Key contribution per bucket unique even across ↵Dan Engelbrecht2022-06-071-2/+2
| | | | namespaces
* Add catch2 support (#101)Stefan Boberg2022-05-201-0/+2
| | | | | | | Added option to use catch2 for unit tests Currently both doctest and catch2 are supported via some compatibility macros. doctest is the default, and ZEN_USE_CATCH2 needs to be defined to switch to catch2. Our goal is to evaluate how well catch2 works and switch to catch2 if everything pans out since UE5 now supports using catch2 for unit tests.
* Make sure blockstore owner and block store state does not get out of sync ↵Dan Engelbrecht2022-05-082-9/+9
| | | | | | when fetching a chunk Move MarkAsDeleteOnClose() to IoBuffer(ExtendedCore) and set it on close, SetFileInformationByHandle sometimes fails if done in parallel with FileMapping
* handle that more than one block can be written to in parallelDan Engelbrecht2022-05-031-7/+9
|
* more tests for block storeDan Engelbrecht2022-05-031-2/+3
|
* cleanupDan Engelbrecht2022-05-031-4/+0
|
* Make sure we close all block files when dropping a cache bucketDan Engelbrecht2022-05-021-0/+1
|
* Refactor WriteChunk to not need callbackDan Engelbrecht2022-05-021-13/+7
|
* Don't use "GC" term in block storeDan Engelbrecht2022-05-021-3/+3
|
* cleanupDan Engelbrecht2022-05-021-14/+19
|
* add back gc space reclaim callDan Engelbrecht2022-05-021-1/+4
|
* use std::vector<std::pair>> instead of mapDan Engelbrecht2022-05-021-4/+4
|
* Move bulk of MigrateLegacyData to blockstore.cppDan Engelbrecht2022-05-021-8/+15
|
* cleanupDan Engelbrecht2022-05-011-9/+12
|
* reimplement CasContainerStrategy::ScrubDan Engelbrecht2022-05-011-0/+4
|
* remove m_TotalSize for blockstoreDan Engelbrecht2022-05-011-1/+0
| | | | fix scrub logic in structured cache store
* collectgarbage for compactcas and structured cache uses shared implementationDan Engelbrecht2022-05-011-0/+3
|
* threading issues resolvedDan Engelbrecht2022-05-011-18/+36
|
* first pass at generic block store with gcDan Engelbrecht2022-04-301-0/+41
|
* correct expire vs contributeDan Engelbrecht2022-04-121-2/+1
|
* always keep full log but read from index snapshot location if availableDan Engelbrecht2022-04-041-6/+8
|
* rename EMode to ModeDan Engelbrecht2022-04-012-7/+7
|
* use std::unsigned_integral for ToHexNumber and ParseHexNumberDan Engelbrecht2022-04-011-1/+1
|
* linux compilation fixesDan Engelbrecht2022-04-011-1/+1
|
* linux fixesDan Engelbrecht2022-04-011-1/+1
|
* Make gc reserve a global resourceDan Engelbrecht2022-04-011-0/+4
|
* reserve space for log entries before replayDan Engelbrecht2022-04-011-0/+1
|
* Truncate migrated and new blocks after gc to save disk spaceDan Engelbrecht2022-03-311-0/+1
|
* Switch from std::shared_ptr<> to Ref<>Dan Engelbrecht2022-03-311-1/+1
| | | | Remove a bunch of 'auto' with explicit type
* Remove redundant lock in BlockStoreFileDan Engelbrecht2022-03-311-1/+0
|
* keep all block files open from startDan Engelbrecht2022-03-311-3/+2
|
* cleanupDan Engelbrecht2022-03-311-6/+1
|
* incremental migration with optional clean of sourceDan Engelbrecht2022-03-311-9/+5
| | | | add more fine-grained access modes for BasicFile
* Migration now works in larger disk IO chunksDan Engelbrecht2022-03-313-6/+29
| | | | BasicFile and CasLogFile now has new explicit modes instead of create true/false
* review feedbackDan Engelbrecht2022-03-311-4/+4
|
* Add CasLog::Append with multiple entriesDan Engelbrecht2022-03-311-0/+2
|
* cleanup duplicate code in CollectGarbageDan Engelbrecht2022-03-311-0/+2
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 17:21:59 +0000