aboutsummaryrefslogtreecommitdiff
path: root/zenstore/compactcas.cpp
Commit message (Collapse)AuthorAgeFilesLines
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-1511/+0
| | | | | | * moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
* save cache rawsize and rawhash in manifest file instead of log file (#251)Dan Engelbrecht2023-04-211-4/+4
| | | | * save cache rawsize and rawhash in manifest file instead of log file * don't use # as prefix for counts in log
* tweaks for enabling unity buildsStefan Boberg2023-04-191-9/+9
| | | | | | | | | | mostly changes to make sure anonymous namespaces don't clash and a change to avoid windows headers from leaking into other compilation units unity builds are not yet enabled by default, but can be enabled by uncommenting this line in the root `xmake.lua` ``` --add_rules("c++.unity_build") ```
* Fix log index snapshot (#210)Dan Engelbrecht2022-12-161-14/+17
| | | | | | * Fix log reading for structured cache store Make sure cache is flushed at exit * dont flush index to disk unless new entries have been written * changelog
* Fix logging of number of read entries from log/index file for stores (#204)Dan Engelbrecht2022-12-081-2/+2
| | | | * Fix logging of number of read entries from log/index file for stores * changelog
* optimizations (#200)Dan Engelbrecht2022-12-071-4/+9
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Size based gc trigger (#197)Dan Engelbrecht2022-12-021-11/+11
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-021-1/+1
|
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-241-19/+10
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-081-1/+1
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-171-596/+149
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* merged from mainStefan Boberg2022-06-161-30/+38
|
* Add catch2 support (#101)Stefan Boberg2022-05-201-1/+5
| | | | | | | Added option to use catch2 for unit tests Currently both doctest and catch2 are supported via some compatibility macros. doctest is the default, and ZEN_USE_CATCH2 needs to be defined to switch to catch2. Our goal is to evaluate how well catch2 works and switch to catch2 if everything pans out since UE5 now supports using catch2 for unit tests.
* Make sure blockstore owner and block store state does not get out of sync ↵Dan Engelbrecht2022-05-081-10/+4
| | | | | | when fetching a chunk Move MarkAsDeleteOnClose() to IoBuffer(ExtendedCore) and set it on close, SetFileInformationByHandle sometimes fails if done in parallel with FileMapping
* handle that more than one block can be written to in parallelDan Engelbrecht2022-05-031-17/+12
|
* more tests for block storeDan Engelbrecht2022-05-031-2/+2
|
* switched back memory_order for m_TotalSize to relaxedDan Engelbrecht2022-05-021-2/+2
|
* simplify lambda capturesDan Engelbrecht2022-05-021-11/+9
|
* Refactor WriteChunk to not need callbackDan Engelbrecht2022-05-021-10/+9
|
* restore cg commentDan Engelbrecht2022-05-021-2/+25
|
* cleanupDan Engelbrecht2022-05-021-9/+4
|
* add back gc space reclaim callDan Engelbrecht2022-05-021-1/+2
|
* use std::vector<std::pair>> instead of mapDan Engelbrecht2022-05-021-18/+2
|
* Move bulk of MigrateLegacyData to blockstore.cppDan Engelbrecht2022-05-021-217/+51
|
* respect Ctx.RunRecovery()Dan Engelbrecht2022-05-011-20/+24
|
* reimplement CasContainerStrategy::ScrubDan Engelbrecht2022-05-011-78/+54
|
* collectgarbage for compactcas and structured cache uses shared implementationDan Engelbrecht2022-05-011-45/+8
|
* threading issues resolvedDan Engelbrecht2022-05-011-15/+38
|
* first pass at generic block store with gcDan Engelbrecht2022-04-301-454/+84
|
* wait until work is completed, not just picked upDan Engelbrecht2022-04-121-12/+21
|
* safer check for added size i threaded testDan Engelbrecht2022-04-121-1/+3
|
* reduce number of chunks in compactcas.threadedinsertDan Engelbrecht2022-04-121-1/+1
|
* remove unused variableDan Engelbrecht2022-04-121-1/+1
|
* Fix failing test due to not generating unique test dataDan Engelbrecht2022-04-121-70/+70
|
* Mac fixMartin Ridgers2022-04-121-1/+1
|
* Be more strict with m_TotalSize atomic operationsDan Engelbrecht2022-04-121-2/+2
|
* clean up variable namingDan Engelbrecht2022-04-061-56/+55
|
* formattingDan Engelbrecht2022-04-061-4/+1
|
* logging tweaksDan Engelbrecht2022-04-061-11/+7
|
* Only validate range of legacy entries on kept entriesDan Engelbrecht2022-04-061-11/+26
|
* fix migration when entries is out of data file rangeDan Engelbrecht2022-04-051-1/+7
|
* Make index/log reading non-static member functionsDan Engelbrecht2022-04-041-485/+469
|
* remove GetTempLogPathDan Engelbrecht2022-04-041-5/+0
|
* remove obsolete GetRecoverLogPathDan Engelbrecht2022-04-041-5/+0
|
* always keep full log but read from index snapshot location if availableDan Engelbrecht2022-04-041-100/+90
|
* logging cleanupDan Engelbrecht2022-04-041-23/+29
|
* fix logging when using disk reserve in compactcasDan Engelbrecht2022-04-021-2/+2
|
* fix gc reclaim logic in compactcasDan Engelbrecht2022-04-021-2/+2
|
* proper error handling when setting file sizeDan Engelbrecht2022-04-021-1/+1
|
* rename EMode to ModeDan Engelbrecht2022-04-011-19/+19
|