aboutsummaryrefslogtreecommitdiff
path: root/zenserver/projectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Clean up project store file structure (#218)Dan Engelbrecht2023-02-011-3097/+0
| | | | * move project store to separate folder * moved import/export project commands into projectstore cmd files
* Retain memory buffer when replaying oplog (#216)Dan Engelbrecht2023-01-261-3/+7
| | | | * Retain memory buffer when replaying oplog * changelog
* Better error responses/logging in project store request (#217)Dan Engelbrecht2023-01-261-93/+139
| | | | * Better error responses/logging in project store request * changelog
* zen command line tool improvements (#212)Dan Engelbrecht2023-01-131-0/+1
| | | | | | | | | | | | - Feature: zen command line tool `cache-info` to show cache, namespace or bucket info - Feature: zen command line tool `project-info` to show store, project or oplog info - Feature: zen command line tool `project-drop` to drop project or oplog - Feature: zen command line tool `gc` to trigger a GC run - Feature: zen command line tool `gc-info` to check status of GC - Improvement: zen command line tool now fails on any unrecognized arguments - Improvement: zen command line tool now displays extra help for all sub-commands - Improvement: host address can now be configured for zen command line tool `drop` command changelog
* Add info (GET) endpoints for structured cache (#211)Dan Engelbrecht2023-01-111-3/+8
| | | | | | | * Add GET requests on cache/namespace/bucket level * Add root route for project store requests (same as /list) * Add markerpath to oplog info object * Add totalsize, opcount and expired to oplog info * Changelog
* oplog level GC (#209)Dan Engelbrecht2022-12-141-29/+183
| | | | | | | | Adds check for marker file supplied by UE to see if an oplog is expired (user has deleted the corresponding cooked folder). Fixed concurrency vulnerabilities is project store related to oplogs. * fix concurrency vulnerabilities * propagate lifetime file path * oplog level gc * changelog
* Changed so CompressedBuffer::DecodeRawHash returns IoHash just like on the ↵Stefan Boberg2022-12-121-2/+2
| | | | | | UE side (#208) removed all use of IoHash::FromBLAKE3() caused by interactions with CompressedBuffer APIs
* optimizations (#200)Dan Engelbrecht2022-12-071-19/+31
| | | | | | | | | | | | | * Use direct file read and direct buffer allocation for small IoBuffer materalization * Reduce range of materialized data in CompositeBuffer reading CompressedBuffer header reading often only need a small part and not the whole file * reduce lock contention in IoBuffer::Materialize * Reduce parsing of compressed headers Validate header type at decompression * faster CreateDirectories - start from leaf going up and recurse back * optimized BufferHeader::IsValid * Add ValidateCompressedHeader to use when we don't need the actual compressed data Validate that we always get compressed data in CidStore::AddChunk * changelog
* Size based gc trigger (#197)Dan Engelbrecht2022-12-021-3/+3
| | | | | - Feature: Disk size triggered GC, a soft disk usage limit for cache data. - Feature: New option `--gc-disk-size-soft-limit` (command line), `gc.cache.disksizesoftlimit` (lua config) controlling limit for soft disk usage limit. Defaults to zero which disables soft disk usage limit. - Improvement: Disk write pressure in GC log and cleaned up clutter in GC logging.
* reduce gc log spam (INFO -> DEBUG) (#199)Dan Engelbrecht2022-12-021-16/+17
|
* Make sure we always store record/op before attachments (#195)Dan Engelbrecht2022-12-011-22/+35
| | | | | | | | * Make sure we always store record/op before attachments We don't want to store attachments first - a GC operation could then remove attachments if triggered before storing record/op * zen::Latch * Use latch to wait for attachments to be stored * use zen::latch when adding attachments from project oplog import * changelog
* Fix disk usage stats (#194)Dan Engelbrecht2022-11-241-2/+42
| | | | | | | | * Improve tracking of used disk space for filecas and compactcas Add tracking of used disk space for project store Remove ZenCacheStore as GcStorage/GcContributor - underlying ZenCacheNamespace instances register themselves directly - removing this also fixes double reporting of GcStorageSize for namespaces * changelog
* changelog and removed obsolete commentDan Engelbrecht2022-11-181-1/+0
|
* Add `import-project` and `export-project` (#183)Dan Engelbrecht2022-11-181-208/+629
| | | * Add `import-project` and `export-project` command line parsing
* move BasicFile to zenutil to remove zenstore dependency from zen command (#190)Dan Engelbrecht2022-11-171-1/+1
|
* Support file reference in package message (#184)Dan Engelbrecht2022-11-071-6/+6
| | | | | | | | * Fix packed message parsing for absolute path * Always enable are sharing when opening files as IoBuffers. * Allow control over sending partial files as localfile ref * Check "AcceptFlags" field in RPC message for allowing localfile ref in reply * make oplog entry add operations ZEN_DEBUG level logs * changelog
* fix concurrency issues in projectstore and enable GC (#181)Dan Engelbrecht2022-10-171-237/+594
| | | | | | | | | | | | | | * Fix concurreny issues when deleting projects/oplogs * remove rocksdb test code * project store unit tests * safer deletion of oplogs/projects * reference count ProjectStore::Project to handle lifetime during GC * Don't open all project oplogs unless we need them * Don't scrub expired projects * Don't gather references from expired projects * added logging details for GC * release lock as soon as folder is moved * more tests for project store * changelog
* disable project store GC (#179)Dan Engelbrecht2022-10-121-1/+2
|
* De/oplog gc lifetime (#178)Dan Engelbrecht2022-10-101-43/+131
| | | | | | | * Make sure we don't use invalidated iterators in projectstore.cpp * project store keeps track of project file and will garbage collect data for a project if the project file no longer exist * Implement GC of projects in project store - still need to fix lifetime issues for Project instances * Add INFO log if project file path is empty in projectstore * changelog
* De/reduce buffer creation in parsepackedmessage (#175)Dan Engelbrecht2022-09-301-7/+7
| | | | | * Don't create call CreateBuffer for attachement data that we only read and not keep * changelog * don't read oplog attachments into memory just to do a redundant store of them
* Adjust errors vs warnings messages (#160)Dan Engelbrecht2022-09-081-4/+4
| | | | * demote a number of ZEN_ERROR to ZEN_WARN * changelog
* Make cas storage an hidden implementation detail of CidStore (#130)v0.1.4-pre6v0.1.4-pre5Dan Engelbrecht2022-06-171-5/+5
| | | | | | | | | | | | | | | | | | - Bumped ZEN_SCHEMA_VERSION - CasStore no longer a public API, it is hidden behind CidStore - Moved cas.h from public header folder - CidStore no longer maps from Cid -> Cas, we store entries in Cas under RawHash - CasStore now decompresses data to validate content (matching against RawHash) - CasChunkSet renames to HashKeySet and put in separate header/cpp file - Disabled "Chunk" command for now as it relied on CAS being exposed as a service - Changed CAS http service to Cid http server - Moved "Run" command completely inside ZEN_WITH_EXEC_SERVICES define - Removed "cas.basic" test - Uncommented ".exec.basic" test and added return-skip at start of test - Moved ScrubContext to separate header file - Renamed CasGC to GcManager - Cleaned up configuration passing in cas store classes - Removed CAS stuff from GcContext and clarified naming in class - Remove migration code
* removed LocalProjectService prototypeStefan Boberg2022-06-131-300/+0
|
* fixed issue where projects would not be discovered via DiscoverProjects due ↵Stefan Boberg2022-06-101-10/+100
| | | | | | to use of stem() vs filename() added /prj/{project}/oplog/{log}/{op} endpoint to allow retrieval of an op entry by LSN. Supports returning CbObject or CbPackage format payloads
* Added GetDirectoryContent utilityDan Engelbrecht2022-05-061-42/+9
|
* simplify lambda capturesDan Engelbrecht2022-05-021-1/+1
|
* always keep full log but read from index snapshot location if availableDan Engelbrecht2022-04-041-21/+24
|
* rename EMode to ModeDan Engelbrecht2022-04-011-4/+4
|
* Don hard fail on removing files we no longer care aboutDan Engelbrecht2022-03-311-3/+2
|
* improved logging for gc/migrationDan Engelbrecht2022-03-311-0/+6
|
* Migration now works in larger disk IO chunksDan Engelbrecht2022-03-311-4/+4
| | | | BasicFile and CasLogFile now has new explicit modes instead of create true/false
* Fixed wrong route pattern.Per Larsson2022-03-231-1/+1
|
* Added route '/prj/list' for retrieving project info.Per Larsson2022-03-231-0/+32
|
* Fixed missing object in project oplog response.Per Larsson2022-01-241-1/+5
|
* Format fix.Per Larsson2022-01-221-1/+2
|
* Unused lambda capture compile errorMartin Ridgers2022-01-101-1/+1
|
* Converted use of _format UDL to fmt::formatMartin Ridgers2022-01-101-12/+7
|
* Added trace scopes.Per Larsson2021-12-151-0/+9
|
* Initialisation order compile errorMartin Ridgers2021-12-151-1/+1
|
* Paths are not wide-chars on non-Windows platformsMartin Ridgers2021-12-151-2/+2
|
* Merged main.Per Larsson2021-12-141-24/+35
|\
| * Removed the special-case path handling to make project tests passMartin Ridgers2021-11-291-12/+0
| |
| * Only Windows paths have drive lettersMartin Ridgers2021-11-291-0/+2
| |
| * ToUtf8() -> PathToUtf8() as the latter is less ambiguousMartin Ridgers2021-11-161-3/+3
| |
| * Stubbed out Project Store's PipeConnection on POSIXMartin Ridgers2021-11-021-0/+17
| |
| * Fixed up some assumptions that satd::fs::path uses wchar_tMartin Ridgers2021-11-011-4/+4
| |
| * Removed unused variableMartin Ridgers2021-11-011-4/+0
| |
| * Initialisation order warningsMartin Ridgers2021-11-011-2/+2
| |
| * Unhandled switch case warningsMartin Ridgers2021-11-011-0/+6
| |
| * Fixed up Windows-specific include statementsMartin Ridgers2021-10-291-1/+4
| |
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 23:24:10 +0000