aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/projectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* clean up GcContributor and GcStorage to be pure interfaces (#485)Dan Engelbrecht2023-10-201-3/+6
|
* add `flush` command and more gc status info (#483)Dan Engelbrecht2023-10-181-2/+3
| | | | | | - Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore - Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore - Improved: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc - Changed: Cache access and write log are disabled by default
* faster oplog iteration (#471)Dan Engelbrecht2023-10-131-22/+34
| | | | | * use a CbObjectView instead of CbObject to avoid creating IOBufferCore instances * use BasicFileBuffer directly where possible * changelog
* cache reference tracking (#455)Dan Engelbrecht2023-10-101-6/+9
| | | | | - Feature: Add caching of referenced CId content for structured cache records, this avoid disk thrashing when gathering references for GC - disabled by default, enable with `--cache-reference-cache-enabled` - Improvement: Faster collection of referenced CId content in project store
* fix gc infinite loop (#453)Dan Engelbrecht2023-10-061-5/+13
| | | | * make sure we update last gc time even if gc fails * If we can't check if an oplog/project markerfile exists, assume it is not expired
* faster accesstime save restore (#439)Dan Engelbrecht2023-10-031-23/+43
| | | | | | | | | | - Improvement: Reduce time a cache bucket is locked for write when flushing/garbage collecting - Change format for faster read/write and reduced size on disk - Don't lock index while writing manifest to disk - Skip garbage collect if we are currently in a Flush operation - BlockStore::Flush no longer terminates currently writing block - Garbage collect references to currently writing block but keep the block as new data may be added - Fix BlockStore::Prune used disk space calculation - Don't materialize data in filecas when we just need the size
* lightweight gc (#431)Dan Engelbrecht2023-10-021-0/+4
| | | | | | - Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store - Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false" - Add `--skipcid` option to `zen gc` command, defaults to false - Add `--gc-lightweight-interval-seconds` option to zenserver
* Improvement: Add names to background jobs for easier debugging (#412)Dan Engelbrecht2023-09-201-28/+32
| | | | Improvement: Background jobs now temporarily sets thread name to background job name while executing Improvement: Background jobs tracks worker thread id used while executing
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-201-8/+102
| | | currently, only Windows (using Projected File System) is supported
* add more trace scopes (#362)Dan Engelbrecht2023-09-151-30/+39
| | | | | * more trace scopes * Make sure ReplayLogEntries uses the correct size for oplog buffer * changelog
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-131-43/+77
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* scan oplog object for fields (#397)Dan Engelbrecht2023-09-131-89/+130
| | | | * scan oplog object for fields * Read all oplog entries but only read op data and get mapping for latest op of each key
* incremental oplog upload for block-based targets (#392)Dan Engelbrecht2023-09-121-2/+20
| | | | | | | * add option for base container for oplog export read base oplog and fetch known blocks * reuse blocks if a known block has 80+ % usage * changelog * better logging and added base to remotestore descriptions
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-7/+8
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* project store gc deadlock (#381)Dan Engelbrecht2023-08-311-29/+42
| | | | * Trying to recursively take a shared RWLock while another thread is waiting for a exclusive lock results in a deadlock. Forward the shared lock so we don't have to grab it recursively. * changelog
* Add `--embedloosefiles` option to `oplog-export` (#376)Dan Engelbrecht2023-08-241-8/+7
| | | | * Add `--embedloosefiles` option to `oplog-export` which adds loose files to the export, removing need to call `oplog-snapshot` * Retain `ServerPath` in oplog when performing `oplog-snapshot`. This is a short-term fix for current incompatability with the UE cooker.
* use robinmap in compact cas (#368)Dan Engelbrecht2023-08-211-4/+5
| | | | | * Use robin-map in compactcas for 30% faster CasContainerStrategy::CollectGarbage * use robin_set in ProjectStore::Oplog::GatherReferences and BlockStore::ReclaimSpace * changelog
* buffered file reading for oplog (#366)Dan Engelbrecht2023-08-211-5/+16
| | | | | * add BasicFileBuffer for buffered read of BasicFile * Use BasicFileBuffer when reading oplog * changelog
* add update/delete endpoint for project and oplog (#353)Dan Engelbrecht2023-08-181-0/+40
| | | | | | | | | | | | | * add update endpoint for project store project * add update endpoint for oplog * changelog * Zen command line tool `project-update` Zen command line tool `project-delete` Zen command line tool `oplog-update` Zen command line tool `oplog-delete` * add --force-update option to project/oplog create remove project/oplog update commnad
* check oplog op attachments when gathering references for GC (#363)Dan Engelbrecht2023-08-181-19/+9
| | | | | * Make sure to check oplog op attachments when gathering references for GC * Add oplog op content to error result if attachment is missing when doing `oplog-export`
* project store stats (#357)Dan Engelbrecht2023-08-141-28/+45
| | | | | | | * add basic stats for project store * stats for BadRequest in ProjectStore::Rpc * changelog * group stats fix ChunkWriteCount when accepting ops
* update vcpkg dependencies (#356)Dan Engelbrecht2023-08-111-2/+2
| | | | | | * bump vcpkg version * fmt lib 10 fixes * xmake dependencies (with linux workarounds) * changelog
* Make sure we always write "data" attachment hash for snapshotted oplog ↵Dan Engelbrecht2023-08-111-7/+25
| | | | | | | | | entries (#355) * Make sure we always write "data" attachment hash for snapshotted oplog entries * Make sure to add chunk mappings for files moved to attatchment in snapshot operation * fix inverted timoute for expiration (we don't want time expiry in these cases) * increase timeout for jupiter oplog in project to 3 min * changelog
* Allow oplog file mapping where ServerPath is missing if a attachment hash is ↵Dan Engelbrecht2023-08-101-4/+9
| | | | specified (#352)
* Add `--assume-http2` option to cloud style import/export command to use a ↵Dan Engelbrecht2023-08-091-1/+3
| | | | HTTP/2 endpoint without without HTTP/1.1 upgrade (#347)
* fix asserts and exceptions (#344)Dan Engelbrecht2023-08-081-1/+5
| | | | | | * Send proper error to caller of GetChunkInfo instead of assert * catch and handle exceptions when checking for state_marker * properly wait for background tasks if oplop-export fails * changelog
* file share support (#328)Stefan Boberg2023-06-161-17/+16
| | | | | | | | | | | | this change adds a serve command to the zen CLI. This can be used to establish links to a set of files which may be served to clients via the project store interface: ```cmd> zen serve Lyra/WindowsClient d:\temp_share\StagedBuilds\WindowsClient``` with the appropriate changes in UE you may then start an instance of the runtime and have it load all files via the remote file connection: ``` Lyra\Binaries\LyraClient.exe ../../../Lyra/Lyra.uproject -pak -basedir=D:\temp_share\StagedBuilds\WindowsClient/Lyra/Binaries/Win64 -Mount=Lyra/WindowsClient ```
* oplog snapshot (#317)Stefan Boberg2023-05-251-6/+154
| | | | | Added "snapshot" oplog RPC this may be used to bring referenced files into the local store instead of referencing them by filename, thus making the project/oplog transportable
* block destructors from throwing exceptions (#321)Dan Engelbrecht2023-05-241-5/+5
| | | | | | * ~FileMapping() is not allowed to throw exceptions * ~ScopedActivityBase() should not call ZEN_ASSERT (which causes SIGABORT on error) * ProjectStore::Project::WriteAccessTimes() which is called from ProjectStore::~Project() must not throw exceptions * changelog
* fixed bug where an oplog delete would not actually delete the oplogStefan Boberg2023-05-191-2/+9
| | | | this would manifest itself if an oplog delete was attempted right after zenserver startup, when the oplog has not yet been instantiated. This fix checks for on-disk state as well as in-memory
* project store refactor (#316)Stefan Boberg2023-05-171-1476/+0
| | | moved HttpProjectService into own file to improve maintainability
* Additional trace instrumentation (#312)Stefan Boberg2023-05-161-0/+36
| | | | | | | | | * added trace instrumentation to upstreamcache * added asio trace instrumentation * added trace annotations for project store * added trace annotations for BlockStore * added trace annotations for HttpClient * added trace annotations for CAS/GC
* Add `--gc-projectstore-duration-seconds` option (#281)Dan Engelbrecht2023-05-161-12/+12
| | | | | | * Add `--gc-projectstore-duration-seconds` option * Cleanup lua gc options parsing * Remove dead configuration values * changelog
* added scrubbing related loggingStefan Boberg2023-05-151-0/+2
|
* minor GC API cleanupStefan Boberg2023-05-151-5/+5
| | | | | Scrub -> ScrubStorage Trigger -> TriggerGc (to make relationship to TriggerScrub clearer)
* WARN level log if we can't write snapshot/manifest/access times (#288)Dan Engelbrecht2023-05-111-4/+11
|
* Low disk space detector (#277)Dan Engelbrecht2023-05-091-38/+80
| | | | * - Feature: Disk writes are now blocked early and return an insufficient storage error if free disk space falls below the `--low-diskspace-threshold` value * Never keep an entry in m_ChunkBlocks that points to a nullptr
* project store gc lifetime (#257)Dan Engelbrecht2023-05-081-68/+320
| | | | | | | * keep track of last access time for project store projects and oplogs * check size on disk for project store total size * read/write access times to disk * changelog * change some std::filesystem::path -> const std::filesystem::path&
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+4082
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 02:05:36 +0000