aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/projectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* remove gc v1 (#121)Dan Engelbrecht2024-10-031-286/+0
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc command attachment options (#176)Dan Engelbrecht2024-09-301-8/+8
| | | * zen command - add options to control meta data cache when triggering gc
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-6/+9
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* gc unused refactor (#165)Dan Engelbrecht2024-09-231-28/+21
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* End was clamped to the wrong side if Count was defaultedMartin Ridgers2024-09-191-1/+1
|
* clang-format on branch's changed filesMartin Ridgers2024-09-191-105/+111
|
* Mandatory IterateOplog() paging argumentMartin Ridgers2024-09-191-12/+9
|
* A clang-format passMartin Ridgers2024-09-191-3/+3
|
* Appease Apple Clang's historical lack of std::span constructorsMartin Ridgers2024-09-191-6/+2
|
* Forgot there was a std::clampMartin Ridgers2024-09-191-7/+4
|
* Placate clangMartin Ridgers2024-09-191-4/+4
|
* Optional paged results when iterating oplog entriesMartin Ridgers2024-09-191-4/+20
|
* fix oplog state check and wiping (#154)Dan Engelbrecht2024-09-111-2/+17
| | | | | fix oplog state check fix wipe of oplog storage + meta only on invalid oplog handle vanishing oplog during gc
* validate oplog before opening - if invalid, warn and wipe oplog (#153)Dan Engelbrecht2024-09-101-2/+12
|
* cleanup oplog logging (#147)Dan Engelbrecht2024-09-051-9/+29
| | | * clean up logging
* fix oplog index snapshot reading (#146)Dan Engelbrecht2024-09-051-5/+29
| | | * fixed parsing and added extra validation of oplog index snapshot
* fix oplog instance leak in project store gc (#145)Dan Engelbrecht2024-09-051-3/+7
|
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-10/+48
| | | - Improvement: Move GC logging in callback functions into "gc" context
* delay oplog read (#141)Dan Engelbrecht2024-09-031-145/+191
| | | - Improvement: Don't keep all oplogs open after GC, close them when references are fetched unless they are open by client
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-176/+706
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-301-19/+127
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-231-1/+1
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* demote project store access time read fail to warning (#127)Dan Engelbrecht2024-08-201-1/+1
|
* verify that project oplog dir exists before trying to iterate it (#123)Dan Engelbrecht2024-08-191-5/+11
|
* fix oplog compact size logDan Engelbrecht2024-08-151-1/+1
|
* adds more information in log when oplog rename/delete fails (#117)Dan Engelbrecht2024-08-151-6/+25
| | | | | | * adds more information in log when oplog rename/delete fails * warn if we fail to delete dropped directory * close oplog storage file inside try/catch * don't try to flush closed files
* add compacting of oplogs as part of GC (#106)Dan Engelbrecht2024-08-121-105/+455
| | | | * add compacting of oplogs as part of GC * force retain of LSN unless we have less than 16 miln entries left
* project store chunk requests that are out of range will be treated as not ↵Dan Engelbrecht2024-08-121-0/+26
| | | | | | found (#108) * project store chunk requests that are out of range will be treaded as not found * add chunkid to log
* make oplog lsn unsigned (#107)Dan Engelbrecht2024-08-121-20/+9
| | | * change oplog lsn to uint32
* project/oplog delete improvements (#105)Dan Engelbrecht2024-08-121-46/+95
| | | | | * make oplog/project folder removeal more robust * report back error to http caller if removal fails
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-061-23/+44
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* hardening read of corrupt oplog (#98)Dan Engelbrecht2024-08-061-24/+50
| | | * Add extra validation of oplog entries when reading oplog
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-131-99/+370
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* cache optimizations (#88)Dan Engelbrecht2024-05-301-4/+4
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* fix iterate chunks crash (#86)Dan Engelbrecht2024-05-271-0/+1
| | | * fix worklatch count in Oplog::IterateChunks
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-021-3/+10
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-021-3/+3
| | | * use write and move in place for safer writing of files
* fix get project files loop (#68)Dan Engelbrecht2024-04-301-23/+20
| | | | | - Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found - Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash - Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-261-40/+112
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* fix oplog import during gcv2 (#62)v5.5.0-pre3v5.5.0-pre2Dan Engelbrecht2024-04-251-39/+44
| | | | - Bugfix: Always pre-cache oplog when creating project store GCv2 referencer - Bugfix: Correctly capture attachments imported with oplog to void them being GCd before oplog is written
* iterate cas chunks (#59)Dan Engelbrecht2024-04-241-28/+27
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-4/+13
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* Bugfix: Only disable oplog update capture if we have started it (#58)Dan Engelbrecht2024-04-241-1/+6
|
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-221-15/+44
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-32/+1
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* safer oplog import (#52)de/safer-oplog-importDan Engelbrecht2024-04-181-8/+71
| | | | * reference cache gc update capture * When importing oplogs we now import all attachments first and (optionally clean) write the oplog on success
* remote project store stats (#44)Dan Engelbrecht2024-04-101-0/+1
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* improved assert (#37)Dan Engelbrecht2024-04-041-5/+5
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-1/+1
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 19:42:55 +0000