aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/projectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* fix time regression on CI - iteratechunks block size (#249)Dan Engelbrecht2024-12-021-7/+7
| | | | * don't allow compact in time critical oplog calls * bump small chunk size limit when doing iteratechunks
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-2/+9
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* fix oplog index path reading error (#246)Dan Engelbrecht2024-11-281-30/+41
| | | * when reading paths for oplog index, make sure we don't point our string view to potentially stale memory
* set content type correctly for getchunkrange (#241)Dan Engelbrecht2024-11-281-3/+3
|
* skip rawsize/size in getchunksinfo if chunk is missing (#238)Dan Engelbrecht2024-11-261-2/+2
|
* snapshot crash (#234)Dan Engelbrecht2024-11-251-1/+6
| | | | * snapshot test * use proper IoBuffer constructor to take handle ownership
* fix oplog snapshot deadlock (#233)Dan Engelbrecht2024-11-251-31/+57
| | | | * store inlined chunk as temp files and store to Cid after oplog iteration is complete and ops updated * make sure we can get to the payload when doing `prep` for new ops
* /chunkinfo /files response size and rawsize size consistency (#230)Dan Engelbrecht2024-11-251-39/+54
| | | * don't provide size if compressed buffer validation fails
* caller controls threshold for bulk-loading chunks in IterateChunks (#222)Dan Engelbrecht2024-11-251-6/+11
| | | | | | * Allow caller to control threshold for bulk-loading chunks in IterateChunks * use smaller batch chunk reading for /fileinfos and /chunkinfos as we do not intend to read the payload * use smaller batch read buffer when just querying for size of attachments
* stronger validation of payload existance (#229)Dan Engelbrecht2024-11-251-41/+106
| | | | | | - Don't add RawSize and Size in ProjectStore::GetProjectFiles response if we can't get the payload - Use validation of payload size/existance in all chunk fetch operations in file cas - In project store oplog validate, make sure we can reach all the payloads - Add threading to oplog validate request
* oplog prep gc fix (#216)Dan Engelbrecht2024-11-151-160/+664
| | | | | | - Added option gc-validation to zenserver that does a check for missing references in all oplog post full GC. Enabled by default. - Feature: Added option gc-validation to zen gc command to control reference validation. Enabled by default. - Added more details in post GC log. - Fixed race condition in oplog writes which could cause used attachments to be incorrectly removed by GC
* oplog capture new attachments for gc (#210)Dan Engelbrecht2024-11-061-7/+10
| | | * make sure we track added attachments properly in oplogs
* Improved oplog import/export progress indicator at commandline (#206)Dan Engelbrecht2024-11-061-15/+32
| | | | Nicer progress bar during oplog import/export Verify that oplog has not been deleted from disk behind our back
* Dashboard: oplog entry data download, more detail, styling tweaks. (#194)Martin Ridgers2024-10-141-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * MSVC's std::fs::path doesn't like appending '/' separated path components * Redirect '/dashboard' to '/dashboard/' * Missed a few copyright headers * Moved unescaped logo somewhere more suitable * More robust catching and displaying of errors * No need for the guard now or for waiting * Formal access to a component's style * Style tables explicitly without instead of via a CSS variable * Highlight a row under the cursor to guide user's eye * Not using css_var() so it was removed * Add more detail to a project's list of oplogs * Disabled test page's CbObject testing * Consider all fields ending in "importedpackageids" as dependencies * Don't wrap sector headers * Package ids were derived with endianess back to front * Moved oplog marker column further left * Adopt a vararg-style to Table.add_row() for cell contents * List and hotlink oplog entries' package data * Modest control over how a table's columns are arranged * Added tables to test column spacing * Keep stat filter input box correctly up to date * A clang-format run * Updated html.zip * validate found oplog directories when doing discover pass --------- Co-authored-by: Dan Engelbrecht <[email protected]>
* Add ability to read the oplog's ReferencedSet, as written by the cook… (#190)v5.5.9-pre7Matt Peters2024-10-111-8/+197
| | | | Add ability to read the oplog's ReferencedSet, as written by the cooker, from the ReferencedSet op. Filter oplog entries requests by the ReferencedSet, if trim_by_referencedset parameter is present.. Add -trim=true/false parameter to oplog-mirror command, default to true, to request the trimmed/not trimmed oplog. Helper functions: Add paging to IterateOpLogWithKey. Add unit tests for IterateOpLog functions. Move OpKeyStringAsOid from httpprojectstore into projectstore.
* remove gc v1 (#121)Dan Engelbrecht2024-10-031-286/+0
| | | | | * kill gc v1 * block use of gc v1 from zen command line * warn and flip to gcv2 if --gc-v2=false is specified for zenserver
* gc command attachment options (#176)Dan Engelbrecht2024-09-301-8/+8
| | | * zen command - add options to control meta data cache when triggering gc
* Add `gc-attachment-passes` option to zenserver (#167)Dan Engelbrecht2024-09-251-6/+9
| | | | | Added option `gc-attachment-passes` to zenserver Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file Fixed project store not properly sorting references found during lock phase
* gc unused refactor (#165)Dan Engelbrecht2024-09-231-28/+21
| | | | | * optimize IoHash and OId comparisions * refactor filtering of unused references * add attachment filtering to gc
* End was clamped to the wrong side if Count was defaultedMartin Ridgers2024-09-191-1/+1
|
* clang-format on branch's changed filesMartin Ridgers2024-09-191-105/+111
|
* Mandatory IterateOplog() paging argumentMartin Ridgers2024-09-191-12/+9
|
* A clang-format passMartin Ridgers2024-09-191-3/+3
|
* Appease Apple Clang's historical lack of std::span constructorsMartin Ridgers2024-09-191-6/+2
|
* Forgot there was a std::clampMartin Ridgers2024-09-191-7/+4
|
* Placate clangMartin Ridgers2024-09-191-4/+4
|
* Optional paged results when iterating oplog entriesMartin Ridgers2024-09-191-4/+20
|
* fix oplog state check and wiping (#154)Dan Engelbrecht2024-09-111-2/+17
| | | | | fix oplog state check fix wipe of oplog storage + meta only on invalid oplog handle vanishing oplog during gc
* validate oplog before opening - if invalid, warn and wipe oplog (#153)Dan Engelbrecht2024-09-101-2/+12
|
* cleanup oplog logging (#147)Dan Engelbrecht2024-09-051-9/+29
| | | * clean up logging
* fix oplog index snapshot reading (#146)Dan Engelbrecht2024-09-051-5/+29
| | | * fixed parsing and added extra validation of oplog index snapshot
* fix oplog instance leak in project store gc (#145)Dan Engelbrecht2024-09-051-3/+7
|
* move gc logs to gc logger (#142)Dan Engelbrecht2024-09-041-10/+48
| | | - Improvement: Move GC logging in callback functions into "gc" context
* delay oplog read (#141)Dan Engelbrecht2024-09-031-145/+191
| | | - Improvement: Don't keep all oplogs open after GC, close them when references are fetched unless they are open by client
* oplog index snapshots (#140)Dan Engelbrecht2024-09-031-176/+706
| | | - Feature: Added project store oplog index snapshots for faster opening of oplog - opening oplogs are roughly 10x faster
* meta info store (#75)Dan Engelbrecht2024-08-301-19/+127
| | | | - Feature: Added option `--gc-cache-attachment-store` which caches referenced attachments in cache records on disk for faster GC - default is `false` - Feature: Added option `--gc-projectstore-attachment-store` which caches referenced attachments in project store oplogs on disk for faster GC - default is `false`
* Make sure `noexcept` functions does not leak exceptions (#136)Dan Engelbrecht2024-08-231-1/+1
|
* separate worker pools into burst/background to avoid background jobs ↵Dan Engelbrecht2024-08-221-2/+2
| | | | blocking client requests (#134)
* demote project store access time read fail to warning (#127)Dan Engelbrecht2024-08-201-1/+1
|
* verify that project oplog dir exists before trying to iterate it (#123)Dan Engelbrecht2024-08-191-5/+11
|
* fix oplog compact size logDan Engelbrecht2024-08-151-1/+1
|
* adds more information in log when oplog rename/delete fails (#117)Dan Engelbrecht2024-08-151-6/+25
| | | | | | * adds more information in log when oplog rename/delete fails * warn if we fail to delete dropped directory * close oplog storage file inside try/catch * don't try to flush closed files
* add compacting of oplogs as part of GC (#106)Dan Engelbrecht2024-08-121-105/+455
| | | | * add compacting of oplogs as part of GC * force retain of LSN unless we have less than 16 miln entries left
* project store chunk requests that are out of range will be treated as not ↵Dan Engelbrecht2024-08-121-0/+26
| | | | | | found (#108) * project store chunk requests that are out of range will be treaded as not found * add chunkid to log
* make oplog lsn unsigned (#107)Dan Engelbrecht2024-08-121-20/+9
| | | * change oplog lsn to uint32
* project/oplog delete improvements (#105)Dan Engelbrecht2024-08-121-46/+95
| | | | | * make oplog/project folder removeal more robust * report back error to http caller if removal fails
* stop exceptions from leaking on threaded work (#102)Dan Engelbrecht2024-08-061-23/+44
| | | | * catch exceptions in threaded work * don't abort all project file/chunk info fetch for single failure
* hardening read of corrupt oplog (#98)Dan Engelbrecht2024-08-061-24/+50
| | | * Add extra validation of oplog entries when reading oplog
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-131-99/+370
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* cache optimizations (#88)Dan Engelbrecht2024-05-301-4/+4
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 23:11:53 +0000