aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* WIPde/memory-stomp-detectorDan Engelbrecht2024-05-216-19/+188
|
* refactor BlockStore IterateChunks (#77)Dan Engelbrecht2024-05-174-279/+308
| | | Improvement: Refactored IterateChunks to allow reuse in diskcachelayer and hide public GetBlockFile() function in BlockStore
* safer partial requests (#82)Dan Engelbrecht2024-05-172-37/+48
| | | | * fix partial cache miss logging * Handle zero size partial requests safer
* fix argument order when logging missed partial chunk (#81)Dan Engelbrecht2024-05-141-8/+11
| | | | | | * fix argument order when logging missed partial chunk * make partial value a temporary variable helps with debugging as the warning on bad range can tell the original size * add asserts
* return as missing if out of range chunk (#78)Dan Engelbrecht2024-05-081-20/+24
| | | * treat invalid partial chunk request as a miss
* Correctly calculate memory view size from Mid/MidInline function if size is ↵Dan Engelbrecht2024-05-081-2/+8
| | | | not given (#76)
* check partial chunk result (#73)Dan Engelbrecht2024-05-031-9/+26
| | | | * validate result when getting partial chunk and warn on failure * changelog
* assert improvements (#72)Dan Engelbrecht2024-05-035-38/+83
| | | | | - Improvement: Asserts gives an immediate ERROR log entry with callstack and reason - Improvement: Asserts flushes the log before sending error report to Sentry
* added logic to change default HTTP server implementation when running on ↵Stefan Boberg2024-05-025-6/+39
| | | | | | Wine (#71) * added logic to change default HTTP server implementation when running on Wine * added log message to inform user about potential problems when running under Wine
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-027-67/+82
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-029-24/+35
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-026-126/+482
| | | - Improvement: Batch scope for put of cache values
* fix get project files loop (#68)Dan Engelbrecht2024-04-305-57/+107
| | | | | - Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found - Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash - Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
* miscellaneous minor bugfixes (#66)v5.5.0Stefan Boberg2024-04-264-11/+16
| | | | | | | | this change addresses some TSAN warnings for improved robustness and less TSAN noise - Added dedicated timer for EnqueueStateExitFlagTimer - Made log formatter `fullformatter` output consistent time stamps across threads - Made Linux/Mac event implementation TSAN clean
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-264-54/+144
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-2613-68/+155
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* fix oplog import during gcv2 (#62)v5.5.0-pre3v5.5.0-pre2Dan Engelbrecht2024-04-253-114/+138
| | | | - Bugfix: Always pre-cache oplog when creating project store GCv2 referencer - Bugfix: Correctly capture attachments imported with oplog to void them being GCd before oplog is written
* zenserverprocess hardening (#61)Dan Engelbrecht2024-04-254-25/+53
| | | | * verify running process before creating event * make sure we don't signal/wait for a zenserver instance that we did not wait for to get ready
* iterate cas chunks (#59)Dan Engelbrecht2024-04-2413-140/+298
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* remove obsolete codeDan Engelbrecht2024-04-241-4/+0
|
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-246-6/+41
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* Bugfix: Only disable oplog update capture if we have started it (#58)Dan Engelbrecht2024-04-241-1/+6
|
* set gc-v2 as default (#56)Dan Engelbrecht2024-04-221-1/+1
| | | - Change: GCv2 is now the default option, use `--gc-v2=false` to fall back to GCv1
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-2214-177/+653
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* fix LogRemoteStoreStatsDetails (#53)Dan Engelbrecht2024-04-201-2/+3
|
* import oplog improvements (#54)Dan Engelbrecht2024-04-2010-205/+384
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* capture zenserver output on error (#51)Dan Engelbrecht2024-04-186-30/+111
| | | | * capture spawned server output and output on launch error * fix logging and launch validation in tests
* safer oplog import (#52)de/safer-oplog-importDan Engelbrecht2024-04-185-76/+201
| | | | * reference cache gc update capture * When importing oplogs we now import all attachments first and (optionally clean) write the oplog on success
* improved lock file handling (#50)Dan Engelbrecht2024-04-1810-70/+317
| | | | | | | | | | | | - Feature: `zen down` - --`data-dir` to specify a data directory to deduce which zen instance to bring down - Feature: `zen attach` - --`data-dir` to specify a data directory to deduce which zen instance to attach to222 - Feature: `zen status` - --`port` filter running zen instances based on port - --`data-dir` filter running zen instances based on information in the data directory - Improvement: Trying to load a compact binary object from an empty file no longer causes access violation
* zen startup hardening (#49)Dan Engelbrecht2024-04-1710-83/+494
| | | | | | | | | | | | | - Feature: `zen up` command improvements - --`port` allows you to specify a base port when starting an instance - --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable - Feature: `zen down` - --`port` allows you to specify a base port when shutting down an instance - --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable - --`force` if regular shutdown fails it tries to find a running zenserver.exe process and terminate it - If it fails to attach to the running server it now waits for it to exit when setting the RequestExit shared memory flag - Improvement: zenserver now checks the RequestExit flag in the shared memory and exist gracefully if it is set - Improvement: When adding a sponsor process to a running zenserver instance, we wait for it to be picked up from the shared memory section to determine success/fail
* gc v2 disk freed space fix and oplog stats report improvement (#45)Dan Engelbrecht2024-04-152-87/+146
| | | | | - Bugfix: Correctly calculate size freed/data moved from blocks in GCv2 - Improvement: Reduced details in remote store stats for oplog export/import to user - Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
* Validate input buffer size when trying to parse package message (#47)Dan Engelbrecht2024-04-132-22/+37
| | | | * add validation of input buffer size when trying to parse package message * avoid doing memcopy when parsing package message
* typo fix in gc.lightweightintervalsecondsStefan Boberg2024-04-111-1/+1
|
* remote project store stats (#44)Dan Engelbrecht2024-04-1011-104/+338
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* Use proper API when checking oplog export blob existance in Jupiter (#42)Dan Engelbrecht2024-04-091-3/+4
|
* precommitzousar2024-04-041-1/+6
|
* Add optional gcpath param to import-oplog cmdzousar2024-04-042-2/+10
|
* Categorize the clean argument on oplog-import as a general argzousar2024-04-042-3/+3
|
* improved assert (#37)Dan Engelbrecht2024-04-0445-167/+591
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* hardening parsepackagemessage (#38)Dan Engelbrecht2024-04-042-37/+74
| | | * hardening of ParsePackageMessage and extended details when malformed attachments are detected
* validate rpc chunk responses (#36)Dan Engelbrecht2024-04-034-20/+34
| | | * Validate size of found chunks in cas/cache
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-037-211/+114
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* compressed header reading opt (#33)Dan Engelbrecht2024-04-031-52/+86
| | | * refactor so we don't have to re-read data from source to get block sizes
* Fix sentry using wrong folder path when data path contains non-anscii ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | characters (#32) * Fix sentry using wrong folder path when data path contains non-ascii characters
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-288-68/+198
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* add "fieldnames" query param for GetProjectFiles/GetProjectChunkInfos (#29)Dan Engelbrecht2024-03-283-23/+137
| | | | | | | | - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/chunkinfos` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `rawhash` and `rawsize` (translates to `?fieldnames=id,rawhash,rawsize`) - Use `?fieldnames=*` to get all the fields - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/files` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `clientpath` and `serverpath` (translates to `?fieldnames=id,clientpath,serverpath`), `filter=client` only applies if `fieldnames` is not given as a parameter - Use `?fieldnames=*` to get all the fields
* Faster reading of compressed buffer headers by not materializing entire ↵Dan Engelbrecht2024-03-281-22/+52
| | | | source buffer (#28)
* Get raw size for compressed chunks correctly for ↵Dan Engelbrecht2024-03-271-1/+7
| | | | `/prj/{project}/oplog/{log}/chunkinfos` (#27)
* make existing block log clearerDan Engelbrecht2024-03-261-1/+1
|
* fix order of parameters in block check result message (#26)Dan Engelbrecht2024-03-261-1/+1
| | | * fix order of parametes in block check result message
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-27 07:08:44 +0000