aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
...
* hardening read of corrupt oplog (#98)Dan Engelbrecht2024-08-061-24/+50
| | | * Add extra validation of oplog entries when reading oplog
* improve mutex startup error (#96)Dan Engelbrecht2024-06-141-2/+7
| | | * separate collision of shared mutex from failure to create shared mutex
* Make sure we monitor for new project, oplogs, namespaces and buckets during ↵Dan Engelbrecht2024-06-132-117/+415
| | | | | | GCv2 (#93) - Bugfix: Make sure we monitor and include new project/oplogs created during GCv2 - Bugfix: Make sure we monitor and include new namespaces/cache buckets created during GCv2
* workspaces config and fixes (#92)Dan Engelbrecht2024-06-112-26/+42
| | | | * fix alias request capture * use single config file for workspaces
* workspace share aliases (#91)Dan Engelbrecht2024-06-042-199/+438
| | | | | | | - Add `zen workspace-share` `--root-path` option - the root local file path of the workspace - if given it will automatically create the workspace before creating the share. If `--workspace` is omitted, an id will be generated from the `--root-path` parameter - Add `/ws/share/{alias}/` endpoint - a shortcut to `/ws/{workspace_id}/{share_id}/` based endpoints using the alias for a workspace share - Add `--alias` option to replace `--workspace` and `--share` options for `workspace-share` zen commands - Rename `zen workspace create` `folder` option to `root-path` - Rename `zen workspace create` `folder` option to `share-path`
* Use a smaller thread pool for network operations when doing oplog import to ↵Dan Engelbrecht2024-05-301-1/+1
| | | | | reduce risk NIC/router failure (#89) Medium worker pool now uses a minimum of 2 threads (up from 1)
* cache optimizations (#88)Dan Engelbrecht2024-05-302-5/+5
| | | | | * message formatting optimizations * bump iostorecompression small value threshold to 1MB
* workspaces review feedbackDan Engelbrecht2024-05-291-1/+1
|
* workspace shares (#84)Dan Engelbrecht2024-05-296-1/+908
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* don't use "error:" in log messages unless there is an error (#87)Dan Engelbrecht2024-05-271-2/+2
| | | Improvement: Don't use "error:" in log messages unless there is an error as Horde CI will pick up that log line and interpret it as an error
* fix iterate chunks crash (#86)Dan Engelbrecht2024-05-271-0/+1
| | | * fix worklatch count in Oplog::IterateChunks
* assert improvements (#72)Dan Engelbrecht2024-05-031-5/+5
| | | | | - Improvement: Asserts gives an immediate ERROR log entry with callstack and reason - Improvement: Asserts flushes the log before sending error report to Sentry
* added logic to change default HTTP server implementation when running on ↵Stefan Boberg2024-05-022-3/+16
| | | | | | Wine (#71) * added logic to change default HTTP server implementation when running on Wine * added log message to inform user about potential problems when running under Wine
* fix zero size attachment replies (#69)Dan Engelbrecht2024-05-023-4/+20
| | | | - Bugfix: Don't try to respond with zero size partial cache value when partial size is zero - Improvement: Added more validation of data read from cache / cas
* use write and move in place for safer writing of files (#70)Dan Engelbrecht2024-05-023-7/+7
| | | * use write and move in place for safer writing of files
* batch cache put (#67)Dan Engelbrecht2024-05-021-5/+11
| | | - Improvement: Batch scope for put of cache values
* fix get project files loop (#68)Dan Engelbrecht2024-04-301-23/+20
| | | | | - Bugfix: Remove extra loop causing GetProjectFiles for project store to find all chunks once for each chunk found - Bugfix: Don't capture ChunkIndex variable in CasImpl::IterateChunks by reference as it causes crash - Improvement: Make FileCasStrategy::IterateChunks (optionally) multithreaded (improves GetProjectFiles performance)
* miscellaneous minor bugfixes (#66)v5.5.0Stefan Boberg2024-04-262-2/+3
| | | | | | | | this change addresses some TSAN warnings for improved robustness and less TSAN noise - Added dedicated timer for EnqueueStateExitFlagTimer - Made log formatter `fullformatter` output consistent time stamps across threads - Made Linux/Mac event implementation TSAN clean
* oplog iterate chunks content type (#65)Dan Engelbrecht2024-04-263-51/+120
| | | | - Bugfix: Properly set content type of chunks fetch from CidStore - Improvement: Add IterateChunks(std::span<Oid>) for better performance in get oplog
* use direct file access for large file hash (#63)Dan Engelbrecht2024-04-261-4/+20
| | | - Improvement: Refactor `IoHash::HashBuffer` and `BLAKE3::HashBuffer` to not use memory mapped files. Performs better and saves ~10% of oplog export time on CI
* fix oplog import during gcv2 (#62)v5.5.0-pre3v5.5.0-pre2Dan Engelbrecht2024-04-253-114/+138
| | | | - Bugfix: Always pre-cache oplog when creating project store GCv2 referencer - Bugfix: Correctly capture attachments imported with oplog to void them being GCd before oplog is written
* iterate cas chunks (#59)Dan Engelbrecht2024-04-242-28/+30
| | | - Improvement: Reworked GetChunkInfos in oplog store to reduce disk thrashing and improve performance
* safer gcv2 on error (#60)Dan Engelbrecht2024-04-241-4/+13
| | | - Bugfix: Harden GCv2 when errors occur and gracefully abort GC operation on error
* Bugfix: Only disable oplog update capture if we have started it (#58)Dan Engelbrecht2024-04-241-1/+6
|
* set gc-v2 as default (#56)Dan Engelbrecht2024-04-221-1/+1
| | | - Change: GCv2 is now the default option, use `--gc-v2=false` to fall back to GCv1
* InsertChunks for CAS store (#55)Dan Engelbrecht2024-04-224-103/+176
| | | - Improvement: Add batching when writing multiple small chunks to block store - decreases I/O load significantly on oplog import
* fix LogRemoteStoreStatsDetails (#53)Dan Engelbrecht2024-04-201-2/+3
|
* import oplog improvements (#54)Dan Engelbrecht2024-04-205-129/+202
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* safer oplog import (#52)de/safer-oplog-importDan Engelbrecht2024-04-184-60/+194
| | | | * reference cache gc update capture * When importing oplogs we now import all attachments first and (optionally clean) write the oplog on success
* improved lock file handling (#50)Dan Engelbrecht2024-04-181-12/+10
| | | | | | | | | | | | - Feature: `zen down` - --`data-dir` to specify a data directory to deduce which zen instance to bring down - Feature: `zen attach` - --`data-dir` to specify a data directory to deduce which zen instance to attach to222 - Feature: `zen status` - --`port` filter running zen instances based on port - --`data-dir` filter running zen instances based on information in the data directory - Improvement: Trying to load a compact binary object from an empty file no longer causes access violation
* zen startup hardening (#49)Dan Engelbrecht2024-04-173-6/+55
| | | | | | | | | | | | | - Feature: `zen up` command improvements - --`port` allows you to specify a base port when starting an instance - --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable - Feature: `zen down` - --`port` allows you to specify a base port when shutting down an instance - --`base-dir` allows you to specify a base directory for the zenserver executable if it is not located next to the zen.exe executable - --`force` if regular shutdown fails it tries to find a running zenserver.exe process and terminate it - If it fails to attach to the running server it now waits for it to exit when setting the RequestExit shared memory flag - Improvement: zenserver now checks the RequestExit flag in the shared memory and exist gracefully if it is set - Improvement: When adding a sponsor process to a running zenserver instance, we wait for it to be picked up from the shared memory section to determine success/fail
* gc v2 disk freed space fix and oplog stats report improvement (#45)Dan Engelbrecht2024-04-151-77/+131
| | | | | - Bugfix: Correctly calculate size freed/data moved from blocks in GCv2 - Improvement: Reduced details in remote store stats for oplog export/import to user - Improvement: Transfer speed for oplog export/import is now an overall number rather than average of speed per single request
* typo fix in gc.lightweightintervalsecondsStefan Boberg2024-04-111-1/+1
|
* remote project store stats (#44)Dan Engelbrecht2024-04-109-39/+233
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* Use proper API when checking oplog export blob existance in Jupiter (#42)Dan Engelbrecht2024-04-091-3/+4
|
* improved assert (#37)Dan Engelbrecht2024-04-0413-49/+46
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-035-193/+74
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* Fix sentry using wrong folder path when data path contains non-anscii ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | characters (#32) * Fix sentry using wrong folder path when data path contains non-ascii characters
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-283-60/+164
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* add "fieldnames" query param for GetProjectFiles/GetProjectChunkInfos (#29)Dan Engelbrecht2024-03-283-23/+137
| | | | | | | | - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/chunkinfos` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `rawhash` and `rawsize` (translates to `?fieldnames=id,rawhash,rawsize`) - Use `?fieldnames=*` to get all the fields - Improvement: It is now possible to control which fields to include in `/prj/{project}/oplog/{log}/files` request by adding a comma delimited list of filed names for `fieldnames` parameter - Default fields are: `id`, `clientpath` and `serverpath` (translates to `?fieldnames=id,clientpath,serverpath`), `filter=client` only applies if `fieldnames` is not given as a parameter - Use `?fieldnames=*` to get all the fields
* Get raw size for compressed chunks correctly for ↵Dan Engelbrecht2024-03-271-1/+7
| | | | `/prj/{project}/oplog/{log}/chunkinfos` (#27)
* make existing block log clearerDan Engelbrecht2024-03-261-1/+1
|
* fix order of parameters in block check result message (#26)Dan Engelbrecht2024-03-261-1/+1
| | | * fix order of parametes in block check result message
* add filter to projectstore entries request (#25)Dan Engelbrecht2024-03-261-6/+65
| | | | | | | * Add HttpServerRequest::Decode to decode http request parameters * Add support to filter projectstore `entries` request using the `fieldfilter` where the wanted fields are comma (,) delimited * Add support for responding with compressed payloads for projectstore `entries` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `files` requests by adding AcceptType `compressed-binary` to the request header * Add support for responding with compressed payloads for projectstore `chunkinfo` requests by adding AcceptType `compressed-binary` to the request header
* remove redundant json11 includesStefan Boberg2024-03-262-2/+0
|
* consistent paths encoding (#24)Dan Engelbrecht2024-03-254-49/+51
| | | | * Don't encode filesystem path to UTF8 unless stored in compactbinary string * Be consistent where we encode/decode paths to UTF8
* add a limit to the number of times we attempt to finalize (#22)Dan Engelbrecht2024-03-251-81/+133
| | | | | | - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import
* use batch request for checking existing blocks as Jupiter is now fixed (#20)v5.4.2-pre15v5.4.2-pre14Dan Engelbrecht2024-03-231-41/+10
|
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-225-43/+167
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* harden attach sponsor process (#14)Dan Engelbrecht2024-03-213-25/+37
| | | | | - Improvement: Delay exiting due to no sponsor processes by one second to handle race conditions - Improvement: Safer IsProcessRunning check - Improvement: make sure we can RequestApplicationExit safely from any thread
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-13 08:14:25 +0000