aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/remoteprojectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* improved assert (#37)Dan Engelbrecht2024-04-041-3/+3
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-1/+1
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* Use multithreading to fetch size/rawsize of entries in ↵Dan Engelbrecht2024-03-281-2/+2
| | | | | | `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` (#30) - Improvement: Use multithreading to fetch size/rawsize of entries in `/prj/{project}/oplog/{log}/chunkinfos` and `/prj/{project}/oplog/{log}/files` - Improvement: Add `GetMediumWorkerPool()` in addition to `LargeWorkerPool()` and `SmallWorkerPool()`
* make existing block log clearerDan Engelbrecht2024-03-261-1/+1
|
* fix order of parameters in block check result message (#26)Dan Engelbrecht2024-03-261-1/+1
| | | * fix order of parametes in block check result message
* add a limit to the number of times we attempt to finalize (#22)Dan Engelbrecht2024-03-251-81/+133
| | | | | | - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-221-24/+71
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* read jupiter token from file (#10)Dan Engelbrecht2024-03-181-5/+12
| | | | | * Add ability to specify a json file for cloud access token OidcToken.exe in UE can be asked to produce such a file with the -OutFile option * avoid division by zero when reporting progress
* special treatment large oplog attachments v2 (#5)Dan Engelbrecht2024-03-141-582/+1070
| | | | | - Bugfix: Install Ctrl+C handler earlier when doing `zen oplog-export` and `zen oplog-export` to properly cancel jobs - Improvement: Add ability to block a set of CAS entries from GC in project store - Improvement: Large attachments and loose files are now split into smaller chunks and stored in blocks during oplog export
* Save compressed large attachments to temporary files on disk (#650)Dan Engelbrecht2024-02-121-182/+216
| | | | | | | | | * Save large compressed large attachments to temporary files on disk * bump oplog block max size up to 64Mb again * Make sure CompositeBuffer::AppendBuffers actually moves inputs when it should * removed parallell execution of fetching payload for block assembly it was not actually helping and added complexity * make sure we move/release payload buffers as soon as possible * make sure we don't read in full large attachments to memory when computing hash
* compress large attachments on demand (#647)Dan Engelbrecht2024-02-051-324/+386
| | | | | | | - Improvement: Speed up oplog export by fetching/compressing big attachments on demand - Improvement: Speed up oplog export by batch-fetcing small attachments - Improvement: Speed up oplog import by batching writes of oplog ops - Improvement: Tweak oplog export default block size and embed size limit - Improvement: Add more messaging and progress during oplog import/export
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-96/+207
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-291-4/+5
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-251-21/+34
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-241-29/+29
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* oplog import/export improvements (#634)Dan Engelbrecht2024-01-231-159/+305
| | | | * improve feedback from oplog import/export * improve oplog save performance
* add --ignore-missing-attachments to oplog-import command (#637)Dan Engelbrecht2024-01-221-34/+59
|
* global thread worker pools (#577)Dan Engelbrecht2023-11-291-12/+4
| | | - Improvement: Use two global worker thread pools instead of ad-hoc creation of worker pools
* disk layer gc and error/warnings cleanup (#515)Dan Engelbrecht2023-11-081-41/+41
| | | | | | | - Improvement: Use GC reserve when writing index/manifest for a disk cache bucket when disk is low when available - Improvement: Demote errors to warning for issues that are not critical and we handle gracefully - Improvement: Treat more out of memory errors from windows as Out Of Memory errors Fixed wrong sizeof() statement for compactcas index (luckily the two structs are of same size)
* added missing includes (#504)Stefan Boberg2023-10-271-0/+2
| | | | | this change adds some includes to files which "inherit" includes from elsewhere this was exposed on another branch when removing some heavy dependencies from central headers
* faster oplog iteration (#471)Dan Engelbrecht2023-10-131-3/+3
| | | | | * use a CbObjectView instead of CbObject to avoid creating IOBufferCore instances * use BasicFileBuffer directly where possible * changelog
* Improvement: Add names to background jobs for easier debugging (#412)Dan Engelbrecht2023-09-201-17/+27
| | | | Improvement: Background jobs now temporarily sets thread name to background job name while executing Improvement: Background jobs tracks worker thread id used while executing
* Add retry if FinalizeRef responds with non-empty "Needs" attachments (#409)Dan Engelbrecht2023-09-191-250/+361
| | | | | * Add retry if FinalizeRef responds with non-empty "Needs" attachments * better logging/progress report * changelog
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-131-65/+241
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* incremental oplog upload for block-based targets (#392)Dan Engelbrecht2023-09-121-0/+90
| | | | | | | * add option for base container for oplog export read base oplog and fetch known blocks * reuse blocks if a known block has 80+ % usage * changelog * better logging and added base to remotestore descriptions
* better sorting of attachments in oplog blocks (#390)Dan Engelbrecht2023-09-111-13/+54
| | | | | - sort attachments based on (first) associated op - sort attachments based on key of op to get a more consistent order rather than order of ops written - keep attachments from same op in same block
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-56/+75
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* retry file create (#383)Dan Engelbrecht2023-09-041-2/+29
| | | | | * add retry logic when creating files * only write disk usage log if disk writes are allowed * changelog
* Add `--embedloosefiles` option to `oplog-export` (#376)Dan Engelbrecht2023-08-241-56/+221
| | | | * Add `--embedloosefiles` option to `oplog-export` which adds loose files to the export, removing need to call `oplog-snapshot` * Retain `ServerPath` in oplog when performing `oplog-snapshot`. This is a short-term fix for current incompatability with the UE cooker.
* check oplog op attachments when gathering references for GC (#363)Dan Engelbrecht2023-08-181-8/+15
| | | | | * Make sure to check oplog op attachments when gathering references for GC * Add oplog op content to error result if attachment is missing when doing `oplog-export`
* fix asserts and exceptions (#344)Dan Engelbrecht2023-08-081-0/+7
| | | | | | * Send proper error to caller of GetChunkInfo instead of assert * catch and handle exceptions when checking for state_marker * properly wait for background tasks if oplop-export fails * changelog
* changed ZEN_ERROR to ZEN_WARN to reduce Sentry noiseStefan Boberg2023-06-081-3/+8
| | | | addresses ZEN-SERVER-W5. This should not be logged as an error since the failure is propagated to the client and the failure is typically due to invalid input
* named oplog upload worker pool threadsStefan Boberg2023-05-251-1/+12
|
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+1036
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-06 04:37:36 +0000