aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/jupiterremoteprojectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-1/+1
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* use batch request for checking existing blocks as Jupiter is now fixed (#20)v5.4.2-pre15v5.4.2-pre14Dan Engelbrecht2024-03-231-41/+10
|
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-221-0/+47
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-1/+3
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-241-68/+14
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* improved errors from jupiter upstream (#636)Dan Engelbrecht2024-01-221-5/+22
| | | * get more detailed error messages from jupiter upstream
* increase jupiter oplog import/export connection timeout to 30 min (#411)Dan Engelbrecht2023-09-191-1/+1
| | | | * increase jupiter oplog import/export connection timeout to 30 min * changelog
* Add retry if FinalizeRef responds with non-empty "Needs" attachments (#409)Dan Engelbrecht2023-09-191-7/+7
| | | | | * Add retry if FinalizeRef responds with non-empty "Needs" attachments * better logging/progress report * changelog
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-131-5/+4
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* incremental oplog upload for block-based targets (#392)Dan Engelbrecht2023-09-121-40/+61
| | | | | | | * add option for base container for oplog export read base oplog and fetch known blocks * reuse blocks if a known block has 80+ % usage * changelog * better logging and added base to remotestore descriptions
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-9/+33
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* Make sure we always write "data" attachment hash for snapshotted oplog ↵Dan Engelbrecht2023-08-111-1/+1
| | | | | | | | | entries (#355) * Make sure we always write "data" attachment hash for snapshotted oplog entries * Make sure to add chunk mappings for files moved to attatchment in snapshot operation * fix inverted timoute for expiration (we don't want time expiry in these cases) * increase timeout for jupiter oplog in project to 3 min * changelog
* Add `response.text` to output in log when jupiter request fails (#354)Dan Engelbrecht2023-08-101-2/+2
| | | | | | * Add `response.text` to output in log when jupiter request fails * only hash jupiter ref once and keep that * Increase request timeout when uploading to Jupiter to 2 min (to handle very large attachments) * changelog
* Add `--assume-http2` option to cloud style import/export command to use a ↵Dan Engelbrecht2023-08-091-1/+2
| | | | HTTP/2 endpoint without without HTTP/1.1 upgrade (#347)
* add requested item in oplog remote op (#340)Dan Engelbrecht2023-08-011-29/+76
| | | * add more context for oplog import/export errors
* removed unnecessary cpr referenceStefan Boberg2023-06-301-4/+0
|
* fix jupiter access token for oplog upload (#330)Dan Engelbrecht2023-06-121-1/+1
| | | | * make sure to prefix auth token correctly when provided with a raw token * changelog
* minor: refcount bump eliminationStefan Boberg2023-05-251-1/+1
|
* move auth code from zenserver into zenhttp (#265)Stefan Boberg2023-05-021-1/+1
| | | this code should be usable outside of zenserver, so this moves it out into zenhttp where it can be used from lower level components
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+244
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-20 01:27:24 +0000