aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/projectstore/jupiterremoteprojectstore.cpp
Commit message (Collapse)AuthorAgeFilesLines
* move remoteproject to remotestorelib (#542)Dan Engelbrecht2025-10-031-402/+0
| | | * move remoteproject code to remotestorelib
* Added `--oidctoken-exe-unattended` to`zen builds` and `zen oplog-download` ↵Dan Engelbrecht2025-09-221-2/+3
| | | | command to use unattended mode when launching oidc-token.exe (#506)
* add validation of compact binary payloads before reading them (#483)Dan Engelbrecht2025-09-041-3/+8
| | | * add validation of compact binary payloads before reading them
* oplog output improvements (#474)Dan Engelbrecht2025-08-221-1/+2
| | | | | | * include session id in oplog import/export output * use Mbit/sec for transfer rates * fix elapsed time conversion in buildremotprojectstore * changelog
* list build part content (#462)Dan Engelbrecht2025-08-111-2/+2
| | | | | | | | | | | - Feature: Added `zen build ls` option to list the content of a build part(s) - Build source is specified using one of the following options - `--cloud-url` cloud artifact URL to build - `--host` or `--override-host`, `--namespace`, `--bucket` and `--buildid` - `--filestorage`, `--namespace`, `--bucket` and `--buildid` - `--build-part-name` to specify a particular build part(s) in the build - `--wildcard` windows style wildcard (using * and ?) to match file paths to include - `--exclude-wildcard` windows style wildcard (using * and ?) to match file paths to exclude. Applied after --wildcard include filter - Improvement: Added `--quiet` option to zen `builds` commands to suppress non-essential output
* builds allow redirect option (#379)Dan Engelbrecht2025-05-051-16/+11
| | | * add --allow-redirect to zen builds upload/download
* `zen oplog-export`, `zen oplog-import` for `--url` (cloud) and `--builds` ↵Dan Engelbrecht2025-04-031-1/+9
| | | | (builds) option now has `--oidctoken-exe-path` to let zen run the OidcToken executable to get and refresh authentication token (#340)
* builds upload command (#278)Dan Engelbrecht2025-02-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Feature: **EXPERIMENTAL** New `zen builds` command to list, upload and download folders to Cloud Build API - `builds list` list available builds (**INCOMPLETE - FILTERING MISSING**) - `builds upload` upload a folder to Cloud Build API - `--local-path` source folder to upload - `--create-build` creates a new parent build object (using the object id), if omitted a parent build must exist and `--build-id` must be given - `--build-id` an Oid in hex form for the Build identifier to use - omit to have the id auto generated - `--build-part-id` and Oid in hex form for the Build Part identifier for the folder - omit to have the id auto generated - `--build-part-name` name of the build part - if omitted the name of the leaf folder name give in `--local-path` - `--metadata-path` path to a json formatted file with meta data information about the build. Meta-data must be provided if `--create-build` is set - `--metadata` key-value pairs separated by ';' with build meta data for the build. (key1=value1;key2=value2). Meta-data must be provided if `--create-build` is set - `--clean` ignore any existing blocks of chunk data and upload a fresh set of blocks - `--allow-multipart` enable usage of multi-part http upload requests - `--manifest-path` path to text file listing files to include in upload. Exclude to upload everything in `--local-path` - `builds download` download a folder from Cloud Build API (**INCOMPLETE - WILL WIPE UNTRACKED DATA FROM TARGET FOLDER**) - `--local-path` target folder to download to - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` a comma separated list of Oid in hex for the build part identifier(s) to download - mutually exclusive to `--build-part-name` - `--build-part-name` a comma separated list of names for the build part(s) to download - if omitted the name of the leaf folder name give in `--local-path` - `--clean` deletes all data in target folder before downloading (NON-CLEAN IS NOT IMPLEMENTED YET) - `--allow-multipart` enable usage of multi-part http download reqeusts - `builds diff` download a folder from Cloud Build API - `--local-path` target folder to download to - `--compare-path` folder to compare target with - `--only-chunked` compare only files that would be chunked - `builds fetch-blob` fetch and validate a blob from remote store - `--build-id` an Oid in hex form for the Build identifier to use - `--blob-hash` an IoHash in hex form identifying the blob to download - `builds validate part` fetch a build part and validate all referenced attachments - `--build-id` an Oid in hex form for the Build identifier to use - `--build-part-id` an Oid in hex for the build part identifier to validate - mutually exclusive to `--build-part-name` - `--build-part-name` a name for the build part to validate - mutually exclusive to `--build-part-id` - `builds test` a series of operation that uploads, downloads and test various aspects of incremental operations - `--local-path` source folder to upload - Options for Cloud Build API remote store (`list`, `upload`, `download`, `fetch-blob`, `validate-part`) - `--url` Cloud Builds URL - `--assume-http2` assume that the builds endpoint is a HTTP/2 endpoint skipping HTTP/1.1 upgrade handshake - `--namespace` Builds Storage namespace - `--bucket` Builds Storage bucket - Authentication options for Cloud Build API - Auth token - `--access-token` http auth Cloud Storage access token - `--access-token-env` name of environment variable that holds the Http auth Cloud Storage access token - `--access-token-path` path to json file that holds the Http auth Cloud Storage access token - OpenId authentication - `--openid-provider-name` Open ID provider name - `--openid-provider-url` Open ID provider url - `--openid-client-id`Open ID client id - `--openid-refresh-token` Open ID refresh token - `--encryption-aes-key` 256 bit AES encryption key for storing OpenID credentials - `--encryption-aes-iv` 128 bit AES encryption initialization vector for storing OpenID credentials - OAuth authentication - `--oauth-url` OAuth provier url - `--oauth-clientid` OAuth client id - `--oauth-clientsecret` OAuth client secret - Options for file based remote store used for for testing purposes (`list`, `upload`, `download`, `fetch-blob`, `validate-part`, `test`) - `--storage-path` path to folder to store builds data - `--json-metadata` enable json output in store for all compact binary objects (off by default) - Output options for all builds commands - `--plain-progress` use plain line-by-line progress output - `--verbose`
* moving and small refactor of chunk blocks to prepare for builds api (#282)Dan Engelbrecht2025-02-121-2/+2
|
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-221-48/+41
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-161-1/+1
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Builds API remote project store (#258)Dan Engelbrecht2024-12-121-25/+65
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* make sure to initialize stats in file, jupiter and zen remote stores (#198)Dan Engelbrecht2024-10-161-7/+7
| | | * make sure to initialize stats in file, jupiter and zen remote stores
* remote project store stats (#44)Dan Engelbrecht2024-04-101-3/+48
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* Use proper API when checking oplog export blob existance in Jupiter (#42)Dan Engelbrecht2024-04-091-3/+4
|
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-1/+1
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* use batch request for checking existing blocks as Jupiter is now fixed (#20)v5.4.2-pre15v5.4.2-pre14Dan Engelbrecht2024-03-231-41/+10
|
* check existance of reused blocks (#18)Dan Engelbrecht2024-03-221-0/+47
| | | | | * Add HasAttachments to RemoteProjectStore so we can query if attachment blocks actually exist * use individual requests for compressed blob check in Jupiter * remove weird 1000.5 to 1000.0 when converting seconds to milliseconds
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-1/+3
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-241-68/+14
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* improved errors from jupiter upstream (#636)Dan Engelbrecht2024-01-221-5/+22
| | | * get more detailed error messages from jupiter upstream
* increase jupiter oplog import/export connection timeout to 30 min (#411)Dan Engelbrecht2023-09-191-1/+1
| | | | * increase jupiter oplog import/export connection timeout to 30 min * changelog
* Add retry if FinalizeRef responds with non-empty "Needs" attachments (#409)Dan Engelbrecht2023-09-191-7/+7
| | | | | * Add retry if FinalizeRef responds with non-empty "Needs" attachments * better logging/progress report * changelog
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-131-5/+4
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* incremental oplog upload for block-based targets (#392)Dan Engelbrecht2023-09-121-40/+61
| | | | | | | * add option for base container for oplog export read base oplog and fetch known blocks * reuse blocks if a known block has 80+ % usage * changelog * better logging and added base to remotestore descriptions
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-061-9/+33
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* Make sure we always write "data" attachment hash for snapshotted oplog ↵Dan Engelbrecht2023-08-111-1/+1
| | | | | | | | | entries (#355) * Make sure we always write "data" attachment hash for snapshotted oplog entries * Make sure to add chunk mappings for files moved to attatchment in snapshot operation * fix inverted timoute for expiration (we don't want time expiry in these cases) * increase timeout for jupiter oplog in project to 3 min * changelog
* Add `response.text` to output in log when jupiter request fails (#354)Dan Engelbrecht2023-08-101-2/+2
| | | | | | * Add `response.text` to output in log when jupiter request fails * only hash jupiter ref once and keep that * Increase request timeout when uploading to Jupiter to 2 min (to handle very large attachments) * changelog
* Add `--assume-http2` option to cloud style import/export command to use a ↵Dan Engelbrecht2023-08-091-1/+2
| | | | HTTP/2 endpoint without without HTTP/1.1 upgrade (#347)
* add requested item in oplog remote op (#340)Dan Engelbrecht2023-08-011-29/+76
| | | * add more context for oplog import/export errors
* removed unnecessary cpr referenceStefan Boberg2023-06-301-4/+0
|
* fix jupiter access token for oplog upload (#330)Dan Engelbrecht2023-06-121-1/+1
| | | | * make sure to prefix auth token correctly when provided with a raw token * changelog
* minor: refcount bump eliminationStefan Boberg2023-05-251-1/+1
|
* move auth code from zenserver into zenhttp (#265)Stefan Boberg2023-05-021-1/+1
| | | this code should be usable outside of zenserver, so this moves it out into zenhttp where it can be used from lower level components
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+244
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-22 02:41:06 +0000