aboutsummaryrefslogtreecommitdiff
path: root/src/zenhttp/httpclient.cpp
Commit message (Collapse)AuthorAgeFilesLines
* use new builds api for oplogs (#464)Dan Engelbrecht2025-08-121-3/+4
| | | | - Improvement: Refactored jupiter oplog export code to reuse builds jupiter wrapper classes - Improvement: If `zen builds`, `zen oplog-import` or `zen oplog-import` command fails due to a http error, the return code for the program will be set to the error/status code
* crash in composite buffer stream (#431)Dan Engelbrecht2025-06-101-1/+1
| | | * fix BufferedReadFileStream calculating buffer end wrong
* http client streaming upload (#413)Dan Engelbrecht2025-06-021-27/+283
| | | - Improvement: Add streaming upload from HttpClient to reduce I/O caused by excessive MMap usage
* handle exception with batch work (#401)Dan Engelbrecht2025-05-191-2/+4
| | | | | | | | | | | | | | | * use ParallelWork in rpc playback * use ParallelWork in projectstore * use ParallelWork in buildstore * use ParallelWork in cachedisklayer * use ParallelWork in compactcas * use ParallelWork in filecas * don't set abort flag in ParallelWork destructor * add PrepareFileForScatteredWrite for temp files in httpclient * Use PrepareFileForScatteredWrite when stream-decompressing files * be more relaxed when deleting temp files * allow explicit zen-cache when using direct host url without resolving * fix lambda capture when writing loose chunks * no delay when attempting to remove temp files
* add retry on internal error / bad gateway (#370)Dan Engelbrecht2025-04-241-1/+3
| | | * do http client retry on internal error and bad gateway
* filesystem retry fixes (#354)Dan Engelbrecht2025-04-101-5/+5
| | | | | | | | | * add more forgiving retries with filesystem * fall back to FindFirstFile if access prevents us from using GetFileAttributes * only validate hash if we have a complete payload in http client * changelog
* optional compress of block chunks (#326)Dan Engelbrecht2025-03-271-1/+1
| | | | | | - Feature: zenserver: Add command line option `--gc-buildstore-duration-seconds` to control GC life time of build store data - Improvement: ELF and MachO executable files are no longer chunked - Improvement: Compress chunks in blocks that encloses a full file (such as small executables) - Bugfix: Strip path delimiter at end of string in StringToPath
* zen build cache service (#318)Dan Engelbrecht2025-03-261-1/+25
| | | | | | | | | - **EXPERIMENTAL** `zen builds` - Feature: `--zen-cache-host` option for `upload` and `download` operations to use a zenserver host `/builds` endpoint for storing build blob and blob metadata - Feature: New `/builds` endpoint for caching build blobs and blob metadata - `/builds/{namespace}/{bucket}/{buildid}/blobs/{hash}` `GET` and `PUT` method for storing and fetching blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/putBlobMetadata` `POST` method for storing metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/getBlobMetadata` `POST` method for fetching metadata about blobs - `/builds/{namespace}/{bucket}/{buildid}/blobs/exists` `POST` method for checking existance of blobs
* partial block fetch (#298)Dan Engelbrecht2025-03-071-66/+95
| | | | | - Improvement: Do partial requests of blocks if not all of the block is needed - Improvement: Better progress/statistics on download - Bugfix: Ensure that temporary folder for Jupiter downloads exists during verify phase
* limit and validate responses before logging the text (#292)Stefan Boberg2025-03-041-0/+46
| | | Improvement: When logging HTTP responses, the body is now sanity checked to ensure it is human readable, and the length of the output is capped to prevent inadvertent log bloat
* refactor use chunk sequence download (#291)Dan Engelbrecht2025-03-031-0/+5
| | | | | | | | | | | | | | | | | | | | | | * work on chunk sequences on download, not paths * write chunksequences to .tmp file and move when complete * cleanup * Added on the fly validation `zen builds download` of files built from smaller chunks as each file is completed Added `--verify` option to `zen builds upload` to verify all uploaded data once entire upload is complete Added `--verify` option to `zen builds download` to verify all files in target folder once entire download is complete Fixed/improved progress updated Multithreaded part validation * added rates to Write Chunks task * b/s -> bits/s * dont validate partial content as complete payload * handle legacy c# builds
* improvements and infrastructure for upcoming builds api command line (#284)Dan Engelbrecht2025-02-251-4/+23
| | | | | | | | | | | | | | * add modification tick to filesystem traversal * add ShowDetails option to ProgressBar * log callstack if we terminate process * handle chunking if MaxSize > 1MB * BasicFile write helpers and WriteToTempFile simplifications * bugfix for CompositeBuffer::IterateRange when using DecompressToComposite for actually comrpessed data revert of earlier optimization * faster compress/decompress for large disk-based files * enable progress feedback in IoHash::HashBuffer * add payload validation in HttpClient::Get * fix range requests (range is including end byte) * remove BuildPartId for blob/block related operations in builds api
* improved builds api interface in jupiter (#281)Dan Engelbrecht2025-02-121-137/+174
| | | | | * multipart upload/download iterface in jupiter * review fixes
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-161-2/+2
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* improved payload validation in HttpClient (#259)Dan Engelbrecht2024-12-101-18/+60
| | | | | * improved payload validation in HttpClient * separate error messages for FromCompressed and Decompress * refactor so we can do retry if decompression of block fails
* make sure we don't throw exception from worker thread (#247)Dan Engelbrecht2024-11-281-2/+2
| | | | | | * Make sure we don't throw exception from worker thread * secure async project flush * secure workspaces * spelling
* add validation of payload responses in http client (#240)Dan Engelbrecht2024-11-271-220/+292
| | | if response payload does not validate properly do a retry if applicable
* workspace share security (#192)Dan Engelbrecht2024-10-231-1/+4
| | | | | | | - Improvement: Reworked workspace shares to be more secure. Workspaces and workspace shares can only be created using the `zen workspace` command, the http endpoint is disabled unless zenserver is started with the `--workspaces-allow-changes` option enabled. - Each workspace are now configured via a `zenworkspaceconfig.json` file in the root of each workspace - A workspace can allow shares to be created via the http interface if the workspace is created with the `--allow-share-create-from-http` option enabled - A new http endpoint at `/ws` - issuing a `Get` operation will get you a list of workspaces - A new http endpoint at `/ws/refresh` - issuing a `Get` will make zenserver scan for edits in workspaces and workspace shares
* safer path from handle (#195)Dan Engelbrecht2024-10-161-2/+12
| | | * remove PathFromHandle that throws to give better context on failures
* Add zenserver session id to Sentry context (#133)Dan Engelbrecht2024-08-221-3/+1
|
* workspace shares (#84)Dan Engelbrecht2024-05-291-0/+19
| | | Feature: New 'workspaces' service which allows a user to share a local folder via zenserver. A workspace can have mulitple workspace shares and they provie an HTTP API that is compatible with the project oplog HTTP API. Workspaces and shares are preserved between runs. Workspaces feature is disabled by default - enable with --workspaces-enabled option when launching zenserver.
* import oplog improvements (#54)Dan Engelbrecht2024-04-201-70/+158
| | | | | | | | | | | * report down/up transfer speed during progress * add disk buffering in http client * offload block decoding and chunk writing form network worker pool threads add block hash verification for blocks recevied at oplog import * separate download-latch from write-latch to get more accurate download speed * check headers when downloading with http client to go directly to file writing for large payloads * we must clear write callback even if we only provide it as an argument to the Download() call * make timeout optional in AddSponsorProcess * check return codes when creating windows threadpool
* improved assert (#37)Dan Engelbrecht2024-04-041-1/+1
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* zenremoteprojectstore with httpclient (#35)Dan Engelbrecht2024-04-031-18/+32
| | | | | | - Bugfix: Fix log of Success/Failure for oplog import - Improvement: Use HttpClient when doing oplog export/import with a zenserver as a remote target. Includes retry logic - Improvement: Increase the retry count to 4 (5 attempts in total) when talking to Jupiter for oplog export/import
* Actually thow exception if we can't parse the jobid when starting async job ↵Dan Engelbrecht2024-01-311-2/+7
| | | | | | (#645) Return system error code on exception throw in zen command Clean up HttpClient::Response::ErrorMessage to remove redundant ": "
* improve oplog export logging (#644)Dan Engelbrecht2024-01-311-9/+9
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-291-1/+1
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-251-1/+4
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Use proper format for range request header (#640)Dan Engelbrecht2024-01-241-4/+10
| | | | Clear header callback after use Use separate temp-vector for headers
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-241-137/+393
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-201-1/+1
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* spdlog implementation hiding (#498)Stefan Boberg2023-11-061-5/+11
| | | | | | | | | this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future. As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link. Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future. The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
* Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-221-2/+2
| | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
* Make sure error logging or destructors don't throw exception when trying to ↵Dan Engelbrecht2023-09-121-14/+21
| | | | | get file name from handle (#393) - Bugfix: Make sure error logging or destructors don't throw exception when trying to get file name from handle
* add console logging to zen command (#389)Dan Engelbrecht2023-09-081-4/+10
| | | | properly set trace log level if IsVerbose add log category to http client
* Extend http client (#387)Dan Engelbrecht2023-09-081-74/+437
| | | * extend http client with configuration, headers, parameters and disk streaming upload/download
* oplog mirror support (#367)Stefan Boberg2023-08-211-1/+22
| | | | feature: added oplog-mirror command. this can be invoked to export oplog contents to corresponding files
* Additional trace instrumentation (#312)Stefan Boberg2023-05-161-0/+17
| | | | | | | | | * added trace instrumentation to upstreamcache * added asio trace instrumentation * added trace annotations for project store * added trace annotations for BlockStore * added trace annotations for HttpClient * added trace annotations for CAS/GC
* some HttpClient changes eliminating some cpr helpersStefan Boberg2023-05-151-18/+6
|
* Better defaults for zen cli (#302)Stefan Boberg2023-05-151-0/+46
| | | | | added ZenCmdBase::ResolveTargetHostSpec - this is a helper which provides better default behaviour for commands which interact with a local zen server instance. more specifically it picks a default based on which processes are actually running on the local machine this change also wires up the Scrub command along with some required HttpClient improvements
* 247 complete httpclient implementation (#269)Stefan Boberg2023-05-051-18/+250
| | | | | | | | | * implemented HttpClient connection pooling * implemented missing verbs * added response helpers (CbObject/CbPackage/text) * added RwLock::WithSharedLock and RwLock::WithExclusiveLock * added some noexcept annotations on RwLock * removed CPR dependency in httpclient.h
* moved source directories into `/src` (#264)Stefan Boberg2023-05-021-0/+176
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-11 13:29:40 +0000