aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver/upstream
Commit message (Collapse)AuthorAgeFilesLines
* move all storage-related services into storage tree (#571)Stefan Boberg2025-10-147-2744/+0
| | | | | | * move all storage-related services into storage tree * move config into config/ * also move admin service into storage since it mostly has storage related functionality * header consolidation
* cacherequests helpers test only (#551)Dan Engelbrecht2025-10-031-1/+1
| | | | * don't use cacherequests utils in cache_cmd.cpp * make zenutil/cacherequests code into test code helpers only
* remove zenutil dependency in zenremotestore (#547)Dan Engelbrecht2025-10-031-2/+2
| | | | | | | | | * remove dependency to zenutil/workerpools.h from remoteprojectstore.cpp * remove dependency to zenutil/workerpools.h from buildstoragecache.cpp * remove unneded include * move jupiter helpers to zenremotestore * move parallelwork to zencore * remove zenutil dependency from zenremotestore * clean up test project dependencies - use indirect dependencies
* make cpr a HttpClient implementation detail (#517)Stefan Boberg2025-09-292-171/+73
| | | | | these changes remove cpr from anything which is not `HttpClient` internals. The goal is to eventually replace cpr with a more direct curl interface to eliminate cpr since it's proven problematic due to their development practices which frequently breaks APIs and prevents us from updating vcpkg. But this PR is limited to refactoring existing cpr code to use `HttpClient` instead.
* revert multi-cid store (#475)Dan Engelbrecht2025-08-262-22/+19
|
* per namespace/project cas prep refactor (#470)Dan Engelbrecht2025-08-202-19/+22
| | | | | | | - Refactor so we can have more than one cas store for project store and cache. - Refactor `UpstreamCacheClient` so it is not tied to a specific CidStore - Refactor scrub to keep the GC interface ScrubStorage function separate from scrub accessor functions (renamed to Scrub). - Refactor storage size to keep GC interface StorageSize function separate from size accessor functions (renamed to TotalSize) - Refactor cache storage so `ZenCacheDiskLayer::CacheBucket` implements GcStorage interface rather than `ZenCacheNamespace`
* builds allow redirect option (#379)Dan Engelbrecht2025-05-051-7/+8
| | | * add --allow-redirect to zen builds upload/download
* jupiter code cleanup (#276)Dan Engelbrecht2025-01-223-69/+56
| | | | | | | * cleanup jupiter * move jupiter files to separate folder * CloudCache -> Jupiter * split up jupiter files * kill redundant JupiterAccessTokenProvider
* move basicfile.h/cpp -> zencore (#273)Dan Engelbrecht2025-01-165-923/+6
| | | | | | move jupiter.h/cpp -> zenutil move packageformat.h/.cpp -> zenhttp zenutil now depends on zenhttp instead of the inverse
* Builds API remote project store (#258)Dan Engelbrecht2024-12-122-0/+199
| | | Feature: zen command oplog-export and oplog-import now supports --builds remote target using the Jupiter builds API
* split zencore/memory.h -> memoryview.h, memcmp.h (#228)Stefan Boberg2024-11-251-1/+1
| | | | | | | minor clean-up `zencore/memory.h` used to contain a variety of things including `Malloc` support along with `MemoryView` etc since the memory allocator stuff moved into `zencore/memory/memory.h` there was basically only `MemoryView` and `MemCmp` in there which seemed better to split out into separate headers to avoid overloading `memory.h`
* remote project store stats (#44)Dan Engelbrecht2024-04-103-13/+25
| | | | | * add remote oplog store statistics * block chunking when uploading oplog to zenserver (mirroring) * make sure we can move temporary dechunked file into cas store
* improved assert (#37)Dan Engelbrecht2024-04-041-10/+10
| | | | - Improvement: Add file and line to ASSERT exceptions - Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
* improve oplog export logging (#644)Dan Engelbrecht2024-01-312-13/+4
| | | | | | - Improvement: More details in oplog import/export logs - Improvement: Switch from Download to Get when fetching Refs from Jupiter as they can't be resumed anyway and streaming to disk is redundant - Bugfix: Make sure we clear read callback when doing Put in HttpClient to avoid timeout due to not sending data when reusing sessions - Bugfix: Respect `--ignore-missing-attachments` in `oplog-export` command when loose file is missing on disk
* fix response error conversion (#643)Dan Engelbrecht2024-01-291-1/+1
| | | | * make sure we properly convert compact-binary results to text when receiving errors * log fix
* add ignore-missing-attachments option to oplog export (debugging tool) (#641)Dan Engelbrecht2024-01-251-1/+3
| | | | | | | * add ignore-missing-attachments option to oplog export (debugging tool) * add more status codes to do retry for in http client * add missing X-Jupiter-IoHash header for jupiter PutRef * reduce oplog block size to reduce amount of redundant chunks to download * improved logging
* Add retry with optional resume logic to HttpClient::Download (#639)Dan Engelbrecht2024-01-242-936/+127
| | | | | | | - Improvement: Refactored Jupiter upstream to use HttpClient - Improvement: Added retry and resume logic to HttpClient - Improvement: Added authentication support to HttpClient - Improvement: Clearer logging in GCV2 compact of FileCas/BlockStore - Improvement: Size details in oplog import logging
* improved errors from jupiter upstream (#636)Dan Engelbrecht2024-01-221-0/+50
| | | * get more detailed error messages from jupiter upstream
* separate RPC processing from HTTP processing (#626)Stefan Boberg2023-12-203-113/+25
| | | | | | * moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore * move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp * introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
* move cachedisklayer and structuredcachestore into zenstore (#624)Stefan Boberg2023-12-192-2/+2
|
* various TSAN/ASAN/LeakAnalyzer fixes (#622)Stefan Boberg2023-12-191-0/+5
| | | | | | | | | * fix JobQueue test threading issue. The inner job queued with `QueueJob` would reference `I` from inside the captured closure which would subsequently disappear * made sure application exit is thread safe * don't try to access string data out of bounds * keep-alive flag is accessed from multiple threads * fix memory leaks in Zen upstream client code * TSAN fixes for Event
* factored out some compiler definitions etc into zenbase (#517)Stefan Boberg2023-11-071-1/+1
| | | | | this is a header-only library which mostly contains definitions to support different platforms and compilers. It is part of the zen codebase but is intended to be consumable separately to zenbase etc to support standalone transport plug-ins and similar.
* spdlog implementation hiding (#498)Stefan Boberg2023-11-063-21/+16
| | | | | | | | | this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future. As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link. Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future. The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
* remove legacy compute interfaces (#461)Stefan Boberg2023-10-116-2227/+15
| | | | | * removed legacy compute code, which will be replaced with a new implementation in the future * also updated references to Jupiter storage
* chache upstream stats improved (#426)Dan Engelbrecht2023-09-271-10/+16
| | | | * cache upstream stats improved * fix exit for monitor thread
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-062-46/+126
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* skip upstream logic early if we have no upstream endpoints (#359)Dan Engelbrecht2023-08-172-0/+8
| | | | | * Skip upstream logic early if we have not upstream endpoints * make cache store logging of CbObjects async * changelog
* Add `response.text` to output in log when jupiter request fails (#354)Dan Engelbrecht2023-08-102-34/+70
| | | | | | * Add `response.text` to output in log when jupiter request fails * only hash jupiter ref once and keep that * Increase request timeout when uploading to Jupiter to 2 min (to handle very large attachments) * changelog
* add extened info in log on all jupiter operation (#350)Dan Engelbrecht2023-08-091-7/+273
|
* Add `--assume-http2` option to cloud style import/export command to use a ↵Dan Engelbrecht2023-08-092-2/+9
| | | | HTTP/2 endpoint without without HTTP/1.1 upgrade (#347)
* use streaming read for PutCompressedBlob if source is single file (#338)Dan Engelbrecht2023-08-091-1/+18
| | | | * use streaming read for PutCompressedBlob if source is single file * changelog
* add more logging on jupiter fail (#345)Dan Engelbrecht2023-08-081-0/+124
| | | | * add more details in log if jupiter operations fail * changelog
* make sure to set error code to zero on successDan Engelbrecht2023-06-161-1/+1
|
* Improve resonse messages from jupiter upstream (#333)Dan Engelbrecht2023-06-152-210/+81
| | | | | | * Improve resonse messages from jupiter upstream changelog * simplify response parsing in jupiter upstream * changelog
* added thread names to timer, upstream monitorStefan Boberg2023-06-071-0/+2
| | | | also altered http-asio thread naming scheme
* cache log sessionid (#297)Stefan Boberg2023-05-232-1/+2
| | | | | | | | | | | * implemented structured cache logging to be used as audit trail to help analyse potential cache pollution/corruption * added common header to all known log targets * made Oid::operator bool explicit to avoid logging/text format mishaps * HttpClient::operator bool -> explicit * changed cache logs to not rotate on start in order to retain more history * added CacheRequestContext * properly initialize request context * log session id and request id on zencacehstore get/put * changelog
* Restructured structured cache store (#314)Stefan Boberg2023-05-171-1/+1
| | | | This change separates out the disk and memory storage strategies into separate cpp/h files to improve maintainability.
* Additional trace instrumentation (#312)Stefan Boberg2023-05-161-8/+18
| | | | | | | | | * added trace instrumentation to upstreamcache * added asio trace instrumentation * added trace annotations for project store * added trace annotations for BlockStore * added trace annotations for HttpClient * added trace annotations for CAS/GC
* all threads should be named (#304)Stefan Boberg2023-05-152-4/+7
| | | | | | | * added WorkerThreadPool naming, packaged_task support * name the http.sys thread pool service threads * added http.sys I/O threadpool naming * upstream cache I/O thread naming
* move testing and observability code to zenhttp (#266)Stefan Boberg2023-05-022-2/+2
|
* move auth code from zenserver into zenhttp (#265)Stefan Boberg2023-05-023-4/+3
| | | this code should be usable outside of zenserver, so this moves it out into zenhttp where it can be used from lower level components
* moved source directories into `/src` (#264)Stefan Boberg2023-05-0212-0/+6196
* moved source directories into `/src` * updated bundle.lua for new `src` path * moved some docs, icon * removed old test trees
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-12 01:03:18 +0000