aboutsummaryrefslogtreecommitdiff
path: root/src/zenserver
Commit message (Collapse)AuthorAgeFilesLines
...
* Correctly calculate the total number of RPC ops in the stats page for ↵Dan Engelbrecht2023-09-291-2/+2
| | | | | structured cache (#433) Change "chunks" title to "count" for RPC chunk requests in stats page for structured cache
* move badrequest count back to old scope for compatability with Zen DashboardDan Engelbrecht2023-09-281-2/+1
|
* adding more stats (#429)Dan Engelbrecht2023-09-2811-58/+374
| | | | | - Feature: Add detailed stats on requests and data sizes on a per-bucket level, use parameter `cachestorestats=true` on the `/stats/z$` endpoint to enable - Feature: Add detailed stats on requests and data sizes on cidstore, use parameter `cidstorestats=true` on the `/stats/z$` endpoint to enable - Feature: Dashboard now accepts parameters in the URL which is passed on to the `/stats/z$` endpoint
* prefer to handle cache RPC requests synchronously (#428)Stefan Boberg2023-09-271-4/+17
| | | | | * only handle RPC requests in a worker thread if we have an upstream. we may as well handle the request inline on the http_io thread if we're only dealing with local data since the response times should be pretty consistent in that case * http.sys: don't create async worker thread pool until it's needed (typically only if we have an upstream)
* chache upstream stats improved (#426)Dan Engelbrecht2023-09-271-10/+16
| | | | * cache upstream stats improved * fix exit for monitor thread
* handle error in error log (#422)Dan Engelbrecht2023-09-251-15/+46
| | | * do not allow exceptions to leak from Sentry error reporting or SentryAssertImpl::OnAssert
* Add runtime status/control of logging (#419)Dan Engelbrecht2023-09-226-26/+151
| | | | | | | | | | | - Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging - `enablewritelog`=`true`/`false` parameter to control cache write logging - `enableaccesslog`=`true`/`false` parameter to control cache access logging - `loglevel` = `trace`/`debug`/`info`/`warning`/`error` - Feature: New zen command `logs` to query/control zen logging - No arguments gives status of logging and paths to log files - `--cache-write-log` `enable`/`disable` to control cache write logging - `--cache-access-log` `enable`/`disable` to control cache access logging - `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
* add trace command to enable/disable tracing at runtime (#416)Dan Engelbrecht2023-09-222-12/+71
| | | | | * add trace command to enable/disable tracing at runtime * rework tracing init/start/stop * changelog
* Improvement: Add names to background jobs for easier debugging (#412)Dan Engelbrecht2023-09-204-46/+66
| | | | Improvement: Background jobs now temporarily sets thread name to background job name while executing Improvement: Background jobs tracks worker thread id used while executing
* controlled zenserver shutdown (#413)Dan Engelbrecht2023-09-201-20/+36
| | | * Do controlled shutdown of zenserver catching any exception thrown during shutdown
* VFS implementation for local storage service (#396)Stefan Boberg2023-09-2013-12/+1013
| | | currently, only Windows (using Projected File System) is supported
* increase jupiter oplog import/export connection timeout to 30 min (#411)Dan Engelbrecht2023-09-191-1/+1
| | | | * increase jupiter oplog import/export connection timeout to 30 min * changelog
* Add retry if FinalizeRef responds with non-empty "Needs" attachments (#409)Dan Engelbrecht2023-09-195-261/+377
| | | | | * Add retry if FinalizeRef responds with non-empty "Needs" attachments * better logging/progress report * changelog
* handle errors in spdlog gracefully (#410)Dan Engelbrecht2023-09-191-0/+29
| | | | * handle errors in spdlog gracefully - try to report and avoid termination * changelog
* add DiskWriteBlocker to structured cache store log writer (#408)Dan Engelbrecht2023-09-153-32/+45
| | | | | * add DiskWriteBlocker to structured cache store log writer * changelog
* add more trace scopes (#362)Dan Engelbrecht2023-09-155-160/+251
| | | | | * more trace scopes * Make sure ReplayLogEntries uses the correct size for oplog buffer * changelog
* More statistics for Cache, Project Store and Cid Store (#405)Dan Engelbrecht2023-09-144-5/+60
| | | | | Cache: requestcount, badrequestcount, writes Project Store: requestcount Cid Store: cidhits, cidmisses, cidwrites
* http and httpsys config options (#401)Dan Engelbrecht2023-09-143-5/+45
| | | | * Added `--http-threads`, `--httpsys-async-work-threads`, `--httpsys-enable-request-logging` and `--httpsys-enable-async-response` command line options to zenserver * remove unused CreateHttpSysServer
* job queue and async oplog-import/export (#395)Dan Engelbrecht2023-09-1313-135/+507
| | | | | | | | | | | | | | | | - Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status - Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status - GET will return a response detailing status, pending messages and progress status - DELETE will mark the job for cancelling and return without waiting for completion - If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again - Feature: New zen command `jobs` to list, get info about and cancel background jobs - If no options are given it will display a list of active background jobs - `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job - `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job - Feature: oplog import and export http rpc requests are now async operations that will run in the background - Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default - Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C - Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
* scan oplog object for fields (#397)Dan Engelbrecht2023-09-131-89/+130
| | | | * scan oplog object for fields * Read all oplog entries but only read op data and get mapping for latest op of each key
* issue warning instead of assert on bad data in cid store (#400)Dan Engelbrecht2023-09-131-10/+14
| | | * issue warning instead of assert on bad data in cid store
* fix url parsing crash (#399)Dan Engelbrecht2023-09-131-1/+1
| | | * Don't index out of string_view range when parsing URI in httpsys
* ZenCacheStore is now reference counted (#398)Stefan Boberg2023-09-133-9/+19
| | | this change also adds a GetNamespaces function which may be used to enumerate all currently known cache namespaces
* incremental oplog upload for block-based targets (#392)Dan Engelbrecht2023-09-128-72/+224
| | | | | | | * add option for base container for oplog export read base oplog and fetch known blocks * reuse blocks if a known block has 80+ % usage * changelog * better logging and added base to remotestore descriptions
* gracefully handle errors when writing cache log (#391)Dan Engelbrecht2023-09-111-29/+50
| | | | | | | * gracefully handle errors when writing cache log * changelog * fix log message
* add `cache-write-log` and `cache-access-log´ configuration options (#394)Dan Engelbrecht2023-09-114-10/+33
| | | * add `cache-write-log` and `cache-access-log´ configuration options
* better sorting of attachments in oplog blocks (#390)Dan Engelbrecht2023-09-111-13/+54
| | | | | - sort attachments based on (first) associated op - sort attachments based on key of op to get a more consistent order rather than order of ops written - keep attachments from same op in same block
* stream oplog attachments from jupiter (#384)Dan Engelbrecht2023-09-067-119/+245
| | | | | | | | | | * stream large downloads from jupiter to temporary file * rework DeleteOnClose - top level marks file for delete and if lower level parts wants to keep it it clears that flag * changelog * log number of attachments to download * add delay on jupiter request failure when retrying * make sure we upload all attachments even if Needs are empty when ForceUpload is true release TempAttachment as soon as it is used * sort attachments so we get predictable blocks for the same oplog
* retry file create (#383)Dan Engelbrecht2023-09-041-2/+29
| | | | | * add retry logic when creating files * only write disk usage log if disk writes are allowed * changelog
* add `--write-config` to zenserver options (#382)Dan Engelbrecht2023-09-011-7/+23
| | | | | | | | | | - Feature: `zen up` command has two new command line options - `--config <file_path>` tells zenserver to start with a specific config file - `--owner-pid <pid>` tells zenserver to start with a owning process id - Feature: `zen attach` command to add additional owning processes to a running zenserver instance - `--owner-pid <pid>` adds pid to running zenserver instance list of owning processes - Feature: `--write-config` command line option for zenserver - `--write-config <file_path>` path to a file which will contain a lua config file for zenserver combining all command line options and optional lua config files - Improvement: `zen up` command will check if zenserver is currently running before starting up a new instance
* lua config file improvements (#379)Dan Engelbrecht2023-08-311-300/+791
| | | | | | - Bugfix: All options given on command line now overrides lua config file settings - Improvement: All options available from command line can now be configured in the lua config file (with a few exceptions such as `owner-pid`, `install` and `uninstall`) We can now also generate a Lua config (just string so far) that includes all options given on command line together with any optional Lua input config file. The idea here is to be able to save config options in the future.
* project store gc deadlock (#381)Dan Engelbrecht2023-08-313-40/+57
| | | | * Trying to recursively take a shared RWLock while another thread is waiting for a exclusive lock results in a deadlock. Forward the shared lock so we don't have to grab it recursively. * changelog
* add helper functions in HttpProjectService to help with readability (#378)Dan Engelbrecht2023-08-292-1195/+1326
|
* Minor: Make sure to reset cache logging worker thread event to avoid ↵Dan Engelbrecht2023-08-241-0/+1
| | | | busy-looping looking for more work
* Add `--embedloosefiles` option to `oplog-export` (#376)Dan Engelbrecht2023-08-243-73/+242
| | | | * Add `--embedloosefiles` option to `oplog-export` which adds loose files to the export, removing need to call `oplog-snapshot` * Retain `ServerPath` in oplog when performing `oplog-snapshot`. This is a short-term fix for current incompatability with the UE cooker.
* use robinmap in compact cas (#368)Dan Engelbrecht2023-08-211-4/+5
| | | | | * Use robin-map in compactcas for 30% faster CasContainerStrategy::CollectGarbage * use robin_set in ProjectStore::Oplog::GatherReferences and BlockStore::ReclaimSpace * changelog
* use atexit hook to shut down tracing (#369)Dan Engelbrecht2023-08-211-1/+9
|
* buffered file reading for oplog (#366)Dan Engelbrecht2023-08-211-5/+16
| | | | | * add BasicFileBuffer for buffered read of BasicFile * Use BasicFileBuffer when reading oplog * changelog
* fix trace close (#365)Dan Engelbrecht2023-08-211-0/+1
| | | * Make sure we close our trace session properly
* add update/delete endpoint for project and oplog (#353)Dan Engelbrecht2023-08-183-2/+135
| | | | | | | | | | | | | * add update endpoint for project store project * add update endpoint for oplog * changelog * Zen command line tool `project-update` Zen command line tool `project-delete` Zen command line tool `oplog-update` Zen command line tool `oplog-delete` * add --force-update option to project/oplog create remove project/oplog update commnad
* check oplog op attachments when gathering references for GC (#363)Dan Engelbrecht2023-08-182-27/+24
| | | | | * Make sure to check oplog op attachments when gathering references for GC * Add oplog op content to error result if attachment is missing when doing `oplog-export`
* Cache process handles for FormatPackageMessage (#360)Dan Engelbrecht2023-08-172-6/+15
|
* single thread async cache log (#361)Dan Engelbrecht2023-08-173-124/+149
| | | | * rework cache store background jogging * correct capture for context
* skip upstream logic early if we have no upstream endpoints (#359)Dan Engelbrecht2023-08-175-37/+85
| | | | | * Skip upstream logic early if we have not upstream endpoints * make cache store logging of CbObjects async * changelog
* project store stats (#357)Dan Engelbrecht2023-08-144-37/+197
| | | | | | | * add basic stats for project store * stats for BadRequest in ProjectStore::Rpc * changelog * group stats fix ChunkWriteCount when accepting ops
* update vcpkg dependencies (#356)Dan Engelbrecht2023-08-114-6/+12
| | | | | | * bump vcpkg version * fmt lib 10 fixes * xmake dependencies (with linux workarounds) * changelog
* Make sure we always write "data" attachment hash for snapshotted oplog ↵Dan Engelbrecht2023-08-114-9/+29
| | | | | | | | | entries (#355) * Make sure we always write "data" attachment hash for snapshotted oplog entries * Make sure to add chunk mappings for files moved to attatchment in snapshot operation * fix inverted timoute for expiration (we don't want time expiry in these cases) * increase timeout for jupiter oplog in project to 3 min * changelog
* Add `response.text` to output in log when jupiter request fails (#354)Dan Engelbrecht2023-08-103-36/+72
| | | | | | * Add `response.text` to output in log when jupiter request fails * only hash jupiter ref once and keep that * Increase request timeout when uploading to Jupiter to 2 min (to handle very large attachments) * changelog
* Allow oplog file mapping where ServerPath is missing if a attachment hash is ↵Dan Engelbrecht2023-08-101-4/+9
| | | | specified (#352)
* add extened info in log on all jupiter operation (#350)Dan Engelbrecht2023-08-091-7/+273
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-03-15 00:35:23 +0000