| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
| |
* zen command - add options to control meta data cache when triggering gc
|
| |
|
|
|
| |
Added option `gc-attachment-passes` to zenserver
Cleaned up GCv2 start and stop logs and added identifier to easily find matching start and end of a GC pass in log file
Fixed project store not properly sorting references found during lock phase
|
| |
|
|
|
| |
* optimize IoHash and OId comparisions
* refactor filtering of unused references
* add attachment filtering to gc
|
| |
|
| |
* add option to force gcv2 to run single threaded
|
| |
|
|
| |
- Improvement: Add file and line to ASSERT exceptions
- Improvement: Catch call stack when throwing assert exceptions and log/output call stack at important places to provide more context to caller
|
| |
|
| |
jobqueue - add AbortReason and properly propagate error when running async command
|
| |
|
|
|
|
| |
* moved all RPC processing from HttpStructuredCacheService into separate CacheRpcHandler class in zenstore
* move package marshaling to zenutil. was previously in zenhttp/httpshared but it's useful in other contexts as well where we don't want to depend on zenhttp
* introduced UpstreamCacheClient, this provides a subset of functions on UpstreamCache and lives in zenstore
|
| | |
|
| |
|
|
|
| |
this also adds a central, shared folder for storing information which may be found by any instance on the host. The directory is currently located alongside the default install and state directory.
Initially this is used to store a collection of known `root_manifest` locations and a copy of the latest manifest version which allow us to find all known locations where zen state is present.
|
| |
|
| |
this adds information on program name and command line to trace initialization
|
| |
|
|
|
|
| |
- Improvement: Scrub command now validates compressed buffer hashes in filecas storage (used for large chunks)
- Improvement: Added --dry, --no-gc and --no-cas options to zen scrub command
- Improvement: Implemented oplog scrubbing (previously was a no-op)
- Improvement: Implemented support for running scrubbint at startup with --scrub=<options>
|
| |
|
|
|
| |
- Feature: New endpoint `/admin/gc-stop` to cancel a running garbage collect operation
- Feature: Added `zen gc-stop` command to cancel a running garbage collect operation
- Bugfix: GCv2 - make sure to discover all projects and oplogs before checking for expired data
|
| |
|
|
|
|
|
|
|
|
|
| |
- Feature: Added new options to zenserver for GC V2
- `--gc-compactblock-threshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `--gc-verbose` GCV2 - enable more verbose output when running a GC pass
- Feature: Added new options to `zen gc` command for GC V2
- `--compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `--verbose` GCV2 - enable more verbose output when running a GC pass
- Feature: Added new parameters for endpoint `admin/gc` (PUT)
- `compactblockthreshold` GCV2 - how much of a compact block should be used to skip compacting the block, default is 90%
- `verbose` GCV2 - enable more verbose output when running a GC pass
|
| |
|
|
|
| |
- Refactor GCV2 so GcReferencer::RemoveExpiredData returns a store compactor, moving out the actual disk work from deleting items in the index.
- Refactor GCV2 GcResult to reuse GcCompactStoreStats and GcStats
- Make Compacting of stores non-parallell to not eat all the disk I/O when running GC
|
| |
|
|
|
| |
- Feature: Writes a `gc.log` with settings and detailed result after each GC execution (version 2 only)
- Break out file name rotate to allow access for gclog
- CompactBinaryToJson(MemoryView Data, StringBuilderBase& InBuilder)
|
| |
|
|
|
|
|
|
|
| |
this change aims to hide logging internals from client code, in order to make it easier to extend and take more control over the logging process in the future.
As a bonus side effect, the generated code is much tighter (net delta around 2.5% on the resulting executable which includes lots of thirdparty code) and should take less time to compile and link.
Client usage via macros is pretty much unchanged. The main exposure client code had to spdlog internals before was the use of custom loggers per subsystem, where it would be common to have `spdlog::logger` references to keep a reference to a logger within a class. This is now replaced by `zen::LoggerRef` which currently simply encapsulates an actual `spdlog::logger` instance, but this is intended to be an implementation detail which will change in the future.
The way the change works is that we now handle any formatting of log messages in the zencore logging subsystem instead of relying on `spdlog` to manage this. We use the `fmt` library to do the formatting which means the client usage is identical to using `spdlog`. The formatted message is then forwarded onto any sinks etc which are still implememted via `spdlog`.
|
| |
|
|
|
| |
- Feature: New parameter for endpoint `admin/gc` (GET) `details=true` which gives details stats on GC operation when using GC V2
- Feature: New options for zen command `gc-status`
- `--details` that enables the detailed output from the last GC operation when using GC V2
|
| |
|
| |
- Feature: New garbage collection implementation, still in evaluation mode. Enabled by `--gc-v2` command line option
|
| |
|
|
| |
m_LastLightweightGcDuration stats (#494)
|
| |
|
|
| |
- Feature: Add `--skip-delete` option to gc command
- Bugfix: Fix implementation when claiming GC reserve during GC
|
| |
|
|
|
|
| |
- Feature: New endpoint `/admin/flush ` to flush all storage - CAS, Cache and ProjectStore
- Feature: New command `zen flush` to flush all storage - CAS, Cache and ProjectStore
- Improved: Command `zen gc-status` now gives details about storage, when last GC occured, how long until next GC etc
- Changed: Cache access and write log are disabled by default
|
| |
|
| |
allows testing impact of calling mi_collect on a running server
|
| |
|
|
|
|
| |
- Feature: Add lightweight GC that only removes items from cache/project store without cleaning up data referenced in Cid store
- Add `skipcid` parameter to http endpoint `admin/gc`, defaults to "false"
- Add `--skipcid` option to `zen gc` command, defaults to false
- Add `--gc-lightweight-interval-seconds` option to zenserver
|
| |
|
|
|
|
|
|
|
|
|
| |
- Feature: New endpoint `/admin/logs` to query status of logging and log file locations and cache logging
- `enablewritelog`=`true`/`false` parameter to control cache write logging
- `enableaccesslog`=`true`/`false` parameter to control cache access logging
- `loglevel` = `trace`/`debug`/`info`/`warning`/`error`
- Feature: New zen command `logs` to query/control zen logging
- No arguments gives status of logging and paths to log files
- `--cache-write-log` `enable`/`disable` to control cache write logging
- `--cache-access-log` `enable`/`disable` to control cache access logging
- `--loglevel` `trace`/`debug`/`info`/`warning`/`error` to set debug level
|
| |
|
|
|
| |
* add trace command to enable/disable tracing at runtime
* rework tracing init/start/stop
* changelog
|
| |
|
|
| |
Improvement: Background jobs now temporarily sets thread name to background job name while executing
Improvement: Background jobs tracks worker thread id used while executing
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Feature: New http endpoint for background jobs `/admin/jobs/status` which will return a response listing the currently active background jobs and their status
- Feature: New http endpoint for background jobs information `/admin/jobs/status/{jobid}` which will return a response detailing status, pending messages and progress status
- GET will return a response detailing status, pending messages and progress status
- DELETE will mark the job for cancelling and return without waiting for completion
- If status returned is "Complete" or "Aborted" the jobid will be removed from the server and can not be queried again
- Feature: New zen command `jobs` to list, get info about and cancel background jobs
- If no options are given it will display a list of active background jobs
- `--jobid` accepts an id (returned from for example `oplog-export` with `--async`) and will return a response detailing status, pending messages and progress status for that job
- `--cancel` can be added when `--jobid` is given which will request zenserver to cancel the background job
- Feature: oplog import and export http rpc requests are now async operations that will run in the background
- Feature: `oplog-export` and `oplog-import` now reports progress to the console as work progress by default
- Feature: `oplog-export` and `oplog-import` can now be cancelled using Ctrl+C
- Feature: `oplog-export` and `oplog-import` has a new option `--async` which will only trigger the work and report a background job id back
|
| |
|
| |
Added zen scrub command which may be triggered via the zen CLI helper. This traverses storage and validates contents either by content hash and/or by structure. If unexpected data is encountered it is invalidated.
|
| |
|
|
|
|
| |
* Add `--gc-projectstore-duration-seconds` option
* Cleanup lua gc options parsing
* Remove dead configuration values
* changelog
|
| |
|
|
|
| |
Scrub -> ScrubStorage
Trigger -> TriggerGc (to make relationship to TriggerScrub clearer)
|
|
|
* moved source directories into `/src`
* updated bundle.lua for new `src` path
* moved some docs, icon
* removed old test trees
|