aboutsummaryrefslogtreecommitdiff
path: root/.github/workflows/validate.yml
Commit message (Collapse)AuthorAgeFilesLines
* zen-test: add CLI integration harness + TestArtifactProvider + CI host stats ↵Stefan Boberg2026-04-201-0/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#985) Establishes a new end-to-end integration test harness for the `zen` CLI, the shared fetcher it uses to pull test artifacts, and the CI plumbing that feeds both. Also lowers the default test-harness log level and broadens the artifact fetcher's credential resolution. ### `zen-test` executable (`src/zen-test/`) - New binary modeled on `zenserver-test`, built only in debug. - `zen-test.{h,cpp}` harness: spawns `zen.exe` via `CreateProc` and captures combined stdout/stderr into a `ZenCommandResult` for assertion. - Registered with `scripts/test.lua` under the short name `zen` (`xmake test --run=zen`) and enabled for `--kill-stale-processes`. - Prints a clear console message when invoked from a release build (tests disabled), so misconfiguration is easy to spot. - Documented in `CLAUDE.md` (test-suite naming table + test projects section) and `README.md`. - Test cases in the `zen.artifactprovider` suite: - `probe.lyra_cook_rpc_recording` — probe against a canonical Lyra cook RPC recording that skips with a diagnostic `MESSAGE` when no artifact source is configured. - `probe.s3_readme` — probes the configured S3 bucket for `README.md` using a fresh temp cache to force the request through to S3; skips on macOS without static creds (no EC2 Mac runners in our fleet). - `zen.utility-cmd` suite: new integration tests exercising `zen print`, `zen wipe`, and `zen copy`. ### `TestArtifactProvider` (`src/zenutil/testartifactprovider.{h,cpp}`) - `Ref<TestArtifactProvider>` factory returning a local-only or S3-backed provider, selected from env vars: - `ZEN_TEST_ARTIFACTS_PATH` — local directory to serve from (write-through cache for remote fetches). - `ZEN_TEST_ARTIFACTS_S3` — S3 URL to fetch from. - `AWS_DEFAULT_REGION` / `AWS_REGION`, `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` — standard AWS config. - `Exists(path)` / `Fetch(path)` API with a `TestArtifactFetchResult` return carrying the content buffer and a diagnostic error string. Content is cached on disk across test runs. - **IMDS credential fallback**: when no static `AWS_ACCESS_KEY_ID` is present, attaches an `ImdsCredentialProvider` so self-hosted EC2 runners with an attached IAM role can sign S3 requests without static credentials (mirrors the pattern in `zenserver/hub/hydration.cpp`). - **IMDS opt-out**: honors the standard `AWS_EC2_METADATA_DISABLED=true` env var, and skips IMDS by default on macOS where the link-local probe would just emit noise. ### Test harness log level (`src/zencore/testing.cpp`) - `TestRunner::ApplyCommandLine` now defaults the global log level to `Info` (was effectively `Trace`), cutting the noise from `xmake test --run=all` now that the suite has grown. Applies uniformly to `zencore-test`, `zenhttp-test`, `zenstore-test`, `zenutil-test`, `zenserver-test`, `zen-test`, etc. `--debug` (Debug) and `--verbose` (Trace) still opt back in when chasing failures. ### CI (`.github/workflows/validate.yml`) - **Runner info step** on all three platforms (Windows/Linux/macOS): prints host, CPU topology, memory, and disk usage before the build/test step, so flakes that correlate with a particular runner or low disk space are easy to spot. - **Artifact env wiring**: passes `ZEN_TEST_ARTIFACTS_S3` and `AWS_DEFAULT_REGION` into the debug Build & Test step on all three platforms so the probe can reach its source when the repo variable is configured. The probe skips cleanly when unset.
* [zenhorde] security + robustness fixes (#972)Stefan Boberg2026-04-171-54/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | ### Critical (cryptographic correctness) - AES-GCM nonce: replace homebrew `N32[0]++; N32[1]--; N32[2] = ^` scheme with NIST SP 800-38D §8.2.1 deterministic construction (64-bit big-endian counter). Session tears down on counter exhaustion instead of reusing a nonce. - Remove `std::random_device` / `mt19937` nonce seed - the deterministic construction from the previous commit doesn't need an RNG, and `std::random_device` isn't guaranteed to be a CSPRNG. - BCrypt return values: check every `BCRYPT_SUCCESS`, cache the `BCRYPT_KEY_HANDLE` on the context instead of re-creating it per message, destroy under null-guards. Closes the silent-downgrade-to-non-GCM path. ### High - OpenSSL: check `EVP_CIPHER_CTX_new` / `EVP_EncryptInit_ex` / `EVP_DecryptInit_ex` return values in the constructor and set `HasErrors` on failure. - Log AES-GCM tag-verification failures distinctly from other decrypt errors (BCrypt `STATUS_AUTH_TAG_MISMATCH` / OpenSSL `EVP_DecryptFinal_ex` post-set-tag), with a sequence counter for correlation. - Thread a bounds-checked `ReadCursor` through every `Read*` parser helper; `ReadException` / `ReadExecuteResult` / `ReadBlobRequest` now return `bool` and callers treat malformed frames as protocol errors. Closes the `0xFF` varint OOB-read. - Validate `ReadBlobRequest` locator as a safe filename component (reject path separators, `..`, NUL/control, drive colons, leading/trailing dot/space, length > 255). Closes the path-traversal attack on the `BundleDir / (Locator + ".blob")` join. - Bind `AsyncAgentMessageChannel`'s timer and `AsyncReadResponse` entry onto the socket's strand; expose `AsyncComputeSocket::GetStrand()`. Removes the race between the bare-io_context timer completion and `OnFrame` on `m_PendingHandler` under the 3-thread pool. - Drop the long-lived `m_EncryptBuffer` member - encrypt into a fresh per-write buffer shared with the completion handler. Also fixes thread-safety of the encrypt path. - Validate server-returned `ClusterId` against `[A-Za-z0-9._-]{1,64}` before concatenating into the `api/v2/compute/<ClusterId>` URL. ### Medium - `EVP_CIPHER_CTX_reset` + re-bind cipher on every encrypt/decrypt so stale state cannot bleed across messages. Also logs EVP failures. - Malformed `ExecuteResult` (size != 4) now tears down the agent instead of silently reporting `ExitCode = -1`. - Replace `assert(Eq != nullptr)` on env var parsing with a `zen::runtime_error` - assert is compiled out in release and `*(Eq+1)` was UB. - Blob name uses `zen::Oid::NewOid()` (24 hex chars, seeded from `random_device` run-id + monotonic serial) instead of predictable `<pid>_<ms>_<counter>`. Refuse to overwrite an existing blob path. - Cap `m_RecentlyDrainedWorkerIds` at 256 entries with an FIFO eviction queue. - `Blob(Data, Length)` rejects `Length > INT32_MAX` instead of wrapping the int32 wire fields. - Static `AuthToken` path uses `HttpClientAccessToken::TimePoint::max()` (never-expires sentinel) instead of synthesizing `now + 24h`. - Remove dead `m_Transport` field and `else if (m_Transport)` branch in `AsyncHordeAgent::Cancel()`.
* kill stale test processes (zenserver, minio, nomad, consul) before and after ↵Stefan Boberg2026-04-011-0/+84
| | | | | CI test runs (#909) Adds steps to the validate workflow on all platforms that kill any zenserver, minio, nomad, or consul processes launched from the build output directory. Runs before tests to clear stale processes from previous runs, and after tests (always, even on failure) to clean up.
* Linux Crashpad fix (#890)Stefan Boberg2026-03-241-0/+6
| | | | | | - **Replace crashpad static-libc++ patch file with `io.replace()` in `on_install`** — The old `.patch` file was fragile (trailing-whitespace stripping on Windows would silently break it). Using `io.replace()` in the xmake build script is more robust and easier to maintain. - **Skip sentry-native `on_test` link check on Linux** — The link test requires `-lc++abi` when building with the UE clang toolchain but adding it unconditionally breaks GCC/libstdc++ builds. The zenserver build itself validates that the library is usable. - **Add `crashpad-test.sh`** — A test script that launches a release zenserver, waits for the health endpoint, then verifies that `crashpad_handler` is running, no `sentry_init` failure was logged, and the handler has no dynamic `libc++.so.1` dependency. - **Add Crashpad Check step to Linux release CI** — Runs `crashpad-test.sh` in the `validate` workflow for release builds to catch crashpad regressions before merge.
* fix for GHES failing on upload-artifacts@v3 (#856)Stefan Boberg2026-03-181-8/+8
| | | Pin version to last v3 version using node20, since our GHES does not support v4
* Push Docker images to ECR after building (#845)Stefan Boberg2026-03-161-4/+10
| | | | | | | - Add ECR login via aws CLI (using IMDS credentials) - Tag and push images to 728559092788.dkr.ecr.us-east-1.amazonaws.com/zenserver - Use tag suffixes (-wine, -linux) to discriminate image variants - Replace read-file-action with shell equivalent for VERSION.txt - Enable docker-build and artifact uploads on all branches for validation
* Add Docker image build for compute workers (#837)Stefan Boberg2026-03-161-0/+45
| | | Adds a Dockerfile (Ubuntu 24.04 + WineHQ) and an `xmake docker` task to build and optionally push a zenserver-compute Docker image, enabling Linux deployment of compute workers that run Windows executables via Wine.
* native xmake toolchain definition for UE-clang (#805)Stefan Boberg2026-03-041-3/+3
| | | | | | | | | | | | | | | This change is meant to provide a smoother experience when working on Linux. After this change, the toolchain setup process is now simply ```bash $ scripts/ue_build_linux/get_ue_toolchain.sh ``` and then at config time the toolchain is automatically detected if you downloaded it to the default location or have the `UE_TOOLCHAIN_DIR` environment variable set ```bash xmake config --mode=debug ``` Compared to the old script-based approach this configures the toolchain more precisely, avoiding leakage into unrelated build processes such as when a package manager decides to build something like Ninja locally etc.
* added `--verbose` option to zenserver-test and `xmake test` (#798)Stefan Boberg2026-03-011-3/+3
| | | | | | * when `--verbose` is specified to zenserver-test, all child process output (typically, zenserver instances) is piped through to stdout. you can also pass `--verbose` to `xmake test` to accomplish the same thing. * this PR also consolidates all test runner `main` function logic (such as from zencore-test, zenhttp-test etc) into central implementation in zencore for consistency and ease of maintenance * also added extended utf8-tests including a fix to `Utf8ToWide()`
* test running / reporting improvements (#797)Stefan Boberg2026-02-281-23/+62
| | | | | | | | | | | | | | | | | | | **CI/CD improvements (validate.yml):** - Add test reporter (`ue-foundation/test-reporter@v2`) for all three platforms, rendering JUnit test results directly in PR check runs - Add "Trust workspace" step on Windows to fix git safe.directory ownership issue with self-hosted runners - Clean stale report files before each test run to prevent false failures from leftover XML - Broaden `paths-ignore` to skip builds for non-code changes (`*.md`, `LICENSE`, `.gitignore`, `docs/**`) **Test improvements:** - Convert `CHECK` to `REQUIRE` in several test suites (projectstore, integration, http) for fail-fast behavior - Mark some tests with `doctest::skip()` for selective execution - Skip httpclient transport tests pending investigation - Add `--noskip` option to `xmake test` task - Add `--repeat=<N>` option to `xmake test` task, to run tests repeatedly N times or until there is a failure **xmake test output improvements:** - Add totals row to test summary table - Right-justify numeric columns in summary table
* retain xmake state between runs (#656)Stefan Boberg2025-11-181-1/+1
| | | | | | this change relocates the xmake global state to a directory beside the workspace directory so it doesn't get wiped on every run thus we can avoid rebuilding every package on every run. Unlike vcpkg, xmake separates revisions of packages into their own tree it's robust enough to handle different versions of different packages on different branches. It's however not clear to me that modifying the contents of an `xmake.lua` package definition file across branches is ok. It *may* be necessary to change the directory name for the shared state in this case but it should be a rare event.
* sentry/asan configuration tweaks (#649)v5.7.10-pre0Stefan Boberg2025-11-131-3/+3
| | | | | * Automated more of the decisions around which options to set when using ASAN * Also disabled Sentry by default as it's a bit annoying to have it upload crashes during development. Sentry is still automatically enabled and integrated as part of the `xmake bundle` step however so released builds will still have it.
* switch to xmake for package management (#611)Stefan Boberg2025-11-071-49/+3
| | | | | | | | | | | | | | | | | | | | | | This change removes our dependency on vcpkg for package management, in favour of bringing some code in-tree in the `thirdparty` folder as well as using the xmake build-in package management feature. For the latter, all the package definitions are maintained in the zen repo itself, in the `repo` folder. It should now also be easier to build the project as it will no longer depend on having the right version of vcpkg installed, which has been a common problem for new people coming in to the codebase. Now you should only need xmake to build. * Bumps xmake requirement on github runners to 2.9.9 to resolve an issue where xmake on Windows invokes cmake with `v144` toolchain which does not exist * BLAKE3 is now in-tree at `thirdparty/blake3` * cpr is now in-tree at `thirdparty/cpr` * cxxopts is now in-tree at `thirdparty/cxxopts` * fmt is now in-tree at `thirdparty/fmt` * robin-map is now in-tree at `thirdparty/robin-map` * ryml is now in-tree at `thirdparty/ryml` * sol2 is now in-tree at `thirdparty/sol2` * spdlog is now in-tree at `thirdparty/spdlog` * utfcpp is now in-tree at `thirdparty/utfcpp` * xmake package repo definitions is in `repo` * implemented support for sanitizers. ASAN is supported on windows, TSAN, UBSAN, MSAN etc are supported on Linux/MacOS though I have not yet tested it extensively on MacOS * the zencore encryption implementation also now supports using mbedTLS which is used on MacOS, though for now we still use openssl on Linux * crashpad * bumps libcurl to 8.11.0 (from 8.8.0) which should address a rare build upload bug
* move cpr in-tree (#605)Stefan Boberg2025-10-241-1/+1
| | | | | | * added cpr 1.10.5 in-tree to allow updates to vcpkg without breaking the build * added asio 1.29.0 in-tree to remove one more vcpkg dependency * bumped vcpkg to 2024.06.15 to address failure to build due to use of deprecated binaries in vcpkg (404 error: `https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst` during build)
* use batching clang-format for quicker turnaround on validate actions (#529)Stefan Boberg2025-09-301-1/+1
| | | this changes the validate job to use a batching version of the clang-format-action which reduces turnaround from some six minutes to six seconds
* update readme with required vcpkg version (#496)Dan Engelbrecht2025-09-151-1/+1
| | | - Improvement: Updated README.md to state the required version vcpkg
* Remove workflow hacks for CI debuggingLiam Mitchell2025-08-221-1/+0
|
* Remove libc++ installation from validate workflowLiam Mitchell2025-08-201-6/+0
|
* Use apt instead of apt-get for installing system packagesLiam Mitchell2025-07-301-2/+3
|
* Install libc++-dev during CI in order to build libsystemd properlyLiam Mitchell2025-07-291-0/+5
|
* Always upload vcpkg logs on failureLiam Mitchell2025-07-261-0/+1
|
* Temporarily enable Validate jobs on push to de/zen-service-command for testingLiam Mitchell2025-07-261-0/+1
|
* Upload vcpkg logs as artifacts on failureLiam Mitchell2025-07-251-0/+6
|
* Revert "remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY ↵Dan Engelbrecht2024-10-101-2/+1
| | | | | (#191)" (#193) This reverts commit e809931618b443809e9740edb70a62d0cab01f87.
* remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY (#191)Stefan Boberg2024-10-081-1/+2
| | | | | | * remove temporary workaround involving _LIBCPP_DISABLE_AVAILABILITY * temp disable signing on Mac this change should be revisited once we have resumed regular service wrt MacOS runners
* Replicate releases on new release creation (#150)Joakim Lindqvist2024-09-101-0/+2
| | | | | | | | | * Ignore changes to the mirror_releases script * Only trigger release mirroring when new releases are made * Added a minimum release number to workaround issues with certain older releases * Lowered number of releases that are replicated
* bump vcpkg and xmake to latest (#40)Dan Engelbrecht2024-04-241-2/+2
| | | - Improvement: Bumped xmake to 2.9.1 and vcpkg version to 2024.03.25
* add a limit to the number of times we attempt to finalize (#22)Dan Engelbrecht2024-03-251-9/+14
| | | | | | - Improvement: Add limit to the number of times we attempt to finalize and exported oplog - Improvement: Switch to large thread pool when executing oplog export/import - Improvement: Clean up reporting of missing attachments in oplog export/import - Improvement: Remove double-reporting of abort reason for oplog export/import
* added action to mirror repo to github.com (#17)Stefan Boberg2024-03-211-1/+4
|
* re-enable macos signing (#15)Dan Engelbrecht2024-03-201-2/+1
|
* remove hv tags on actions since they are no longer usefulStefan Boberg2024-03-201-5/+5
| | | | we use `zen` tags to discriminate now instead
* Enabled signing on windows agents again (#12)Joakim Lindqvist2024-03-201-2/+2
| | | * Enabled signing on windows agents again
* make sure zenserver reacts and exist on SIGTERM signal (#8)Dan Engelbrecht2024-03-151-6/+7
| | | | | * make sure zenserver reacts and exist on SIGTERM signal * add zen tag to all runners * temp disable mac codesigning
* updates to signingStefan Boberg2024-03-131-2/+2
| | | | | macOS signs executables again Windows releases do not (neither does validate)
* workflow tweaksStefan Boberg2024-03-121-5/+5
|
* workflow fixes (#1)Stefan Boberg2024-03-121-16/+14
| | | | | * update validate action for enterprise github * updated runner setup options * updated release workflow for enterprise github
* also update validate to avoid code sign problemsStefan Boberg2024-03-051-1/+2
|
* windows executable signing (#566)Dan Engelbrecht2023-12-151-1/+2
| | | Windows executables are now signed with official cert when creating a release
* embed html frontend content as binary compiled data (#559)Dan Engelbrecht2023-11-231-1/+1
| | | | | | - Feature: Added xmake task `updatefrontend` which updates the zip file containing the frontend html (`/src/zenserver/frontend/html.zip`) - Improvement: The frontend html content is no longer appended at the end of the executable which prevented signing, instead it is compiled in from the `/src/zenserver/frontend/html.zip` archive - Improvement: MacOS now does ad-hoc code signing by default when issuing `xmake bundle`, signing with proper cert is done on CI builds
* bump xmake CI version to 2.8.2 (#514)Dan Engelbrecht2023-11-071-9/+6
| | | | * bump xmake CI version to 2.8.2 * show test times in log
* revive UE toolchain build (#343)Dan Engelbrecht2023-08-221-11/+10
| | | | | | | | | * add comment about 7z usage in xmake bundle * add ue-libcxx lib/headers and licence + tps * update get_ue_toolchain.sh to use embedded libc++ and decouple from p4 * clearer output when falling back to zip from 7z * update ci scripts to use ue toolchain on linux * updated linux build README.md * changelog
* update vcpkg dependencies (#356)Dan Engelbrecht2023-08-111-6/+6
| | | | | | * bump vcpkg version * fmt lib 10 fixes * xmake dependencies (with linux workarounds) * changelog
* narrowed scope of clang-format actionStefan Boberg2023-06-091-2/+1
| | | | also removed some unnecessary entries from .gitignore
* Increased timeout to 25 min for Windows validationStefan Boberg2023-05-311-1/+1
|
* bumped xmake and vcpkg versionsStefan Boberg2023-05-261-6/+6
| | | | | xmake is now 2.7.9 (was 2.6.4) vcpkg is now 2023.04.15 (was 2022.08.15)
* extend timeout for linux validation on ciDan Engelbrecht2023-05-101-1/+1
|
* junit test reporting (#239)Dan Engelbrecht2023-02-231-24/+24
| | | | - Feature: `--junit` switch to `xmake test` to generate junit style reports of tests. - Feature: CI build on GitHub now uploads junit test reports as artifact to the check for PR validation and mainline validation
* actions/[email protected] -> actions/checkout@v3Dan Engelbrecht2022-11-141-4/+4
|
* update github actions versions to avoid node js warningsDan Engelbrecht2022-11-141-5/+5
|
* update action cancel old build to 0.11.0 (#186)v0.1.9-pre1Dan Engelbrecht2022-11-081-1/+1
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-07 17:48:39 +0000