wip: restore minimum configs for complement tests [DevOps/Pkging team input wanted] #1539

gamesguru · 2026-03-13T15:46:39Z

gamesguru commented

2026-03-13 15:46:39 +00:00

This pull request makes an attempt to bridge a testing gap and provide the ability to definitively and easily document complement deltas per commit (i.e., see what you broke/fixed), providing such feedback ideally within 15-25 minutes.

Current configs almost certainly do not work. They are copied verbatim from my fork, and config values won't work here as is. It may even depend on the Makefile, which I pruned. This is really posted to get preliminary feedback and possibly involvement from the packaging team.

I would like to maintain full compatibility both ways. In order to get this running on your guy's copy of the repo (cough docker-in-docker), some more modularization via resuable actions is likely needed, as well as pulling even more presently hard-coded config values dynamically from the repository env vars.

Ledger is stored as jsonl on an orphan git branch, _metadata/badges. From this, the README.md is expected to populate the status badge intelligently based on branch and/or commit (master may have more failing tests than your PR branch, for example). Gitea and Github both have python on the backend, but interpolation in the README badge is quite limited.

Pull request checklist:

This pull request targets the main branch, and the branch is named something other than
main.
I have written an appropriate pull request title and my description is clear.
I understand I am responsible for the contents of this pull request.
I have followed the contributing guidelines:
- My contribution follows the code style, if applicable.
- I ran pre-commit checks before opening/drafting this pull request.
- I have tested my contribution (or proof-read it for documentation-only changes)
  myself, if applicable. This includes ensuring code compiles.
- My commit messages follow the commit message format and are descriptive.
- I have written a news fragment for this PR, if applicable.

This pull request makes an attempt to bridge a testing gap and provide the ability to definitively and easily document complement deltas per commit (i.e., see what you broke/fixed), providing such feedback ideally within 15-25 minutes. Current configs almost certainly do not work. They are copied verbatim from my fork, and config values won't work here as is. It may even depend on the `Makefile`, which I pruned. This is really posted to **get preliminary feedback** and possibly **involvement from the packaging team**. **I would like to maintain full compatibility both ways.** In order to get this running on your guy's copy of the repo (*cough* docker-in-docker), some more modularization via resuable actions is likely needed, as well as pulling even more presently hard-coded config values dynamically from the repository env vars. Ledger is stored as `jsonl` on an orphan git branch, `_metadata/badges`. From this, the README.md is expected to populate the status badge intelligently based on branch and/or commit (master may have more failing tests than your PR branch, for example). Gitea and Github both have python on the backend, but interpolation in the README badge is quite limited.     **Pull request checklist:**  - [ ] This pull request targets the `main` branch, and the branch is named something other than `main`. - [ ] I have written an appropriate pull request title and my description is clear. - [ ] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [ ] My contribution follows the [code style][c2], if applicable. - [ ] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [ ] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [ ] My commit messages follow the [commit message format][c1cm] and are descriptive. - [ ] I have written a [news fragment][n1] for this PR, if applicable.  [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages [n1]: https://towncrier.readthedocs.io/en/stable/tutorial.html#creating-news-fragments

gamesguru added 1 commit

2026-03-13 15:46:39 +00:00

wip: restore minimum configs for complement tests

Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions

Details

07eb80a659

gamesguru added 1 commit

2026-03-13 16:23:47 +00:00

auto merge strategy for *.jsonl (although, .gitattributes must exist on orphan branch not main)

Documentation / Build and Deploy Documentation (pull_request) Has been cancelled

Details

Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled

Details

Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled

Details

e50981dd4a

lveneris reviewed

2026-03-14 00:44:21 +00:00

lveneris left a comment

This gets the award for longest CI script step I have seen ever, I believe. In general, storing results on a disconnected branch, and suggesting branch-aware README badges is highly unusual. The conventional approach (see, for example, hyperfine) is to store baseline results as JSON tidied away in a directory on main and compare PR results with those to check regressions. I'm not sure how easy scripting PR comments as a bot is with Forgejo, nor if it has issue matchers, but either of those would be preferable to just logging the results. You should also make an effort to break these steps up, using things like actions/checkout and actions/artifact (or whatever the equivalent here is if those don't work on Forgejo).

This gets the award for longest CI script step I have seen ever, I believe. In general, storing results on a disconnected branch, and suggesting branch-aware README badges is highly unusual. The conventional approach (see, for example, hyperfine) is to store baseline results as JSON tidied away in a directory on main and compare PR results with those to check regressions. I'm not sure how easy scripting PR comments as a bot is with Forgejo, nor if it has issue matchers, but either of those would be preferable to just logging the results. You should also make an effort to break these steps up, using things like `actions/checkout` and `actions/artifact` (or whatever the equivalent here is if those don't work on Forgejo).

README.md

					
				@ -66,3 +79,2 @@

				  * Your username for matrix will be in the form of `@username:federated.nexus`, however you can simply use the `username` part to log in. Your password is your password.

				* There's also [https://continuwuity.rocks/](https://continuwuity.rocks/). You can register a new account using Cinny via [this convenient link](https://app.cinny.in/register/continuwuity.rocks), or you can use Element or another matrix client *that supports registration*.

				- You can head over to [https://federated.nexus](https://federated.nexus/) in your browser.

You appear to have included changes outside of the scope of the PR again, just to let you know.

The issue with that is then every time someone makes a PR merge, they need to update the complement results.

And I would prefer this be done on a commit-by-commit basis, not just on master. This would require two commits for every one commit, which I don't like.

Mozilla uses this orphan branch/metadata branch pattern. It's used more often than one might think.

web-platform-tests/results-analysis at gh-pages
https://github.com/web-platform-tests/results-analysis/tree/gh-pages

The issue with that is then every time someone makes a PR merge, they need to update the complement results. And I would prefer this be done on a commit-by-commit basis, not just on master. This would require two commits for every one commit, which I don't like. Mozilla uses this orphan branch/metadata branch pattern. It's used more often than one might think. _web-platform-tests/results-analysis at gh-pages_ https://github.com/web-platform-tests/results-analysis/tree/gh-pages

orphaned branches and commits get GC'd so im not sure that'd work

gamesguru commented

2026-03-14 16:37:18 +00:00

This could probably be accomplished by a service like grafana or codecov.

But unless someone has plans to get that working next week, this PR serves as an stopgap.

PR comments are nice, but also tricky to pull off, and can be quite annoying. Some of my PRs have over 300 commits I definitely don't want 300 bot comments.

PR comments are not easily traceable either. With the jsonl file in git, I can do a whole bunch of cool queries and comparisons on the command line, let alone with more sophisticated tools.

Ideally we would also track the exact deltas, test names, but currently that info is just available for 90 days on the Forge Runner logs from the web UI.

This could probably be accomplished by a service like grafana or codecov. But unless someone has plans to get that working next week, this PR serves as an stopgap. PR comments are nice, but also tricky to pull off, and can be quite annoying. Some of my PRs have over 300 commits I definitely don't want 300 bot comments. PR comments are not easily traceable either. With the `jsonl` file in `git`, I can do a whole bunch of cool queries and comparisons on the command line, let alone with more sophisticated tools. Ideally we would also track the exact deltas, test names, but currently that info is just available for 90 days on the Forge Runner logs from the web UI.

s1lv3r commented

2026-03-14 16:49:21 +00:00

So basically this is intended to impl. gitlab's unit test reports if i understand it correctly?

I'm a lot more inclined to do this through a bot of some sort instead of a separate branch, as well. And update a single comment instead of commenting multiple times.

So basically this is intended to impl. gitlab's unit test reports if i understand it correctly? ![image](/attachments/52e1af36-23bb-4804-b514-9810bb3bad8e) I'm a lot more inclined to do this through a bot of some sort instead of a separate branch, as well. And update a single comment instead of commenting multiple times.

image.png

71 KiB

s1lv3r commented

2026-03-14 17:06:02 +00:00

I could potentially see using something like https://github.com/test-summary/action. If our test outputs were to go to .xml files (as is the general standard), then that action can generate a markdown summary which can be used as a PR comment. I think that'd be the best course of action?

Optionally then making those xml files part of the artifacts for local inspection

I could potentially see using something like https://github.com/test-summary/action. If our test outputs were to go to .xml files (as is the general standard), then that action can generate a markdown summary which can be used as a PR comment. I think that'd be the best course of action? Optionally then making those xml files part of the artifacts for local inspection

gamesguru commented

2026-03-14 17:22:35 +00:00

The complement tests don't produce JUnit compliant xml output as far as I know. They don't follow a standard.

Furthermore, this is a gitlab specific plugin that gitea and github may not follow. That needs to be checked. I would like this to be something people can run on their forks, their PRs, using their own runners, without much additional config or installing any additional plugins.

Look, again guys I know my solution is rough around the edges, but it's something I can essentially have working tomorrow and decommission in 15 minutes if someone gets a more enterprise solution. To me it's important to get testing implemented ASAP.

The complement tests don't produce JUnit compliant `xml` output as far as I know. They don't follow a standard. Furthermore, this is a gitlab specific plugin that gitea and github may not follow. That needs to be checked. I would like this to be something people can run on their forks, their PRs, using their own runners, without much additional config or installing any additional plugins. Look, again guys I know my solution is rough around the edges, but it's something I can essentially have working tomorrow and decommission in 15 minutes if someone gets a more enterprise solution. To me it's important to get testing implemented ASAP.

gamesguru commented

2026-03-14 17:34:17 +00:00

I can remove the ledger aspect of this, although I really prefer not to since it to me is the genesis of tracking useful information.

I think the current file in .forge/ works with neither GitLab nor GitHub. It's possible just adding a symlink would resolve this.

But my biggest ask for right now is just that it run on forge (gitea) forks. Since that way people can immediately run it on their PRs and advertise the unchanged or improved test results thru the status check.

I can remove the ledger aspect of this, although I really prefer not to since it to me is the genesis of tracking useful information. I think the current file in `.forge/` works with neither GitLab nor GitHub. It's possible just adding a symlink would resolve this. But my biggest ask for right now is just that it run on forge (gitea) forks. Since that way people can immediately run it on their PRs and advertise the unchanged or improved test results thru the status check.