Continuous performance tracking to avoid major regressions

[rami3l]

So far the only ways we use to track performance are:

• Coarse-grained: running hyperfine.
• Fine-grained: building with otel and looking at the timeline view.

I think the first part can be automated in some ways: when reading @blyxyas' https://blog.goose.love/posts/making-a-clippy-lint-faster-by-3133x I realized that it may be helpful to keep tracking certain aspects of our runtime performance.

The most important one to track at the moment would be:

• improve proxy_mode (e.g. rustc wrapper) startup time? #2626

[djc]

In other projects we've successfully used Codspeed's GitHub Actions to do this in CI -- might be nice?

For example, in upki:

https://github.com/rustls/upki/blob/main/.github/workflows/bench.yml

[rami3l]

@djc Interesting that you mentioned Codspeed, because I was talking to them about a possible integration in rustup back in July. However our bench suite, if there is one, would be quite unconventional to them because it will rely on a subprocess call (?)

Of course I can mock it away in this particular case, but once we add something just slightly more complex (e.g. if we really need something to happen in that subprocess call), this approach will no longer work.

I'll dig into this and see what I can do about it.

[rami3l]

Regarding the subprocess thing, it looks like this is no longer an issue since last month: CodSpeedHQ/action#235