Fix queue hang when collector exits without successfully benchmarking

[Mark-Simulacrum]

We have a general problem today that there's no general error feedback from the collector to the site -- only on a per-benchmark collection basis. That means that when there's some problem with the startup phase (e.g., no benchmarks match a filter, or SHA passed doesn't have artifacts), we end up hanging the queue until an operator manually intervenes and either removes or fixes the failing entry. (Currently done via manual database edit).

I think a good solution here would add some general error mechanism from the collector that could be posted back to the site. I think addressing the cases where the collector returns an Err is already 80% good enough -- we can work on avoiding panic! after that.

[Kobzol]

Regarding the input configuration (e.g. include etc.), what option would you prefer?

1. When we gather @rust-timer build/queue, and the parameters are bad, we don't even queue the PR into the DB, and just post a comment with an error.
2. If bench_next fails, a comment is sent to the PR (and the entry is removed from the PR).

I think that for fixing invalid input parameters, 1) would be preferable.

[Mark-Simulacrum]

IMO, (1) is nice-to-have validation, but we can't expect to duplicate all of that validation into the site -- that's not going to be maintainable. So I think pursuing (2) as the more general option is a better idea.

[Kobzol]

Well site already depends on collector, so it wouldn't be that hard to expose a few functions that check the validity of CLI parameters (probably to avoid panicking in bench_next we will need to do that anyway). I guess that we can do both.

[Mark-Simulacrum]

Sure, if that's easy we can do so. Some of the deeper validation - e.g., downloading the artifacts - feels a little harder to do well at that point, so I think we still want the more general feedback mechanism anyway.