Implement back publishing support

[eed3si9n]

Fixes #102
Ref scalameta/metals#3557
Ref typelevel/kind-projector#206
Ref https://github.com/scala/scala-parser-combinators/blob/d2e6685145ca4a9298c3d2ac0bde5ccf090f5d99/build.sh#L13-L15

Problem

For plugin code, like sbt plugins and Scala compiler plugin, and even normal libraries, it's common for the maintainers to want to back publish the current code base against a new version of Scala (or sbt, Scala Native etc) instead of bumping the version.

The current "bump the version" approach effectively pushes the work of figuring out the correct patch version number to the end users. In addition, if the plugin or the library does not cross publish all variants, then the end user might need to figure out different patch version to use effectively the same code.

Solution

This implements a mini DSL for tag:

version[@command|@a.b.c|@a.b."x"][#comment]

, which allows plugin authors to back publish a code base. A back-publish tag is split via @ character, and uses # to denote comments. The comment is useful when multiple tags must be made pointing to the same version, which might be needed to recover from failures or for sbt plugin migration.

• @a.b."x" for example @3.x expands to ;++3.x;publishSigned
• @a.b.c for example @2.12.20 expands to ;++2.12.20!;publishSigned
• @command for example @foo/publishSigned expands to foo/publishSigned

For example, to back publish sbt-pgp plugin 2.3.0, we will tag [email protected]#sbt-2.0.0-M3.

[mzuehlke]

Would be great to have the DSL explained in the README

[eed3si9n]

@mzuehlke I'd be happy to add documentation before merging this, but I didn't know how this PR would be received so first I wanted to float the idea and see what people thought on if we should do it, and if the syntax looks ok.

[mzuehlke]

Makes sense. Sorry for nagging

[eed3si9n]

Constructive feedbacks are always welcome :)

[SethTisue]

I'm strongly in favor of this. I never really understood @olafurpg's and @dwijnand's objections on #102, and the lack of this feature has, for some years now, been steadily wasting small amounts of my time — but those amounts add up! — in a lot of repos that I work in.

And the workaround of never backpublishing ends up wasting a much larger amount of time downstream, as a large number of people have to try to figure out what versions of things they need to be using. The amount of time wasted per person is pretty small, but again, it adds up.

[SethTisue]

@lrytz any thoughts on the syntax? I think it was you who designed the similar thing we did for the Scala modules

[lrytz]

Syntax looks fine. I wonder more if it is generic enough. Could we think of some concrete examples and how they would be addressed? E.g., re-release 1.2.3 of some library for a new version of Scala Native?

[eed3si9n]

Scala Version	JVM	JS (1.x)	Native (0.4.x)	Native (0.5.x)
2.13.x	_	_	_	case 3
3.x	case 2 (republish)	_	_	case 3
2.13.15	case 1 (all vs sub)	case 1 (all vs sub)	case 1 (all vs sub)	_

case 1.all

Publish all subprojects for a new Scala version

[email protected]

case 1.sub (to be impl)

Publish one subproject for a new Scala version

[email protected]@foo/publishSigned

User can define a new command or a subproject to aggregate 2 or more subprojects.

lazy val backPublishProj = (project in file("backPublish))
  .aggregate(foo, bar)
  .settings(publish / skip := true)

[email protected]@backPublishProj/publishSigned

case 2

Republish for 3.x (sbt plugin likely would do this)

[email protected]#comment

1. Branch off of v1.2.3 to create release/1.2.3 branch, and send a PR to bump sbt version
2. Republish the subproject

case 3

Publish one subproject for a new sbt plugin like Scala.JS or Scala Native.

v1.2.3@+foo_native/publishSigned#native=0.5.5

1. Branch off of v1.2.3 to create release/1.2.3 branch, and send a PR to bump the Scala Native version.
2. Republish the subproject.

[BillyAutrey]

Reading the code, it seems like the # is just a comment identifier - a note to let folks know they can "tag" git tags with similar scala versions, so they can differentiate their scalaJS/sbt/scala native versions from one another.

I assume this means that these branches/tags will exist for the whole set of supported version combinations that might exist. A scalaJS + multiScalaVersion sbt plugin is going to be pushing a lot of tags.

[eed3si9n]

I assume this means that these branches/tags will exist for the whole set of supported version combinations that might exist. A scalaJS + multiScalaVersion sbt plugin is going to be pushing a lot of tags.

Not sure what you mean here. I'm proposing this mechanism for back publishing purpose only, which means that a normal tag like v1.2.3 is used to publish a build matrix combination, let's say on 2024-01-01, and later, on 2024-06-01, a new version of something comes out, like a new Scala 2.13 release, and ScalaMeta folks wants to publish semanticdb compiler plugin for 2.13.15. I don't think this would necessarily increase the number of tags more than today. It would just not create new numbers as much.

[lrytz]

1. Branch off of v1.2.3 ...
2. Republish the subproject

OK, so the goal is not to handle each and every scenario only through the tag name. I think that's good, it would be way too open ended otherwise.

One could argue, if you have to create a branch with some patches for the backpublishing anyway, v1.2.3#someCommentToDrop would be enough. Bumping versions, selecting subprojects etc could be handled by the patch.

Branching is not needed to publish for a new Scala version, but that's only useful for fully cross-versioned projects, like compiler plugins.

[eed3si9n]

One could argue, if you have to create a branch with some patches for the backpublishing anyway, v1.2.3#someCommentToDrop would be enough. Bumping versions, selecting subprojects etc could be handled by the patch.

Yea, technically you can hardcode version, release command etc all in a branch, which is the reason the original issue was closed.

The benefit this PR could bring is making the process a bit smoother by removing the hardcoding. For example, if we bumped up the Scala Native version in a branch, that branch can still merge back into main.

[lrytz]

The feature of dropping a #.* suffix on the tag is helpful though, as the plain tag already exists when backpublishing.

So maybe that basic feature, plus documentation how to narrow down the build to publish only specific artifacts, is enough?

[eed3si9n]

The comment only route of the documentation would look like..

Case n: Publish some subprojects for Scala 3.x

1. Branch off of v1.2.3 to create release/1.2.3 branch, and send a PR to:
   a. bump appropriate dependency (sbt, Scala Native etc)
   b. Modify the CI_RELEASE environment variable to ;++3.x;foo/publishSigned. For GitHub Actions, it would be in .github/workflows/release.yml
2. Tag the branch to v1.2.3#some_unique_comment. For record keeping it would likely make sense to encode the version you're trying to back publishing for e.g. v1.2.3#native0.5_3

If some project wants to do that, they can do that, but I now you have to look at both the build and the YAML to understand what's going on, and the YAML is different for different CI systems (a committer may not even have access to change it, in case of Jenkins).

[lrytz]

Thanks Eugene. In any case, this PR enables the comment-only workflow you describe. If the additional feature is helpful, then I guess why not? I'm not a maintainer here and I'm certainly not going to be in the way :)