Write down source compatibility guidelines between Scala dialects

[Blaisorblade]

Again and again, we want to merge pull requests which change the Scala source language.

We should evolve precise guidelines for how to handle them and where to draw the line.
I think changing the source language is the point of the exercise, but of course we need good changes.

Guidelines to decide what's in and what's out are hard, that's why I say "evolve"; but already now we should have some guidelines on how to do the breaking changes we want.

1. We should require good documentation for users, possibly of better quality than the existing Scalac documentation. I wouldn't rule out altogether forking the spec, though that's probably a hard sell unless somebody volunteers for that.
2. Of course, we should check for compatibility with the community build. Over time, we'll have to figure out what kind of breakage we accept.

[puffnfresh]

Here's the first guideline, from the blog post:

Other fixes, whilst affecting semantics, would do so only in a conservative way – programs which are valid when compiled with the Typesafe Scala compiler would have the same meaning and binary representation when compiled with the fixes in place.

[djspiewak]

Proposed generic informal goals:

1. Any project source that compiles and runs under scalac should compile and run under tlc with the same set of flags. Two differences are allowed: warnings issued, and classpath (e.g. tlc may require some classpath entries that scalac does not, or vice versa)
2. Sources that compile under both scalac and tlc with the same flags should have the same semantics.
3. Any project binaries produced by tlc should load cleanly from a scalac project and should be callable without FFI. In other words, tlc should produce Scala APIs. The exact bytecode metadata may be distinct, so long as it is compatible.

The first and second items should be easily testable just using projects in the Scala community (similar to what Typesafe does with its compatibility tests). The third item is somewhat harder and may require specific tests, but probably eased through the same sort of mass continuous integration.

[non]

It's possible that (1) is slightly too conservative. For example, consider the following valid Scala program:

\u006f\u0062\u006a\u0065\u0063\u0074\u0020\u004d\u0061\u0069\u006e\u0020\u007b
\u0020\u0020\u0064\u0065\u0066\u0020\u006d\u0061\u0069\u006e\u0028\u0061\u0072\u0067\u0073\u003a\u0020\u0041\u0072\u0072\u0061\u0079\u005b\u0053\u0074\u0072\u0069\u006e\u0067\u005d\u0029\u0020\u007b
\u0020\u0020\u0020\u0020\u0070\u0072\u0069\u006e\u0074\u006c\u006e\u0028\u0022\u0068\u0065\u006c\u006c\u006f\u0020\u0077\u006f\u0072\u006c\u0064\u0022\u0029
\u0020\u0020\u007d
\u007d

My proposed change to unicode escaping (#7) would make this currently valid Scala program fail to compile.

[puffnfresh]

@djspiewak @non well, now I'm conflicted.

[djspiewak]

@non For this reason, your Unicode stuff should be behind a flag. That is
implicit in my proposed restrictions.

[non]

Sure, possibly -Zsafe-unicode-escapes or similar.

[puffnfresh]

@non is it actually easy to preserve both behaviours?

[non]

Yes. In fact, my original patch does preserve both behaviors since we need to keep using it when parsing Java sources. If you look at the commit (sorry don't have a link) you'll see that I add a flag about whether to do pre-parse escaping or not.

[non]

(Of course, the patch would have to be updated to ensure that we don't "double parse" \uXXXX escapes before parsing and also during string parsing. But that's just work.)

[Blaisorblade]

So, we can still do anything sensible violating those guidelines, as long as it's behind a flag? I'd be happy with that, as long as we get -Zdefault and -Zall.

BTW, calling -Zdefault -Ztypelevel would be risky, because we might well end up with some feature for type-level computation some day.

Of course, the patch would have to be updated to ensure that we don't "double parse" \uXXXX escapes before parsing and also during string parsing. But that's just work.)

Generally speaking, we'll have to weight the maintenance overhead. Especially since we're keeping merge compatibility, so we can't go and simplify other stuff.

[non]

Right. Features that follow the "spirit" of these rules (expected to work widely with realistic Scala sources but break on edge cases or dysfunctional code) might be grouped into a -Zomnibus flag.

(Came up with the name on #typelevel but I'm sort of enamored with it now.)

[non]

One thing to add though is that as @milessabin said the fork is intended to be conservative.

The goal is not to add a ton of flags to create a totally different language. But rather, to use flags in cases where we can't quite meet our compatibility requirements but want to accept a benign change that meets the spirit of them.

[aloiscochard]

Indeed, It would be kind of ironic if compiling scalaz/shapeless/spire with the fork would require a complex combination of flags.

[non]

One other thing to note, in the rules above, rule (2) should probably be amended to rule out reflection.

Otherwise, it will be impossible to even add annotations to the stdlib (since that would change the behavior of valid code that gets all the annotations for a class).

This is relevant due to #17.

[Blaisorblade]

Otherwise, it will be impossible to even add annotations to the stdlib (since that would change the behavior of valid code that gets all the annotations for a class).

Since that's about adding something to a library, this reminds me: What about forward compatibility guarantees between Typelevel Scala minor releases? What's the binary version of Typelevel Scala releases?

[milessabin]

@aloiscochard actually, I wouldn't be at all surprised if shapeless or Scalaz were the first to break ... but that should be fine if we're prepared to branch those projects to support tlc (we are, that's kinda the point).

[Blaisorblade]

The discussion moved to https://github.com/typelevel/scala/wiki/Typelevel-Scala-Compatibility-Guide & https://groups.google.com/d/msg/typelevel/bCJhQvIhqG4/6y4nf5kFpJMJ.
The issue should be closed (IMHO) when the policy is in place.

[puffnfresh]

I've started documenting our features on the wiki: https://github.com/typelevel/scala/wiki/Differences

[propensive]

I also added a page called "Roadmap". It basically says we don't have one
(beyond what Miles published in his original announcement), though a couple
of people have asked me about our roadmap, so I thought I'd put it into
words...

https://github.com/typelevel/scala/wiki/Roadmap

That said, it would be good to open the discussion on what we should expect
to see in the first (and maybe later) releases of Typelevel Scala.
Thoughts, anyone?

Jon

On 6 October 2014 16:29, Brian McKenna [email protected] wrote:

I've started documenting our features on the wiki:
https://github.com/typelevel/scala/wiki/Differences

—
Reply to this email directly or view it on GitHub
#14 (comment).

Jon Pretty | @propensive