loop expressions / collection-for alternative

[hydro63]

Motivation / Inspiration

This proposal is proposes a sort of alternative / anti-thesis to the current collection-for we have, it would allow the collection-for to be more powerful and expressive, and it would solve almost all the proposals asking for extending the collection-for. The inspiration / motivation comes from #4139. I've also searched for similar proposals, because i was sure there had to be one already, but there was none exactly to my liking. The most similar proposal i found is #1633. If there is some proposal i've overlooked, please point me to it.

Proposal

The proposal is straightforward. I propose allowing for / while to be used as expressions. The return value for the loop expression would be Iterable<T> / Stream<T>, which would be lazily evaluated. The values would be passed to the output iterable with a yield keyword. Basically, you can think of it as a modification on proposal #1633, which makes it imo easier and more compact to use.

// example from #4139
// iter -> Iterable<int> = inferred
var iter = for(final item in [1,2,3]){
    if(item < 2) continue;
    doSomething();
    doSomethingElse();
    yield item + 3;
  }

// iter -> Iterable<num> = explicitly typed
var iter = for<num>(final item in [1,2,3]) yield item*2;

// stream -> Stream<int>
var stream = for(final item in [1,2,3]) async {
  await foo;
  yield item * 2; 
}

// embedded into list / list comprehension
final list = [
  firstElement,
  ... for(final item in [1,2,3]){
    if(item < 2) continue;
    doSomething();
    doSomethingElse();
    yield item + 3;
  },
  lastElement,
];

This allows it to be used outside the collections, and allows for easier representation of complex operation on some list.

collection-for and for as expressions interaction

These two paradigms don't mix well / use the same syntax which makes them ambiguous, which is which. That is a big problem, of which i'm aware of. This proposal is more of an antitheses to the collection-for and as such it is a different look at how to do list comprehention. One way to fix the ambiguity is to assume all for is used as expression and just spread the resulting iterable. This could be easily done with a migration script.

// before migration script
final list = [
  firstElement,
  for(final item in [1,2,3]) item + 3,
  lastElement,
];

// after migration script
final list = [
  firstElement,
  ... for(final item in [1,2,3]) yield item + 3,
  lastElement,
];

// possibly the yield keyword could be removed if there is only one statement

Other

I personally think that that for as expressions is a much better way to do list comprehention compared to the python-style collection-for we have today. My main argument for this is that for as expressions allow for a lot more flexibility than collection-for without using any of the usefulness (they can also be easily embedded into collections by spreading).

[tatumizer]

How do you differentiate between for-expression and for-statement inside sync* function?
I suggest that you have to add some marker for this. For uniformity, it could look like this:

var x= sync for(int i=0; i<10; i++) yield i*2;
var y = async for(int i=0; i<10; i++) yield (await foo())*2;

[hydro63]

@tatumizer I don't think for-expression and for-statement is ambiguous. It would be parsed as for-expression everywhere an expression is expected, similarly to switch-expression. The yield in for-expression would be scoped only to the for-expression and would not yield value from the parent generator.

// (0, 1, 2, 3, 4, 2, 550)
Iterable<int> gen() sync* {
  for(int i=0; i<5; i++){
    yield i;
  }

  yield 2;
  yield 550;
}

// the for-expression yield doens't yield to main function, only the resulting for iterator
// (2, 550, 0)
Iterable<int> gen() sync* {
  var iter = for(int i=0; i<5; i++){
    yield i;
  };
  
  yield 2;
  yield 550;
  yield iter.first;
}

If you are talking about making it more readable and accessable to the users, it is also possible to add a new keyword pass, that would replace the proposed yield in the for-expression. I think it is also a viable approach, since it would work nicely with block expression, where we would be able to return value from block by passing it.

// pass keyword with for-expression
Iterable<int> gen() sync* {
  var iter = for(int i=0; i<5; i++){
    pass i;
  };
  
  yield 2;
  yield 550;
}

// pass keyword with block expression
print({
  var a = 5;
  a *= 10;
  pass a;
});

// pass keyword with multistatement switch expression
var value = switch(...){
  // other patterns
  ...

  int age => {
    if(age > 60) pass "Too old";
    if(age < 18) pass  "Too young";
    pass "Good age";
  }
  _ => "Undefined"
}

It's also possible that there is some syntax problem i'm not aware of, and if that's the case, please provide example so i can understand it.

[mmcdon20]

Technically there is already a way to generate an Iterable or Stream in expression form. You just have to wrap your for loop in an anonymous generator function, and then immediately invoke the function.

So this example

var iter = for(final item in [1,2,3]){
    if(item < 2) continue;
    doSomething();
    doSomethingElse();
    yield item + 3;
  }

can be written in current dart as

var iter = () sync* {
  for (final item in [1, 2, 3]) {
    if (item < 2) continue;
    doSomething();
    doSomethingElse();
    yield item + 3;
  }
}();

[hydro63]

I'm aware that i can do that (it was mentioned in #1633, if i remember correctly), but i don't really like the syntax. It's sort of unintuitive to read, and also clunky (you have to wrap it inside a generator, and call it).

I understand that it's more or less just a sugar, and it isn't really needed, since it doesn't provide any new functionality, but i'd argue it does. Currently, i'd say the reason why people never do it, despite the benefits over collection-for, is that it is really clunky to do it. I think that if it were implemented, it would be used at least somewhat frequently.

Still, that's just my biased opinion, but imo the existence of the generator way does not meaningfully justify disregarding this proposal.

[tatumizer]

@hydro63: interpretation of expression should not depend on whether its value is assigned to some variable or not. E.g. those are both valid expressions:

var x = a > b ? a : b;
a > b ? a: b;

Dart's "switch expression" is not a typical "expression" - it's basically a hack; the support of two totally different syntactic forms of switch is one of the most unfortunate parts of the language IMO.

[hydro63]

@tatumizer that's why i've provided another way, where the yield keyword would be replaced by pass keyword so that it wouldn't cause any problems

I know that making a new keyword is a last resort, but that's why i've provided other uses for the new keyword, where it could be used in other proposals. Also, i've found another proposal where it could be used to make a feature less confusing - #4141 with a return pattern.

var x = foo case Foo(:String prop && pass prop) ||(String() && pass);

Still, i know that's not perfect, but since this feature is more or less a syntactic sugar, i don't want to compromise on the ease of writing it, if possible. The sync and async keywords bloat the syntax, so i'm not a fan of it.

[lrhn]

This is different from Iterable literals, and probably more viable
Iterable literals suffer from being code that is implicitly asynchronous, it needs to be able to suspend at each yield, but it looks like normal code. (So does sync*, but it's delimited by a function body. Iterable expressions are in the middle of other expressions. They need serious synthetic help to be viable, to the point where (() sync* {...}()) doesn't feel like that bad an alternative.)

This proposal would run synchronously and emit all the elements while building the collection literal.
But it's really just a way to end the imperative way to build a collection in the middle of the declarative element notation.
The "statements + yield" could be written as actual statements + collection.add. The only thing you gain by doing it inline is that you can write more normal elements afterwards and that you can write statements inside expressions.

The latter is something you can't do today.
It would effectively allow

[sync* {
  Any statements whatsoever, with no yield; 
  yield value;
}].last)

to execute any statements (except yield for an outer function) inside an expression, and evaluate to a value.

I think we might want to consider the consequences of that. (What if it does return, break or continue, assuming we haven't already made those expressions?)

So let's look at it from that perspective instead: of we have a functionality to allow statements inside an expression which evaluates to a single value, should that be extended to allow statements inside an element that emotes zero or more elements.
I wouldn't do only the latter, and the two functionalities should be designed together.

[hydro63]

This proposal would run synchronously and emit all the elements while building the collection literal

That is possible, and probably the best way to do it, since i suppose it would see most use with flutter.

(What if it does return, break or continue, assuming we haven't already made those expressions?)

In my mind, continue and break work the same as with any loop, and would not emit value. continue would just continue with next iteration, and the break would break out of the loop and end evaluating the iterable. return could either return from the parent function, or be disallowed. I personally am for disallowing return in the iterable literal altogether.

I wouldn't do only the latter, and the two functionalities should be designed together.

You are right, that statements in expressions should be designed together, if this were to be implemented, but AFAIK that is the blocked expression proposal.

Also, i'm starting to realise the limitations of the purely for syntax that i proposed, and i'm starting to warm up to the sync* for syntax @tatumizer proposed.

[tatumizer]

What if it does return, break or continue...

An interesting point about return. Inside sync* function, return means "no more values"
If we re-make some sync* function into a sync* for block, as proposed, return should preserve that meaning.
Which precludes the interpretation of "return" as "return from the containing function".
This, in turn, means that in a hypothetical block-expression (discussed elsewhere) we must also preserve that meaning.

f() {
  var a = do {
     //...
     return 0; // return from the do-block, not from f
  }
}

As you remember, the controversy about the treatment of return has been a major stumbling block in the discussion of block expressions (pun not intended).

@lrhn: could you warm up to an idea of return by default meaning "return from the block" now as we have a strong argument for it?
There's an option of using another form of "labelled return" like return f:0, break myLabel etc, but inside block-expressions of any kind (do, for, whatever), the label should be mandatory whenever we refer to the code outside the current block

(Expression block can take several forms: do {...}, sync* for, async for, do for-else, do async for-else etc).

[munificent]

Back when I was first working on collection-for, one of the things I really struggled with was whether to allow blocks and statements inside the collection literal. Doing so is obviously useful, because it means you can do things like define local variables for computations before emitting values.

The main problem I ran into is a question of defaults:

1. Is the body of a collection for implicitly a yielded value and you have to opt out of that to use statements?
2. Or is the body implicitly a statement and you have to opt out of that to yield a value?

The feature we shipped basically takes approach #1 (except that there is no real support for opting in to statements/blocks). Your proposal here is #2.

As you note, it has the advantage of being more powerful. You can put all kinds of statements in there. But it comes at the expense of being imperative instead of declarative. The nice thing about an expression like a collection literal is that it's inherently declarative: it implicitly is a value. You don't have to do make in some imperative way. It just is.

That's one of the reason why Flutter widget building code can be so nicely declarative and markup-like: Often a build() method is just one big expression with nothing really imperative going on. I wanted to allow control flow in collections while still keeping the resulting code as otherwise declarative as possible.

If we made the default be "block of statements" then you'd have to imperatively yield each element value. That makes common simple cases more imperative and verbose than I wanted. Here are a few random samples of collection-for in some pub packages with the current syntax and what you propose here:

// Current:
[
  for (final column in indexedColumns)
    IndexedColumn(Reference(columnName: column.nameInSql)),
]

// Proposed:
[
  for (final column in indexedColumns) {
    yield IndexedColumn(Reference(columnName: column.nameInSql));
  }
]

And:

// Current:
[
  if (title != null) ...{
    Text(
      title,
      style: const TextStyle(
        color: Color(0xFF212124),
        fontSize: 16,
        fontWeight: FontWeight.w500,
      ),
    ),
    const SizedBox(height: 8),
  },
  for (var child in children) child,
]

// Proposed (I'm assuming here that we do something similar for if):
[
  if (title != null) {
    yield Text(
      title,
      style: const TextStyle(
        color: Color(0xFF212124),
        fontSize: 16,
        fontWeight: FontWeight.w500,
      ),
    );
    yield const SizedBox(height: 8);
  },
  for (var child in children) {
    yield child;
  }
]

And:

// Current:
[
  for (
    int i = 0;
    i < widget.footerCards!.length;
    ++i
  ) ...[
    if (i > 0) ...[
      SizedBox(
        height: TodaySpacing.tdsSpace2.spacing,
      ),
    ],
    widget.footerCards![i],
  ],
]

// Proposed:
[
  for (
    int i = 0;
    i < widget.footerCards!.length;
    ++i
  ) {
    if (i > 0) {
      yield SizedBox(
        height: TodaySpacing.tdsSpace2.spacing,
      );
    },
    yield widget.footerCards![i];
  },
]

And:

// Current:
[
  for (final frameData in data.frames)
    SpriteAnimationFrame(
      Sprite(
        image,
        srcSize: frameData.srcSize,
        srcPosition: frameData.srcPosition,
      ),
      frameData.stepTime,
    ),
]

// Proposed:
[
  for (final frameData in data.frames) {
    yield SpriteAnimationFrame(
      Sprite(
        image,
        srcSize: frameData.srcSize,
        srcPosition: frameData.srcPosition,
      ),
      frameData.stepTime,
    );
  }
]

There's definitely some nice stuff about what you propose. Users wouldn't have to use the ...[ stuff ] pattern as much. But I feel like the simple cases get a little more verbose.

This is obviously a biased sample because I'm only including cases that are already using collection-for. Since what you propose is more expressive, then more code would likely be able to use collection-for that isn't able to today (for example if it wants to declare a local variable).

I think this area is worth exploring, but it's tricky to make collection literals more expressive without leading simple cases to be more verbose. I also feel like at some point if you're doing a lot of imperative behavior... maybe it's good to hoist that logic out of a collection literal anyway?

[munificent]

See also: #4024.