Add parsing of type alias statements i.e. the `type` keyword #97

zanieb · 2023-07-13T19:15:22Z

Extends #95
Closes #82

Adds parsing of new type soft keyword for defining type aliases.

Supports type alias statements as defined in PEP 695 e.g.

# A non-generic type alias
type IntOrStr = int | str

# A generic type alias
type ListOrSet[T] = list[T] | set[T]

# A type alias that includes a forward reference
type AnimalOrVegetable = Animal | "Vegetable"

# A generic self-referential type alias
type RecursiveList[T] = T | list[RecursiveList[T]]

All type parameter kinds are supported as in #95.

Builds on soft keyword abstractions introduced in RustPython/RustPython#4519

parser/src/soft_keywords.rs

parser/src/parser.rs

parser/src/python.lalrpop

parser/src/soft_keywords.rs

parser/src/parser.rs

MichaReiser · 2023-07-14T16:43:36Z

parser/src/soft_keywords.rs

+            // 2. The type token is followed by a name token.
+            // 3. The name token is followed by an equality token.
+            else if matches!(tok, Tok::Type) {
+                if !self.start_of_line {


I'll look at this on Monday. I need a fresh mind to process this

parser/src/soft_keywords.rs

charliermarsh · 2023-07-14T17:28:47Z

Nice job figuring this out. A couple questions to confirm my understanding of the soft-keyword heuristics.

parser/src/soft_keywords.rs

youknowone

Looks great, thank you so much.

MichaReiser · 2023-07-17T14:30:45Z

parser/src/soft_keywords.rs

+                                        Tok::Lpar | Tok::Lsqb | Tok::Lbrace => nesting += 1,
+                                        Tok::Rpar | Tok::Rsqb | Tok::Rbrace => nesting -= 1,


Do we need to support parentheses for the type case? I didn't see any test case that uses a parenthesized name.

I think we can be more strict in the type implementation and only skip over NonLogicalNewline, Continuation, and Comment. Whatever token comes after must be the = or it isn't a type assignment

I believe this is valid: type Foo[T] = ....

Yeah I cannot find a case where parens are allowed, although we do need to support brackets for the generic definition

Makes sense. I guess we then still need to support skipping over arbitrary tokens.

These are generally syntax errors and won't result in an identifier at runtime so I'm actually not sure if we want to change the type token to an identifier.

b2d67c7
af508c2

We can add more complex parsing in a follow-up that includes tests that assert syntax errors if we want.

MichaReiser

Ready to merge when you are

…als` placement

…hon#97) Extends RustPython#95 Closes RustPython#82 Adds parsing of new `type` soft keyword for defining type aliases. Supports type alias statements as defined in PEP 695 e.g. ```python type IntOrStr = int | str type ListOrSet[T] = list[T] | set[T] type AnimalOrVegetable = Animal | "Vegetable" type RecursiveList[T] = T | list[RecursiveList[T]] ``` All type parameter kinds are supported as in RustPython#95. Builds on soft keyword abstractions introduced in RustPython/RustPython#4519

zanieb force-pushed the parse/typealias branch 2 times, most recently from d04c741 to 52d1a41 Compare July 14, 2023 15:18

zanieb marked this pull request as ready for review July 14, 2023 15:18

zanieb commented Jul 14, 2023

View reviewed changes

parser/src/soft_keywords.rs Outdated Show resolved Hide resolved

MichaReiser requested a review from youknowone July 14, 2023 16:35

MichaReiser reviewed Jul 14, 2023

View reviewed changes

charliermarsh reviewed Jul 14, 2023

View reviewed changes

parser/src/soft_keywords.rs Outdated Show resolved Hide resolved

parser/src/soft_keywords.rs Outdated Show resolved Hide resolved

zanieb commented Jul 14, 2023

View reviewed changes

parser/src/soft_keywords.rs Outdated Show resolved Hide resolved

charliermarsh reviewed Jul 14, 2023

View reviewed changes

parser/src/soft_keywords.rs Outdated Show resolved Hide resolved

charliermarsh approved these changes Jul 14, 2023

View reviewed changes

zanieb mentioned this pull request Jul 14, 2023

Consolidate parsing of Expr::Name in TypeAliasName, MatchName, and Atom<Goal> #98

Open

youknowone approved these changes Jul 15, 2023

View reviewed changes

zanieb changed the title ~~Add parsing of type alias statements~~ Add parsing of type alias statements i.e. the type keyword Jul 15, 2023

zanieb added 16 commits July 17, 2023 09:12

Add parsing of type alias statements

f96642a

Format

b9d283f

Add tests from the CPython corpus

b5e0f57

Add more cases to type identifier test

50f1b6a

Add handling for soft keywords as type alias names

4b99996

Move soft keyword cases to proper test

2e95af7

Restore comment specific to match/case

bbedb61

Use match statement in soft keyword token handler

d1c319b

Add some multiline identifier tests

7bbd80e

Add multiline type alias tests with line continuations

1622c8f

Add test for parenthized multiline type as identifier

e2e1c2b

Add parenthized multiline type alias test case

8e5d0cc

Add test for type identifier assignment

4776f68

Require the name token to immediately follow the type token

4170b22

Fix newline indentation

8c81332

Refactor to avoid traversing the line unecessarily

6050b5e

zanieb force-pushed the parse/typealias branch from e7ac2ef to 6050b5e Compare July 17, 2023 14:12

MichaReiser reviewed Jul 17, 2023

View reviewed changes

zanieb added 2 commits July 17, 2023 09:41

Add multiline generic test cases

fcdd605

Drop support for parsing type aliases with parens and braces

b2d67c7

MichaReiser approved these changes Jul 17, 2023

View reviewed changes

Update type alias soft keyword heuristic to be more strict about `Equ…

af508c2

…als` placement

charliermarsh merged commit 704eb40 into RustPython:main Jul 17, 2023

charliermarsh mentioned this pull request Mar 16, 2024

Support for Python 3.12-style f-strings fpgmaas/deptry#611

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add parsing of type alias statements i.e. the `type` keyword #97

Add parsing of type alias statements i.e. the `type` keyword #97

zanieb commented Jul 13, 2023 •

edited

Loading

MichaReiser Jul 14, 2023

charliermarsh commented Jul 14, 2023

youknowone left a comment

MichaReiser Jul 17, 2023

charliermarsh Jul 17, 2023

zanieb Jul 17, 2023

MichaReiser Jul 17, 2023

zanieb Jul 17, 2023

zanieb Jul 17, 2023

MichaReiser left a comment

		Tok::Lpar \| Tok::Lsqb \| Tok::Lbrace => nesting += 1,
		Tok::Rpar \| Tok::Rsqb \| Tok::Rbrace => nesting -= 1,

Add parsing of type alias statements i.e. the type keyword #97

Add parsing of type alias statements i.e. the type keyword #97

Conversation

zanieb commented Jul 13, 2023 • edited Loading

MichaReiser Jul 14, 2023

Choose a reason for hiding this comment

charliermarsh commented Jul 14, 2023

youknowone left a comment

Choose a reason for hiding this comment

MichaReiser Jul 17, 2023

Choose a reason for hiding this comment

charliermarsh Jul 17, 2023

Choose a reason for hiding this comment

zanieb Jul 17, 2023

Choose a reason for hiding this comment

MichaReiser Jul 17, 2023

Choose a reason for hiding this comment

zanieb Jul 17, 2023

Choose a reason for hiding this comment

zanieb Jul 17, 2023

Choose a reason for hiding this comment

MichaReiser left a comment

Choose a reason for hiding this comment

Add parsing of type alias statements i.e. the `type` keyword #97

Add parsing of type alias statements i.e. the `type` keyword #97

zanieb commented Jul 13, 2023 •

edited

Loading