Python: Todo list

[maximiliankaul]

The experimental python branch has been merged to master. It currently supports some python language features but is far from complete. This issue is supposed to track open todos for the python implementation.

Tasks

Beta Give feedback

1. Python self and constructor support #402
   python +1
   
2. Publish to pip (to be discussed) -> not relevant with the Kotlin rewrite
3. Split implementation into multiple files
4. Python frontend: better argument handling #1653
   python +1
   
5. Support stub files from PEP-484 (.pyi) #1640
   python +1
6. Improved import handling in Python #1604
   python +1
   
7. Python frontend: Implement missing AST nodes #1654
   python +1
   
8. Implement support for multiple return values (see for example: https://github.com/Fraunhofer-AISEC/cpg/blob/06995d44afb21212c72fe18320324d76d562575a/cpg-language-python/src/main/kotlin/de/fraunhofer/aisec/cpg/frontends/python/ExpressionHandler.kt#L175 or related https://github.com/Fraunhofer-AISEC/cpg/blob/06995d44afb21212c72fe18320324d76d562575a/cpg-language-python/src/main/kotlin/de/fraunhofer/aisec/cpg/frontends/python/ExpressionHandler.kt#L246)
9. Field initializes
10. Python type_comments #1649
    python +1
11. slices (DFG)
12. constant propagation ((multi) value evaluator)
13. Python frontend: async functions #1651
    python +1
14. Python frontend: handle class keywords #1652
    python +1
    
15. handle fromat_spec for FormattedValues https://github.com/Fraunhofer-AISEC/cpg/blob/06995d44afb21212c72fe18320324d76d562575a/cpg-language-python/src/main/kotlin/de/fraunhofer/aisec/cpg/frontends/python/ExpressionHandler.kt#L89
16. Consistency: use private fun or internal fun consistently.
17. Python: improve scoping #1677
    python +1
18. Support multiple vars in a forloop #1702
    python +1
    
19. Add declaration pass: traverse EOG instead of AST
20. The Python.AST.type_ignore seems not implemented #1843
    python +1
    
21. The python AST has several types of nodes for which no dedicated handler exists #1841
    python +1
    
22. Support remaining python match patterns #1835
    python +1
    

Several nodes are unhandled in the Python frontend. This Issue is here to track whether we can and have implemented the parser node and what the current Ideas as on how we map these nodes to the current graph(existing nodes or a new node):

Tasks (StatementHandler)

Beta Give feedback

1. Python frontend: Handle delete statement #1709
   python +1
   
2. Handle global keyword #1712
   python +1
   
3. Match
4. Handle nonlocal keyword #1713
   python +1
   
5. Implement Raise in python #1710
   python +1
   
6. Initial version of python try with all things #1704
   python +1
   
7. TryStar
8. Implement python With #1711
   python +1
   
9. AsyncWith
10. Implement orElse branch all constructs, e.g. for-loops #1718
    python +1
    
11. GeneratorExp #1714
    python +1
    
12. ListComp #1715
    python +1
    
13. SetComp #1716
    python +1
    
14. DictComp #1717
    python +1
    
15. Await
16. Yield #1719
    python +1
    
17. YieldFrom

[KuechA]

Some notes concerning the above tasks:

Add declaration pass: traverse EOG instead of AST

This doesn't work because the Pass dependencies don't allow it.

constant propagation ((multi) value evaluator)

What is expected here? Basic concepts are available (i.e., +, -, etc.) If we need some specific information in a special scenario, we should probably extract this to another evaluator

Field initializes

What should we do here? This needs a more precise description

Slices (DFG)

Task description too imprecise

Match

Basic support/implementation in #1801 . Many of the cases are missing.