dev call 20210304

Dev Call: 4 March 2021

Present

Florian, Tim, Michael, Ronan, Antocuni, Simon, Armin

Updating PyPy

Matti would like to do a new release of PyPy. PyPy's HPy universal has fallen behind HPy master and it would be good to update it. There are two major HPy updates and many smaller ones. The major ones are:

• Debug mode was added.
• Legacy structs were added.

Debug mode shouldn't be too hard to add to PyPy (it's a wrapper around a generic HPyContext, so the challenges are largely including it in the build process and hooking it up).

Legacy structs might be harder to add (it requires using cpyext from HPy for the legacy structs and there might be some tricky interactions to take care of).

GraalPython and HPy

Florian gave an overview of GraalPython with some emphasis on how HPy will help GraalPython run C extensions more cleanly and efficiently.

The slides from Florian's presentations are available at https://securesites-prodapp.cec.ocp.oraclecloud.com/documents/link/LDA46DA216DFCE653D471899E2F60DED45C05EA61B32/fileview/D4AF9369F18428234687C5616BF5FA162766B603954D/_2018-03-04_Python_on_GraalVM.pdf

Florian also recorded his talk. If you would like access to the recording, please contact Florian directly (e.g. by asking on IRC in #hpy on Freenode).

tp_destroy

Florian brought up https://github.com/hpyproject/hpy/issues/175 for discussion.

Currently, tp_destroy by design does not receive the HPyContext as an argument. The intention is that tp_destroy should not attempt to retrieve the context or call into the Python runtime.

The underlying reason for these restrictions is that tp_destroy will be called by the garbage collector (GC). Exactly when is up to the implementation of the GC and it could be called from a separate GC thread running in parallel to the rest of the interpreter. Thus it is safest if tp_destroy is thread-safe and does not access any interpreter state or call any interpreter methods.

All of these restrictions need to be documented.

In simple cases tp_destroy only needs to free any additional memory associated with the instance being destroyed. When this memory contains no Python objects, tp_destroy has no need for the HPyContext.

Currently there is no way within HPy to store long-lived handles to Python objects. The plan is to add HPyField and is described in https://github.com/hpyproject/hpy/issues/9. tp_destroy will possibly need to free such `HPyField`s and this will introduce complications.

The use of HPyField`s will very likely require the use of a slot similar to `tp_traverse that will allow the runtime to determine the location of HPyField`s. It is possible that we might use this to assist in keeping the job required of `tp_destroy fairly simple and context independent, but we will need to start implementing HPyField and see what happens.

In the old C API, the storage of PyObject *`s inside the memory of custom type was fairly common (e.g. a custom type might hold an array of `PyObject`s). While `HPyField is being implemented, we will need to document how to port such types to HPy and in particular, how to replace calling PyDECREF inside tp_dealloc.

Changing HPyContext to HPyContext *

In issue https://github.com/hpyproject/hpy/issues/150 Antonio suggested renaming HPyContext to HPyContext * to make it explicit that the ctx passed to many HPy functions is a pointer.

In the very early days of HPy the ctx was a little more opaque so having the ctx not obviously be a pointer made sense. However, the introduction of constants to the ctx lead to it being fairly common to write code such as ctx->h_None where the ctx is dereferenced to look up a member.

Given that it is now common to explicitly derefence the ctx, it seems like a good idea to make it explicit that it is a pointer to the context.

This is a disruptive but largely cosmetic change. After a short discussion, there was unanimous agreement to make this change now. It will only become harder to make as the number of places in which HPy is used grows.