Handling nodes with files

[ltalirz]

For the JSON-serializable part of AiiDA entities, great libraries like pydantic exist that make it easy to integrate a model definition into a web API (be it via a corresponding REST endpoint or a GraphQL model).

In AiiDA, however, nodes can have one or more attached files.
Those binary blobs could in principle be included in a JSON representation via BASE64 encoding, but this would add significant computational and memory overhead (these files may not be small).
We therefore want a way to handle these files separately from the JSON-serializable information stored in the datatabase.

This poses a couple of questions:

1. How should the serialization of a "node with attached files" look like?

2. How do we define the model of a node with associated files in aiida-core in such a way that a web API for that node type can be derived automatically?

3. How should the web API for up/downloading a "node with attached files" look like?

Let's use this thread to discuss these questions

[ltalirz]

3. upload

At the HTTP level, there are at least two approaches one could take:
A) Upload metadata and files in the same request (multipart/form-data)
B) Upload metadata first, returning a link for file upload(s). Upload file(s) in subsequent requests

B) is more flexible but involves potentially a large number of requests per node. It requires additional logic on the client side, as well as on the server side (e.g. to delay creation of the node on the server until all files have been received).

A) is the route chosen in the current implementation of the SinglefileData REST endpoint in aiida-restapi.
multipart/form-data, unless given a specific MIME type, will send files as the binary application/octet-stream. While the implementation in the requests client library requires all data to be loaded into memory before the request is sent, requests_toolbelt contains methods for streaming multipart/form-data uploads directly from disk.

It appears that route A) is a sensible way forward.

3. download

Creating an AiiDA entity on the server ("upload") is typically an "all or nothing" operation, since you cannot modify a node's attributes / file repository after its creation.
Download is different: the user may just be interested in some of the information (or some files) provided by the node [1]

The aiida-core REST API has specific /nodes/<id>/repo/list and /nodes/<id>/repo/contents endpoints for listing and downloading files associated with a given node one by one.
This is probably a sensible approach, although it does mean that users who do want all files associated with a given node will again need to make a potentially large number of requests.

In any case, it should be straightforward to add a "give me all" endpoint that returns a zip archive of the JSON-serialized information of a node, together with its associated files from the repository, if the use case arises.

[1] And, indeed, the GraphQL API gives the user the freedom to choose exactly which fields of the model to receive, see the example.