A common/shared development environment for solvcon / turgon code

[yungyuc]

Yesterday @j8xixo12 and I had a discussion about tracing for Python reference count. It occurred to me that it is non-intuitive for people working in the Python land to start tracing into such details of the interpreter implementation. I am wondering if it is a good idea to try to set up a common development / runtime environment for the code development related to solvcon (including https://github.com/yungyuc/turgon and other prototyping / research project). A starting point is to take https://github.com/yungyuc/hack into the solvcon and make improvements for common use (it's made as my personal environment).

If we go toward this direction, I think there might be some basic requirements:

1. Support both macOS and ubuntu.
2. Manage dependencies in the source-code level. It may build all dependencies from source code and retain the source code for debugger use.
3. Allow building a selected set of packages from repository / master. The minimal set includes cpython, numpy, and pybind11.
4. Allow having debug & release / optimized build at the same time.

Comments?

[tai271828]

I am not sure the context of tracing for Python reference count. Is it only for fun, or we want to resolve any issue?

For your reference, I use the follow development environment for SOLVCON:

• PyCharm: pros: easier to debug interactively during CESE marching and nice UI. cons: I am not sure if it is feasible to look down deeper to work together with c++ level and/or Python reference count.
• containers (docker and singularity): pros: easier for users to get well-prepared development environment. There are docker files and singularity recipe in contrib. Able to manage dependencies. In theory they could also work together with PyCharm. cons: learning curve to manipulate containers and know their limitation.
• Github workflow: pros: SOVLCON is running with github workflow, and github also provides macOS and Ubuntu environment. cons: unfriendly to debug. The turn-around time is a bit longer.

I suppose that the solution is mixed with all of the above items. Locally developing with containers alongside PyCharm on macOS or Ubuntu, and delegate github workflow to take care another OS.

[yungyuc]

Does PyCharm allow gdb/lldb to work with its Python debugger (pdb/pdb-like)? Yes, we need gdb/lldb for C++-level debugging, along with the debug symbols.

The same question goes to containers. I don't know well enough for gdb/lldb with containers, and my past attempts didn't succeed.

CI / remote execution is bad for debugging.

The development environment here would also serve as runtime environment. It's actually more like a runtime environment tailored for code development.

[j8xixo12]

@tai271828 I'm newbie at python, @yungyuc suggested that I could start with python memory management, that's why I tried to trace reference count.
I used pdb for python debug and gdb for C-level debug, but I still tried to figure out how to use gdb pull out python stack informations.

[tai271828]

As far as I know PyCharm is used for pure python project. For python/c++ mixed, I heard[1] Clion + its python plugin should work. Clion is also released by Jetbrains. If one has been PyCharm user, he/she should be familiar with UI of Clion.

I expected VSCode should also work for c++/python project, but I did not try it. Besides, VSCode is also supporting to debug with a container, and VSCode now is cross-platform.

[1] https://intellij-support.jetbrains.com/hc/en-us/community/posts/115000477130-How-to-handle-a-mixed-Python-C-C-project-

[yungyuc]

I use viscose with lldb. In my experience, when debugging in C++, it's annoying to miss the Python stack, but it's not the end of the world. I think having lldb/gdb scripts to pull out the Python stack while debugging will be sufficient. IDEs are not really necessary.

I prefer script-based solutions which will work for terminal, GUI IDE, as well as CI.

[tai271828]

@j8xixo12 @yungyuc if our common consensus is script-based + pdb or pdb-like + gdb is a must, the first thing comes to my mind is ipython.

ipython is script-based and supports gdb very well. Besides, ipython also support remote debugging so I expect it could be used for container development environment. There is a nice presentation shared by Scott Tsai in PyCon TW 2015 a.k.a PyCon Apac 2015. I would recommend this presentation to people who have been using ipython but want to know more advanced usages.

And this is the corresponding slide if you don't manage to find one.

offtopic: Hmmm interesting. I have no idea that github discussion could be threaded (replied per messages). Neat!

[yungyuc]

I think we can try them all. But it might be giving us a taste by starting with a common build system. We need support on both macos and Linux, and probably Windows in the future. A runtime that is independent to OS-specific packaging system may be necessary.

And I hope to be Anaconda-free. I think packaging is too much a problem in the time being.