Segmentation fault when attempting to open Kilosort4 output in Phy #1287
Comments
|
Could I also get OS? Also did you switch the numpy through conda, pip or at the yaml level? Another edit: could I see the file structure (either list of files present or screen shot of your file explorer/ finder). |
|
Ubuntu 22.04.2 LTS; yaml level. amplitudes.npy |
|
Do you get any pop errors or any others in the terminal when this happens or does the seg fault happen silently? |
|
Silently. |
|
Could you post the contents of As the line where you seg faulted. |
|
params.py
phy.log
|
|
Also I want to confirm what you mean at the beginning here:
Do you mean that all formats fail with Phy (no matter how you run KS4) or that one of them works? (or to ask another way is phy globally failing or failing for specific use cases). |
|
Sorry, I mean that all outputs from SpikeInterface fail. Most outputs run without issue if they are from Kilosort4 on its own (not via SpikeInterface), but some also fail with a segmentation fault. However, Phy loads any output in the trace-gui, so all of these issues apply only to the template-gui. |
|
It's weird because the issue is occurring while we are still making the gui. Sometimes we get index errors with kilosort because it extra spikes (past recording edge) sometimes. Are you doing I'm trying to track this down to: I had thought it was Phy. The best way to test would be could you try spikesorting the same data with mountainsort5, spykingcircus 2 or TDC2-then export to phy and then try to load with phy and see what happens? If we can't piece this together we may need you to share a "fail" dataset that failed even from native KS4 to try to track this down. What is the binary file you're using when you use native KS4? Could you copy the params.py from a native KS4 file that failed and one that worked? |
|
If it helps, here are the output files when I use Kilosort4 alone (not with SpikeInterface) for a case that fails (segmentation fault):
And here are the files for a case that loads without issue, from the non-SI output:
The only difference is the lack of a cluster_info.tsv in the first list. |
|
When using SpikeInterface, I use |
|
So could you try sorting with a different sorter, do export to phy and then see if it repeats. Then maybe we narrow back down to spikeinterface and we actually switch back to your other issue. If it is fine with the other sorter than it might be how spikeinterface is exporting the KS4 data itself (still an SI issue, but KS4 has been changing so much it is tricky to keep up). |
|
Update: The KS4 output files that I was previously able to load in Phy are now giving me the segmentation fault. Maybe this is an issue with my system? Could it be memory related? |
|
Typically seg faults occur with things working at the c-api level. This could be something between python-numpy-qt (but looking at your python and numpy I think you should be fine there). The other issue could be that KS is returning some sort of value that is leading the gui to try to access memory it shouldn't. so I have to go back to my previous statement. We need to test another sorter and exporting to phy. |
|
I tested SpyKING CIRCUS, but I get the same error:
|
|
I'm having a similar problem when opening phy2 on Ubuntu 22.04 but not in Win10. 19:07:23.349 [W] model:667 Skipping spike waveforms that do not exist, they will be extracted on the fly from the raw data as needed. Here is my Conda env list: # Name Version Build Channel |
|
I think this is either a python point release issue. Last time I checked it was 3.11.4 or .5 I think and we are onto 3.11.9. Could you try installing with python 3.10 instead and see if that fixes it. The other potential issue would be a point release in any of the qt packages. Until I get the CI working I don't have access to a linux system to test this myself. So If you're willing to put in the time to find the exact point releases we need for linux we can limit this in the install instruction. But since this works on Windows I'm not sure how I can recreate the python + qt stuff to find the exact problem. |
|
I installed python 3.10 but that did not solved the problem. Here is the debug info: 13:38:11.975 [D] init:68 Start capturing exceptions. |
|
The following installation(#1283 (comment)) solve my problem in Ubuntu 22.04: name: phy2_test |
|
Update: After restarting my computer (which runs Ubuntu 22.04, and Python 3.11 in my SI, KS, and Phy envs), I actually can load most outputs into Phy, regardless of whether or not I used SI. However, in most cases I get a new message upon loading the data (see below). Also, importantly, no waveforms are loaded and the columns for KSLabel, ContamPct, and Amplitude are missing. In a few cases, I get a new kind of error (see second readout below). I didn't update or change any environments or versions, so I'm not sure why restarting the computer resulted in such different behavior, or why I get varied behavior: out of nine datasets, one loads and shows waveforms; six load but don't show waveforms; and two don't load and give the error below (see table below). I have yet to get the segmentation fault again after restarting.
|
|
@410pfeliciano I created a new environment based on what you shared above, but it did not solve the problem for me. Here is the debug report:
Seems like the issue might be mismatched dependencies due to version settings. I'm using NumPy 1.24.1, and have tried it with Python 3.11, 3.10, and 3.9. Maybe there is some other package that needs a specific version? |
|
Try the following: |
|
I tried that, but due to other dependencies, the latest version I could install was 1.25.2. And I get the same error:
|
|
that error I actually know. It is because the NumPy limits aren't quite correct. So you installed some versions of packages necessary for NumPy 2.0 but when you downgraded to Numpy 1.25 or 1.26 etc it didn't downgrade the other packages. This is an unfortunate side effect of how pip resolves from the toml. So my advice would be to delete the conda environment all together and make a new one where you limit the numpy during the conda creation because if you install the wrong numpy during the conda install pip might not be able to fix it even if it downgrades the numpy itself. Does that make sense? But I think we are close. |
|
I also create #1289 if you want to test with the built in limit to numpy for making your environment. |
|
That makes sense, but what I previously did sounds like what you've described. I created a new Phy environment from a new .yml:
Should I manually set every package version to be compatible with the NumPy version? |
|
Oh, sorry I misunderstood. I thought you were just doing a pip install numpy=1.25 which would downgrade numpy but wouldn't necessarily fix the other packages. If you were installing from the beginning then one of the other packages must still be installing a too recent version. If you resize the window can you see where the pickle is being called from? ie can you give us a fuller file path in the debug window? Or share the log (note the log will have full file paths so edit that if you don't want all file paths displayed). |
|
Update: After restarting the computer and removing all but the most functional Phy environment (see package list below), I am able to load all Phy output from Kilosort4 and see the data as normal. When instead I try to load the output from SpikeInterface after running KS4 via the Docker, there is always one or more of the following problems: no waveforms, unusual waveforms, waveforms not shown according to the probe layout, or the numpy._core error. The same is true if I use a non-KS sorter. So my Phy-related problem is solved enough for me to move forward. I'm giving up on SI for now, but I can open a new issue on the SI page if you'd like (or update the previous issue). Thanks for your help!
|
Yeah based on this it seems like maybe some numpy stuff is has made our exporter not fully work. Feel free to reopen and update the old issue! |
|
I also encounter the Segmentation fault (core dumped) issue when trying to input my data (generated with kilosort4) into Phy2 on Ubuntu 24.04.1 LTS. I was able to solve the problem using the |
|
Just wanted to add that I used the yml file that @a-savoy provided to create a new environment and that completely fixed the issues I was getting with opening Kilosort4 outputs in Phy. |
|
@nataliekoh Wow, something I posted on Github actually helped someone?? This might be the happiest moment of my entire grad school experience! 🥹 |
When sorting the continous.dat file from a Neuropixels 1.0 recording, I have no issues sorting and preparing the output to Phy using Kilosort4 (all default settings), via the Docker container with SpikeInterface or without using SpikeInterface. I created a Phy environment from the environment.yml file (the NumPy version was an issue, but it worked after I changed the version to 1.24.1). When I run
phy trace-gui params.pythere is no issue, but when I runphy template-gui params.py, I see the following, regardless of the dataset:If I run Kilosort4 outside of SpikeInterface, sometimes Phy can open the output, but sometimes I get the same segmentation fault.
Here is an example debug output:
Here are the packages in the Phy environment:
The text was updated successfully, but these errors were encountered: