Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bugs in the Jupyter notebook launcher #91

Merged
merged 7 commits into from
Nov 13, 2024
Merged

Fix bugs in the Jupyter notebook launcher #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1b26e47
[BUGFIX] Use correct path to environment binary directory
gkaf89 Nov 13, 2024
d31172d
[BUGFIX] Ensure that the Jupyter server is listening at the loopback IP
gkaf89 Nov 13, 2024
ea77d18
[BUGFIX] Fix broken internal links
gkaf89 Nov 13, 2024
89907a4
[STYLE] Remove dead-space
gkaf89 Nov 13, 2024
24f0bed
[STYLE] Use a more appropriate name for the cluster key
gkaf89 Nov 13, 2024
560ff44
[STYLE] Present more demonstrative options in port forwarding example
gkaf89 Nov 13, 2024
96f5ae8
[BUGFIX] Fix typo in the port forwarding example descriptions
gkaf89 Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions docs/connect/ssh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -522,31 +522,33 @@ The ssh agent is [not configured in the login nodes](#on-ulhpc-clusters) for sec

To avoid exposing your keys at your personal machine, create and share a new key. Create a key in your local machine,
```bash
ssh-keygen -a 127 -t ed25519 -f ~/.ssh/hpc_id_ed25519
ssh-keygen -a 127 -t ed25519 -f ~/.ssh/ulhpc_id_ed25519
```
and then copy both the private and public keys in your HPC account,
```bash
scp ~/.ssh/hpc_id_ed25519* aion-cluster:~/.ssh/
scp ~/.ssh/ulhpc_id_ed25519* aion-cluster:~/.ssh/
```
where the command assumes that you have setup your [SSH configuration file](#ssh-configuration). Finally, add the key to the list of authorized keys:
```bash
ssh-copy-id -i ~/.ssh/hpc_id_ed25519 aion-cluster
ssh-copy-id -i ~/.ssh/ulhpc_id_ed25519 aion-cluster
```
Then you can connect without a password to any compute node at which you have a job running with the command:
```bash
ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-aion.uni.lu:8022 ${USER}@<node address>
ssh -i ~/.ssh/ulhpc_id_ed25519 -J ${USER}@access-aion.uni.lu:8022 ${USER}@<node address>
```

In the `<node address>` option you can use the node IP address or the node name.

#### Port forwarding over SSH jumps

You can combine the jump command with other options, such as [port forwarding](#ssh-port-forwarding), for instance to access from you local machine a web server running in a compute node. Assume for instance you have a server running in `iris-014` and listens at port `2222`, and that you would like to forward the port `2222` to the `2222` port of you local machine. The, call the port forwarding command with a jump though the login node:
You can combine the jump command with other options, such as [port forwarding](#ssh-port-forwarding), for instance to access from you local machine a web server running in a compute node. Assume for instance that you have a server running in `iris-014` and listens at the IP `127.0.0.1` and port `2222`, and that you would like to forward the remote port `2222` to the `1111` port of you local machine. The, call the port forwarding command with a jump though the login node:

```bash
ssh -J iris-cluster -L 1111:iris-014:2222 <cluster username>@iris-014
ssh -J iris-cluster -L 1111:127.0.0.1:2222 <cluster username>@iris-014
```

This command can be combined with [passwordless access](#passwordless-ssh-jumps) to the cluster node.

## Extras Tools around SSH

* [Assh](https://github.com/moul/advanced-ssh-config) - Advanced SSH config is a transparent wrapper that make `~/.ssh/config` easier to manage
Expand Down
101 changes: 51 additions & 50 deletions docs/services/jupyter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ $ python -m pip install jupyterlab
```

!!! warning
Modules are not allowed on the access servers. To test interactively Singularity, remember to ask for an [interactive job](../jobs/interactive.md) first using for instance the `si` tool.
Modules are not allowed on the access servers. To test interactively Singularity, remember to ask for an [interactive job](/jobs/interactive) first using for instance the `si` tool.

Once JupyterLab is installed along with , you can start to configure your installation setting the environment variables corresponding to your needs:

Expand Down Expand Up @@ -50,7 +50,7 @@ JupyterLab makes sure that a default [IPython kernel](https://ipython.readthedoc

You can [setup kernels with different environments on the same notebook](https://ipython.readthedocs.io/en/stable/install/kernel_install.html). Create the environment with the Python version and the packages you require, and then register the kernel in any environment with Jupyter (lab or classic notebook) installed. For instance, if we have installed Jupyter in `~/environments/jupyter_env`:
```shell
source ~/environments/other_python_venv/bins/activate
source ~/environments/other_python_venv/bin/activate
python -m pip install ipykernel
python -m ipykernel install --prefix=${HOME}/environments/jupyter_env --name other_python_env --display-name "Other Python env"
deactivate
Expand All @@ -70,7 +70,7 @@ will make your conda environment, `conda_env`, available in the kernel launched

## Starting a Jupyter Notebook

Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The following script is a template for Jupyter submission scripts that will rarely need modifications. Most often you will need to modify the session duration (`--time` SBATCH option).
Jupyter notebooks must be started as [slurm jobs](/jobs/submit). The following script is a template for Jupyter submission scripts that will rarely need modifications. Most often you will need to modify the session duration (`--time` SBATCH option).

!!! example "Slurm Launcher script for Jupyter Notebook"
```slurm
Expand All @@ -92,20 +92,21 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
module load lang/Python
source "${HOME}/environments/jupyter_env/bin/activate"

declare loopback_device="127.0.0.1"
declare port="8888"
declare connection_instructions="connection_instructions.log"

jupyter lab --ip=$(hostname -i) --port=${port} --no-browser &
jupyter lab --ip=${loopback_device} --port=${port} --no-browser &
declare lab_pid=$!

# Add connection instruction
echo "# Connection instructions" > "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "ssh -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:${loopback_device}:${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "To access the jupyter notebook if you have setup a special key (e.g ulhpc_id_ed25519) to connect to cluster nodes execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:${loopback_device}:${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "Then navigate to:" >> "${connection_instructions}"

Expand All @@ -118,7 +119,7 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
--retry-delay 1 \
--retry-connrefused \
--silent --show-error --fail \
"http://$(hostname -i):${port}" > /dev/null
"http://${loopback_device}:${port}" > /dev/null
# Note down the URL
jupyter lab list 2>&1 \
| grep -E '\?token=' \
Expand All @@ -127,7 +128,6 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
>> "${connection_instructions}"

# Save some debug information

echo -e '\n===\n'

echo "AVAILABLE LABS"
Expand All @@ -147,14 +147,13 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
jupyter kernelspec list

# Wait for the user to terminate the lab

wait ${lab_pid}
```

Once your job is running (see [Joining/monitoring running jobs](/jobs/submit.md#joiningmonitoring-running-jobs)), you can combine
Once your job is running (see [Joining/monitoring running jobs](/jobs/submit#joiningmonitoring-running-jobs)), you can combine

- [`ssh` forwarding](/connect/ssh.md#ssh-port-forwarding), and
- an [`ssh` jump](/connect/ssh.md#port-forwarding-over-ssh-jumps) through the login node,
- [`ssh` forwarding](/connect/ssh#ssh-port-forwarding), and
- an [`ssh` jump](/connect/ssh#port-forwarding-over-ssh-jumps) through the login node,

to connect to the notebook from your laptop. Open a terminal on your laptop and copy-paste the ssh command contained in the file `connection_instructions.log`, and then navigate to the webpage link provided.

Expand All @@ -164,23 +163,23 @@ to connect to the notebook from your laptop. Open a terminal on your laptop and
# Connection instructions

To access the jupyter notebook execute on your personal machine:
ssh -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
ssh -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]

To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:
ssh -i ~/.ssh/hpc_id_ed25519 -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
To access the jupyter notebook if you have setup a special key (e.g ulhpc_id_ed25519) to connect to cluster nodes execute on your personal machine:
ssh -i ~/.ssh/ulhpc_id_ed25519 -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]

Then navigate to:
http://127.0.0.1:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
```

As the instructions suggest, you access the jupyter lab server in the compute node by calling
```shell
ssh -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
ssh -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]
```
an SSH command that

- opens a connection to your allocated cluster node jumping through the login node (`-J [email protected]:8022 [email protected]`), and
- exports the port to the jupyter server in the local machine (`-L 8888:172.21.12.29:888`).
- exports the port to the jupyter server in the local machine (`-L 8888:127.0.0.1:8888`).

Then, open the connection to the browser in your local machine by following the given link:
```
Expand All @@ -192,43 +191,48 @@ The link provides the access token, so you should be able to login without a pas
!!! warning
Do not forget to click on the `quit` button when finished to stop the Jupyter server and release the resources. Note that in the last line of the submission script the job waits for your Jupyter service to finish.

If you encounter any issues, have a look in the debug output in `Jupyter_<job id>.out`.
If you encounter any issues, have a look in the debug output in `Jupyter_<job id>.err`. Generic information about the setup of your system is printed in `Jupyter_<job id>.out`.

??? example "Typical content of ``Jupyter_<job id>.out`"
??? example "Typical content of `Jupyter_<job id>.err`"
```shell
> cat Jupyter_3776876.out
[I 2024-10-16 15:13:14.150 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-10-16 15:13:14.154 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-10-16 15:13:14.158 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-10-16 15:13:14.346 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-10-16 15:13:14.409 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-10-16 15:13:14.412 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-10-16 15:13:14.413 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab extension loaded from /home/users/gkafanas/environments/jupyter_env/lib/python3.11/site-packages/jupyterlab
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab application directory is /mnt/aiongpfs/users/gkafanas/environments/jupyter_env/share/jupyter/lab
[I 2024-10-16 15:13:14.415 LabApp] Extension Manager is 'pypi'.
[I 2024-10-16 15:13:14.425 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-10-16 15:13:14.426 ServerApp] Serving notebooks from local directory: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
[I 2024-10-16 15:13:14.426 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-10-16 15:13:14.426 ServerApp] http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-10-16 15:13:14.429 ServerApp]
> cat Jupyter_3664038.err
[I 2024-11-13 23:19:52.538 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-11-13 23:19:52.543 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-11-13 23:19:52.547 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-11-13 23:19:52.766 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-11-13 23:19:52.808 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-11-13 23:19:52.812 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-11-13 23:19:52.813 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-11-13 23:19:52.814 LabApp] JupyterLab extension loaded from /home/users/gkafanas/environments/jupyter_env/lib/python3.11/site-packages/jupyterlab
[I 2024-11-13 23:19:52.814 LabApp] JupyterLab application directory is /mnt/aiongpfs/users/gkafanas/environments/jupyter_env/share/jupyter/lab
[I 2024-11-13 23:19:52.815 LabApp] Extension Manager is 'pypi'.
[I 2024-11-13 23:19:52.826 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-11-13 23:19:52.827 ServerApp] Serving notebooks from local directory: /mnt/aiongpfs/users/gkafanas/support/jupyter
[I 2024-11-13 23:19:52.827 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-11-13 23:19:52.827 ServerApp] http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.827 ServerApp] http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.827 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-11-13 23:19:52.830 ServerApp]

To access the server, open this file in a browser:
file:///mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime/jpserver-3804563-open.html
file:///home/users/gkafanas/.local/share/jupyter/runtime/jpserver-2253096-open.html
Or copy and paste one of these URLs:
http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.441 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-10-16 15:13:14.518 ServerApp] 302 GET / (@172.21.12.29) 0.51ms
http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.845 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-11-13 23:19:53.824 ServerApp] 302 GET / (@127.0.0.1) 0.47ms
```

??? example "Typical content of `Jupyter_<job id>.err`"
```shell
> cat Jupyter_3664038.out

===

AVAILABLE LABS

Currently running servers:
http://172.21.12.29:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2 :: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
http://127.0.0.1:8888/?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d :: /mnt/aiongpfs/users/gkafanas/support/jupyter

===

Expand All @@ -241,22 +245,19 @@ If you encounter any issues, have a look in the debug output in `Jupyter_<job id
/etc/jupyter
data:
/home/users/gkafanas/environments/jupyter_env/share/jupyter
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter
/home/users/gkafanas/.local/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime
/home/users/gkafanas/.local/share/jupyter/runtime

===

KERNEL SPECIFICATIONS

Available kernels:
other_python_env /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/other_python_env
python3 /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/python3
slurmstepd: error: *** JOB 3776876 ON aion-0113 CANCELLED AT 2024-10-16T15:25:20 ***
[C 2024-10-16 15:25:20.236 ServerApp] received signal 15, stopping
[I 2024-10-16 15:25:20.237 ServerApp] Shutting down 4 extensions
python3 /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/python3
```

### Password protected access
Expand Down
Loading