Skip to content

Commit

Permalink
Merge pull request #91 from gkaf89/master
Browse files Browse the repository at this point in the history 
Fix bugs in the Jupyter notebook launcher
  • Loading branch information
@gkaf89
gkaf89 authored Nov 13, 2024
2 parents bbe9a7b + 96f5ae8 commit 8c5f36a
Show file tree
Hide file tree
Showing 2 changed files with 59 additions and 56 deletions.
14 changes: 8 additions & 6 deletions docs/connect/ssh.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -522,31 +522,33 @@ The ssh agent is [not configured in the login nodes](#on-ulhpc-clusters) for sec

To avoid exposing your keys at your personal machine, create and share a new key. Create a key in your local machine,
```bash
ssh-keygen -a 127 -t ed25519 -f ~/.ssh/hpc_id_ed25519
ssh-keygen -a 127 -t ed25519 -f ~/.ssh/ulhpc_id_ed25519
```
and then copy both the private and public keys in your HPC account,
```bash
scp ~/.ssh/hpc_id_ed25519* aion-cluster:~/.ssh/
scp ~/.ssh/ulhpc_id_ed25519* aion-cluster:~/.ssh/
```
where the command assumes that you have setup your [SSH configuration file](#ssh-configuration). Finally, add the key to the list of authorized keys:
```bash
ssh-copy-id -i ~/.ssh/hpc_id_ed25519 aion-cluster
ssh-copy-id -i ~/.ssh/ulhpc_id_ed25519 aion-cluster
```
Then you can connect without a password to any compute node at which you have a job running with the command:
```bash
ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-aion.uni.lu:8022 ${USER}@<node address>
ssh -i ~/.ssh/ulhpc_id_ed25519 -J ${USER}@access-aion.uni.lu:8022 ${USER}@<node address>
```

In the `<node address>` option you can use the node IP address or the node name.

#### Port forwarding over SSH jumps

You can combine the jump command with other options, such as [port forwarding](#ssh-port-forwarding), for instance to access from you local machine a web server running in a compute node. Assume for instance you have a server running in `iris-014` and listens at port `2222`, and that you would like to forward the port `2222` to the `2222` port of you local machine. The, call the port forwarding command with a jump though the login node:
You can combine the jump command with other options, such as [port forwarding](#ssh-port-forwarding), for instance to access from you local machine a web server running in a compute node. Assume for instance that you have a server running in `iris-014` and listens at the IP `127.0.0.1` and port `2222`, and that you would like to forward the remote port `2222` to the `1111` port of you local machine. The, call the port forwarding command with a jump though the login node:

```bash
ssh -J iris-cluster -L 1111:iris-014:2222 <cluster username>@iris-014
ssh -J iris-cluster -L 1111:127.0.0.1:2222 <cluster username>@iris-014
```

This command can be combined with [passwordless access](#passwordless-ssh-jumps) to the cluster node.

## Extras Tools around SSH

* [Assh](https://github.com/moul/advanced-ssh-config) - Advanced SSH config is a transparent wrapper that make `~/.ssh/config` easier to manage
Expand Down
101 changes: 51 additions & 50 deletions docs/services/jupyter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ $ python -m pip install jupyterlab
```

!!! warning
Modules are not allowed on the access servers. To test interactively Singularity, remember to ask for an [interactive job](../jobs/interactive.md) first using for instance the `si` tool.
Modules are not allowed on the access servers. To test interactively Singularity, remember to ask for an [interactive job](/jobs/interactive) first using for instance the `si` tool.

Once JupyterLab is installed along with , you can start to configure your installation setting the environment variables corresponding to your needs:

Expand Down Expand Up @@ -50,7 +50,7 @@ JupyterLab makes sure that a default [IPython kernel](https://ipython.readthedoc

You can [setup kernels with different environments on the same notebook](https://ipython.readthedocs.io/en/stable/install/kernel_install.html). Create the environment with the Python version and the packages you require, and then register the kernel in any environment with Jupyter (lab or classic notebook) installed. For instance, if we have installed Jupyter in `~/environments/jupyter_env`:
```shell
source ~/environments/other_python_venv/bins/activate
source ~/environments/other_python_venv/bin/activate
python -m pip install ipykernel
python -m ipykernel install --prefix=${HOME}/environments/jupyter_env --name other_python_env --display-name "Other Python env"
deactivate
Expand All @@ -70,7 +70,7 @@ will make your conda environment, `conda_env`, available in the kernel launched

## Starting a Jupyter Notebook

Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The following script is a template for Jupyter submission scripts that will rarely need modifications. Most often you will need to modify the session duration (`--time` SBATCH option).
Jupyter notebooks must be started as [slurm jobs](/jobs/submit). The following script is a template for Jupyter submission scripts that will rarely need modifications. Most often you will need to modify the session duration (`--time` SBATCH option).

!!! example "Slurm Launcher script for Jupyter Notebook"
```slurm
Expand All @@ -92,20 +92,21 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
module load lang/Python
source "${HOME}/environments/jupyter_env/bin/activate"

declare loopback_device="127.0.0.1"
declare port="8888"
declare connection_instructions="connection_instructions.log"

jupyter lab --ip=$(hostname -i) --port=${port} --no-browser &
jupyter lab --ip=${loopback_device} --port=${port} --no-browser &
declare lab_pid=$!

# Add connection instruction
echo "# Connection instructions" > "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "ssh -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:${loopback_device}:${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:$(hostname -i):${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "To access the jupyter notebook if you have setup a special key (e.g ulhpc_id_ed25519) to connect to cluster nodes execute on your personal machine:" >> "${connection_instructions}"
echo "ssh -i ~/.ssh/hpc_id_ed25519 -J ${USER}@access-${ULHPC_CLUSTER}.uni.lu:8022 -L ${port}:${loopback_device}:${port} ${USER}@$(hostname -i)" >> "${connection_instructions}"
echo "" >> "${connection_instructions}"
echo "Then navigate to:" >> "${connection_instructions}"

Expand All @@ -118,7 +119,7 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
--retry-delay 1 \
--retry-connrefused \
--silent --show-error --fail \
"http://$(hostname -i):${port}" > /dev/null
"http://${loopback_device}:${port}" > /dev/null
# Note down the URL
jupyter lab list 2>&1 \
| grep -E '\?token=' \
Expand All @@ -127,7 +128,6 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
>> "${connection_instructions}"

# Save some debug information

echo -e '\n===\n'

echo "AVAILABLE LABS"
Expand All @@ -147,14 +147,13 @@ Jupyter notebooks must be started as [slurm jobs](../jobs/submit.md). The follow
jupyter kernelspec list

# Wait for the user to terminate the lab

wait ${lab_pid}
```

Once your job is running (see [Joining/monitoring running jobs](/jobs/submit.md#joiningmonitoring-running-jobs)), you can combine
Once your job is running (see [Joining/monitoring running jobs](/jobs/submit#joiningmonitoring-running-jobs)), you can combine

- [`ssh` forwarding](/connect/ssh.md#ssh-port-forwarding), and
- an [`ssh` jump](/connect/ssh.md#port-forwarding-over-ssh-jumps) through the login node,
- [`ssh` forwarding](/connect/ssh#ssh-port-forwarding), and
- an [`ssh` jump](/connect/ssh#port-forwarding-over-ssh-jumps) through the login node,

to connect to the notebook from your laptop. Open a terminal on your laptop and copy-paste the ssh command contained in the file `connection_instructions.log`, and then navigate to the webpage link provided.

Expand All @@ -164,23 +163,23 @@ to connect to the notebook from your laptop. Open a terminal on your laptop and
# Connection instructions

To access the jupyter notebook execute on your personal machine:
ssh -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
ssh -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]

To access the jupyter notebook if you have setup a key to connect to cluster nodes execute on your personal machine:
ssh -i ~/.ssh/hpc_id_ed25519 -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
To access the jupyter notebook if you have setup a special key (e.g ulhpc_id_ed25519) to connect to cluster nodes execute on your personal machine:
ssh -i ~/.ssh/ulhpc_id_ed25519 -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]

Then navigate to:
http://127.0.0.1:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
```

As the instructions suggest, you access the jupyter lab server in the compute node by calling
```shell
ssh -J [email protected]:8022 -L 8888:172.21.12.29:8888 [email protected]
ssh -J [email protected]:8022 -L 8888:127.0.0.1:8888 [email protected]
```
an SSH command that

- opens a connection to your allocated cluster node jumping through the login node (`-J [email protected]:8022 [email protected]`), and
- exports the port to the jupyter server in the local machine (`-L 8888:172.21.12.29:888`).
- exports the port to the jupyter server in the local machine (`-L 8888:127.0.0.1:8888`).

Then, open the connection to the browser in your local machine by following the given link:
```
Expand All @@ -192,43 +191,48 @@ The link provides the access token, so you should be able to login without a pas
!!! warning
Do not forget to click on the `quit` button when finished to stop the Jupyter server and release the resources. Note that in the last line of the submission script the job waits for your Jupyter service to finish.

If you encounter any issues, have a look in the debug output in `Jupyter_<job id>.out`.
If you encounter any issues, have a look in the debug output in `Jupyter_<job id>.err`. Generic information about the setup of your system is printed in `Jupyter_<job id>.out`.

??? example "Typical content of ``Jupyter_<job id>.out`"
??? example "Typical content of `Jupyter_<job id>.err`"
```shell
> cat Jupyter_3776876.out
[I 2024-10-16 15:13:14.150 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-10-16 15:13:14.154 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-10-16 15:13:14.158 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-10-16 15:13:14.346 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-10-16 15:13:14.409 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-10-16 15:13:14.412 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-10-16 15:13:14.413 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab extension loaded from /home/users/gkafanas/environments/jupyter_env/lib/python3.11/site-packages/jupyterlab
[I 2024-10-16 15:13:14.414 LabApp] JupyterLab application directory is /mnt/aiongpfs/users/gkafanas/environments/jupyter_env/share/jupyter/lab
[I 2024-10-16 15:13:14.415 LabApp] Extension Manager is 'pypi'.
[I 2024-10-16 15:13:14.425 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-10-16 15:13:14.426 ServerApp] Serving notebooks from local directory: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
[I 2024-10-16 15:13:14.426 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-10-16 15:13:14.426 ServerApp] http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.426 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-10-16 15:13:14.429 ServerApp]
> cat Jupyter_3664038.err
[I 2024-11-13 23:19:52.538 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-11-13 23:19:52.543 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-11-13 23:19:52.547 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-11-13 23:19:52.766 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-11-13 23:19:52.808 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-11-13 23:19:52.812 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-11-13 23:19:52.813 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-11-13 23:19:52.814 LabApp] JupyterLab extension loaded from /home/users/gkafanas/environments/jupyter_env/lib/python3.11/site-packages/jupyterlab
[I 2024-11-13 23:19:52.814 LabApp] JupyterLab application directory is /mnt/aiongpfs/users/gkafanas/environments/jupyter_env/share/jupyter/lab
[I 2024-11-13 23:19:52.815 LabApp] Extension Manager is 'pypi'.
[I 2024-11-13 23:19:52.826 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-11-13 23:19:52.827 ServerApp] Serving notebooks from local directory: /mnt/aiongpfs/users/gkafanas/support/jupyter
[I 2024-11-13 23:19:52.827 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-11-13 23:19:52.827 ServerApp] http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.827 ServerApp] http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.827 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-11-13 23:19:52.830 ServerApp]

To access the server, open this file in a browser:
file:///mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime/jpserver-3804563-open.html
file:///home/users/gkafanas/.local/share/jupyter/runtime/jpserver-2253096-open.html
Or copy and paste one of these URLs:
http://172.21.12.29:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
http://127.0.0.1:8888/lab?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2
[I 2024-10-16 15:13:14.441 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-10-16 15:13:14.518 ServerApp] 302 GET / (@172.21.12.29) 0.51ms
http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
http://127.0.0.1:8888/lab?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d
[I 2024-11-13 23:19:52.845 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-11-13 23:19:53.824 ServerApp] 302 GET / (@127.0.0.1) 0.47ms
```

??? example "Typical content of `Jupyter_<job id>.err`"
```shell
> cat Jupyter_3664038.out

===

AVAILABLE LABS

Currently running servers:
http://172.21.12.29:8888/?token=b7cf9d71d5c89627250e9a73d4f28cb649cd3d9ff662e7e2 :: /mnt/aiongpfs/users/gkafanas/support/RITM0195641
http://127.0.0.1:8888/?token=fe665f90872927f5f84be627f54cf9056908c34b3765e17d :: /mnt/aiongpfs/users/gkafanas/support/jupyter

===

Expand All @@ -241,22 +245,19 @@ If you encounter any issues, have a look in the debug output in `Jupyter_<job id
/etc/jupyter
data:
/home/users/gkafanas/environments/jupyter_env/share/jupyter
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter
/home/users/gkafanas/.local/share/jupyter
/usr/local/share/jupyter
/usr/share/jupyter
runtime:
/mnt/aiongpfs/users/gkafanas/.local/share/jupyter/runtime
/home/users/gkafanas/.local/share/jupyter/runtime

===

KERNEL SPECIFICATIONS

Available kernels:
other_python_env /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/other_python_env
python3 /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/python3
slurmstepd: error: *** JOB 3776876 ON aion-0113 CANCELLED AT 2024-10-16T15:25:20 ***
[C 2024-10-16 15:25:20.236 ServerApp] received signal 15, stopping
[I 2024-10-16 15:25:20.237 ServerApp] Shutting down 4 extensions
python3 /home/users/gkafanas/environments/jupyter_env/share/jupyter/kernels/python3
```

### Password protected access
Expand Down

0 comments on commit 8c5f36a

Please sign in to comment.