Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building container image failes due to missing files #51

Closed
serverhorror opened this issue Nov 25, 2020 · 8 comments · May be fixed by #54
Closed

Building container image failes due to missing files #51

serverhorror opened this issue Nov 25, 2020 · 8 comments · May be fixed by #54

Comments

@serverhorror
Copy link

There are at least 2 files missing.

I just did a clean checkout of the repository and wanted to build the image (yes, I'm on Windows -- not all of us have a choice here).

PS C:\Users\marcherm\src\single-cell-tutorial> git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
PS C:\Users\marcherm\src\single-cell-tutorial> git rev-parse --short HEAD
8168bcb

It fails with the following message:

> docker build -t single-cell-tutorial .
[+] Building 9.8s (40/41)
 => [internal] load .dockerignore                                                                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                  0.3s
 => => transferring dockerfile: 3.23kB                                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/debian:10                                                                                                                                                          9.2s
 => [internal] load build context                                                                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                                                                       0.0s
 => CANCELED [1/37] FROM docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                          0.1s
 => => resolve docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                                    0.0s
 => CACHED [2/37] RUN apt-get update && apt-get install -y --no-install-recommends build-essential apt-utils ca-certificates zlib1g-dev gfortran locales libxml2-dev libcurl4-openssl-dev libssl-dev libzmq3-dev lib  0.0s
 => CACHED [3/37] RUN apt-get update && apt-get install -y --no-install-recommends wget curl htop less nano vim emacs git                                                                                             0.0s
 => CACHED [4/37] RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen     && locale-gen en_US.utf8     && /usr/sbin/update-locale LANG=en_US.UTF-8                                                                        0.0s
 => CACHED [5/37] WORKDIR /opt/R                                                                                                                                                                                      0.0s
 => CACHED [6/37] RUN wget https://cran.rstudio.com/src/base/R-3/R-3.6.3.tar.gz                                                                                                                                       0.0s
 => CACHED [7/37] RUN tar xvfz R-3.6.3.tar.gz && rm R-3.6.3.tar.gz                                                                                                                                                    0.0s
 => CACHED [8/37] WORKDIR /opt/R/R-3.6.3                                                                                                                                                                              0.0s
 => CACHED [9/37] RUN ./configure --enable-R-shlib --with-cairo --with-libpng --prefix=/opt/R/                                                                                                                        0.0s
 => CACHED [10/37] RUN make && make install                                                                                                                                                                           0.0s
 => CACHED [11/37] WORKDIR /opt/R                                                                                                                                                                                     0.0s
 => CACHED [12/37] RUN rm -rf /opt/R/R-3.6.3                                                                                                                                                                          0.0s
 => CACHED [13/37] RUN Rscript -e "update.packages(ask=FALSE, repos='https://ftp.gwdg.de/pub/misc/cran/')"                                                                                                            0.0s
 => CACHED [14/37] RUN Rscript -e "install.packages(c('devtools', 'gam', 'RColorBrewer', 'BiocManager', 'IRkernel'), repos='https://ftp.gwdg.de/pub/misc/cran/')"                                                     0.0s
 => CACHED [15/37] RUN Rscript -e "devtools::install_github(c('patzaw/neo2R', 'patzaw/BED'))"                                                                                                                         0.0s
 => CACHED [16/37] RUN Rscript -e "BiocManager::install(c('scran','MAST','monocle','ComplexHeatmap','limma','slingshot','clusterExperiment','DropletUtils'), version = '3.10')"                                       0.0s
 => CACHED [17/37] RUN Rscript -e 'writeLines(capture.output(sessionInfo()), "../package_versions_r.txt")' --default-packages=scran,RColorBrewer,slingshot,monocle,gam,clusterExperiment,ggplot2,plyr,MAST,DropletUt  0.0s
 => CACHED [18/37] WORKDIR /opt/python                                                                                                                                                                                0.0s
 => CACHED [19/37] RUN wget https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tgz                                                                                                                                  0.0s
 => CACHED [20/37] RUN tar zxfv Python-3.7.7.tgz && rm Python-3.7.7.tgz                                                                                                                                               0.0s
 => CACHED [21/37] WORKDIR /opt/python/Python-3.7.7                                                                                                                                                                   0.0s
 => CACHED [22/37] RUN ./configure --enable-optimizations --with-lto --prefix=/opt/python/                                                                                                                            0.0s
 => CACHED [23/37] RUN make && make install                                                                                                                                                                           0.0s
 => CACHED [24/37] WORKDIR /opt/python                                                                                                                                                                                0.0s
 => CACHED [25/37] RUN rm -rf /opt/python/Python-3.7.7                                                                                                                                                                0.0s
 => CACHED [26/37] RUN ln -s /opt/python/bin/python3 /opt/python/bin/python                                                                                                                                           0.0s
 => CACHED [27/37] RUN ln -s /opt/python/bin/pip3 /opt/python/bin/pip                                                                                                                                                 0.0s
 => CACHED [28/37] RUN pip install --no-cache-dir -U pip wheel setuptools cmake                                                                                                                                       0.0s
 => CACHED [29/37] RUN pip install --no-cache-dir -U scanpy==1.4.6 python-igraph==0.8.0 louvain==0.6.1 jupyterlab=2.1.0 cellxgene==0.15.0 rpy2==3.2.7 anndata2ri==1.0.2 leidenalg==0.7.0 fa2==0.3.5 MulticoreTSNE==0  0.0s
 => CACHED [30/37] RUN pip install --no-cache-dir git+https://github.com/le-ander/epiScanpy.git                                                                                                                       0.0s
 => CACHED [31/37] RUN jupyter contrib nbextension install --system                                                                                                                                                   0.0s
 => CACHED [32/37] RUN jupyter nbextension enable --py widgetsnbextension                                                                                                                                             0.0s
 => CACHED [33/37] RUN pip freeze > ../package_versions_py.txt                                                                                                                                                        0.0s
 => CACHED [34/37] RUN Rscript -e "IRkernel::installspec(user = FALSE)"                                                                                                                                               0.0s
 => ERROR [35/37] COPY .bashrc_docker /root/.bashrc                                                                                                                                                                   0.0s
 => ERROR [36/37] COPY .profile_docker /root/.profile                                                                                                                                                                 0.0s
------
 > [35/37] COPY .bashrc_docker /root/.bashrc:
------
------
 > [36/37] COPY .profile_docker /root/.profile:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/.profile_docker" not found: not found
@LuckyMD
Copy link
Contributor

LuckyMD commented Nov 25, 2020

Hi @serverhorror,

I don't think you need these .bashrc and .profile_docker files, so you could take the lines out of the Dockerfile. We put the Dockerfile in the repo in the first place so people could build upon this themselves. It might be easiest to just use the image uploaded the docker hub here.

Otherwise we could always make a docker/ folder and add those files as well @le-ander. What do you think?

@le-ander
Copy link
Member

Hm, good point. Thanks for raising this.

The content of the two files is actually quite basic, however it might help to just add them to this repo as well for completeness.

For now, you can just create them yourself:
.bashrc_docker

# Manually added .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

export USER=root
export LOGNAME=root
export HOME=/root
export PATH=/root/.local/bin:/root/bin:/opt/python/bin:/opt/R/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LD_LIBRARY_PATH=/opt/R/lib/R/lib
export SHELL=/bin/bash
export TERM=xterm-256color
export LANG=en_US.UTF-8

alias ll='ls -lha --color=auto'
alias jn='jupyter-notebook --no-browser --ip=0.0.0.0 --allow-root /root/host_home'
alias jl='jupyter-lab --no-browser --ip=0.0.0.0 --allow-root /root/host_home'
alias cx='cellxgene launch --host 0.0.0.0 --port 8888'

.profile_docker

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login exists.

# if running bash, source ~/.bashrc
if [ -n '$BASH_VERSION' ]; then
    # include .bashrc if it exists
    if [ -f '/root/.bashrc' ]; then
        . '/root/.bashrc'
    fi
fi

@LuckyMD
Copy link
Contributor

LuckyMD commented Nov 26, 2020

@le-ander do you want to make a new docker folder in the repo in a PR so you're credited for code contribution, or shall I just create these files and do that quickly?

@le-ander le-ander mentioned this issue Dec 9, 2020
Open
@serverhorror
Copy link
Author

Hi,

I'm just reporting this because it seems pretty unreliable to "pull code from a repo that is broken in the first place". I'm not in the bio* domain at all. Just needed to use it a while ago as a component that would "just work". So I can't comment about the quality of the results.

I'm on the software development end and it seems strange to me having to pull in "untrusted" code that is in a random docker repository on the internet or even something where I (the client) doesn't control availability of image.

@LuckyMD
Copy link
Contributor

LuckyMD commented Dec 21, 2020

I'm not quite sure I understand what you mean. I don't know if you're aware, but the code you are pulling is a workflow in a Jupyter notebook that goes with a tutorial for scRNA-seq analysis. The workflow is extensively documented so that people trying to update their pipelines or learn how to perform the analysis can adapt this to their own needs. None of this is an out-of-the-box pipeline that "just works" on any data and is immediately ready to use.

The provision of the docker image is a work-around for people who don't want to install the environment manually (instructions in the README). And the code for this image is primarily there for adaptation purposes. There are plenty of ways in which you can get around the issue you are facing here (including just quickly creating the files that were missing yourself, or removing the relevant lines in the Dockerfile).

We are working on updating the workflow now and are adding the missing files. However, none of these changes will make the notebooks here any more of a production level tool that "just works".

@serverhorror
Copy link
Author

@LuckyMD I am aware what I'm working with. Let me try and rephrase: Usually when dealing with software projects one of the expectations is that the code is able to build succsessfully from source. There are quite a lot of environments where container images are not a workaround but the norm. Even more so, a requirement. So when a Dockerfile is present then a lot of people -- including me -- expect the source to build.

As I said. I come from the engineering side. All I'm trying to do is report the findings that I see that make it irreproducible. At least they make it hard to reproduce.

@LuckyMD
Copy link
Contributor

LuckyMD commented Dec 23, 2020

Yes, thanks for reporting this. We are working to not only make this docker image build from source in #54 but also to update the jupyter notebook so it runs with the newest versions of several dependencies.

@serverhorror
Copy link
Author

Cleaning up old issues on my profile

@serverhorror serverhorror closed this as not planned Won't fix, can't repro, duplicate, stale Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add missing files for docker container building le-ander/single-cell-tutorial
3 participants
@serverhorror @LuckyMD @le-ander