Skip to content

ubuntu 9.10

Jump to bottom Edit New page
Justin Riley edited this page Jul 11, 2011 · 3 revisions

StarCluster AMI Cookbook Ubuntu 9.10

Below are pseudo-instructions for creating the latest StarCluster AMI (Ubuntu 9.10 Karmic) from scratch:

Launch AMI and Install Packages

  1. Launch an alestic ubuntu AMI (x86: ami-bb709dd2, amd64: ami-55739e3c)

  2. Login via ssh as the ubuntu user and switch to root:

    sudo -i

  3. Edit /etc/apt/sources.list and add multiverse to the end of all lines except deb-src lines

  4. Update the package list and upgrade all packages:

    apt-get update
apt-get upgrade

  5. Install the necessary build tools:

    apt-get --reinstall install build-essential python-dev swig gfortran -y
apt-get remove python-nose g77 python-setuptools -y

  6. Install and configure blas/lapack:

    rm -rvf /usr/lib{,64}/atlas/
apt-get remove liblapack"*" -y
apt-get remove libblas"*" -y
apt-get -y --reinstall install libatlas-base-dev libatlas3gf-base

  7. Install and configure gfortran:

    apt-get --reinstall install gfortran-4.2 -y
ln -sf /usr/bin/gfortran{-4.2,}

  8. Remove any existing numpy/scipy installations:

    rm -rvf /usr/local/lib/python2.6/*-packages/numpy*
rm -rvf /usr/local/lib/python2.6/*-packages/scipy*

  9. Install and configure the base dependencies:

    apt-get install python-dev git-core vim mercurial subversion cvs encfs
openmpi-bin libopenmpi-dev nfs-kernel-server python-django
sun-java6-jdk keychain screen tmux zsh ksh csh tcsh python-pip
python-setuptools python-virtualenv python-imaging python-boto unzip
rar unace build-essential gfortran ec2-api-tools ec2-ami-tools
mysql-server mysql-client apache2 libapache2-mod-wsgi sysv-rc-conf pssh
emacs cython irssi python-distutils-extra python-profiler htop
vim-scripts python-multiprocessing python-multiprocessing-dbg
python-ctypes

  1. Install latest versions of ipython and nose:

    pip install ipython
pip install nose

Configure Root Login ==

The alestic AMI's have been configured to disable root logins. Follow the instructions below to undo this behavior:

  1. Edit /etc/ec2-init/ec2-config.cfg and set disable_root=0
  2. Edit /root/.ssh/authorized_keys and remove prefix commands from pubkey entry (anything before the ssh-rsa...)

Configure MOTD and Bash Completion

  1. Customize motd in /etc/update-motd.d/* and /etc/motd.tail
  2. Uncomment bash completion section in /etc/bash.bashrc

Configure NFS

StarCluster relies on /etc/init.d/nfs existing. Run the command below to ensure this exists:

ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

Configure Sun Grid Engine

  1. Launch one of the 32bit or 64bit StarCluster AMI's and copy /opt/sge6-fresh to /opt/sge6-fresh on your new image host.

  2. Unpack python-drmaa (0.2) to /opt, set SGE6_ROOT='/opt/sge6-fresh' in setup.py and run:

    python setup.py install

  3. Unpack python-drmaa (0.4b3) to /opt and run the following command inside the source folder:

    python setup.py install

  4. Link the DRMAA library to /usr/lib64/libdrmaa.so:

    ln -s /opt/sge6-fresh/lib/lx24-{x86|amd64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so.1.0
ln -s /usr/lib{64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so

  5. Test that the following commands complete without error:

    python -c "import DRMAA"
python -c "import drmaa"

Rebuild Ubuntu's OpenMPI packages with Sun Grid Engine Support

  1. Install the build deps for libopenmpi-dev:

    apt-get build-dep libopenmpi-dev

  2. Get the source for the libopenmpi-dev debian package:

    cd /usr/local/src
apt-get source libopenmpi-dev

  3. Change into the libopenmpi-dev package's debian folder:

    cd openmpi-1.3.2/debian

  4. Modify the 'rules' file and add --with-sge to the configure arguments:

    vim rules
(add --with-sge to configure args)

  5. Rebuild the libopenmpi-dev package:

    cd ..
dpkg-buildpackage -rfakeroot -b

  6. Install the newly rebuild package:

    cd ..
dpkg -i *.deb

  7. Verify Sun Grid Engine support:

    ompi_info | grep -i grid
    MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.3.2)

Configure Optimized Atlas/UMFPACK/AMD/FFTW/Scipy/Numpy

The next sections describe installing an optimized scipy/numpy using Atlas/UMFPACK/AMD/FFTW libraries from source. A lot of these instructions were merged from http://scipy.org/Installing_SciPy/Linux

Install lapack/atlas from source

  1. Checkout npinto-toolbox:

    cd /usr/local/src
git clone git://github.com/jtriley/npinto-toolbox
cd npinto-toolbox

  2. Run the ATLAS installer:

    ./install_atlas3.8.3_Ubuntu9.10-{x86,x86_64}-root.bash

  3. Remove /usr/lib/liblapack.so:

    rm /usr/lib/liblapack.so

  4. Update /usr/lib/gfortran.so symlink:

    ln -s /usr/lib/libgfortran.so.3 /usr/lib/gfortran.so

Install UMFPACK/AMD/UFconfig from source

  1. Create a directory to unpack the source into:

    mkdir /usr/local/src/umfpack

  2. Get the latest versions of AMD, UFconfig and UMFPACK and untar them into /usr/local/src/umfpack

  3. Modify /usr/local/src/umfpack/UFconfig/UFconfig.mk to look like (change -m64 to -m32 in the flags below for a 32bit system):

    CC = gcc
CFLAGS = -O3 -fexceptions -m64 -fPIC -DNCHOLMOD
F77 = gfortran
F77FLAGS = -O -m64 -fPIC

BLAS = -lptf77blas -lptcblas -latlas -lgfortran
LAPACK = -llapack -lptf77blas -lptcblas -latlas -lgfortran

  4. Run 'make' in the /usr/local/src/umfpack/UMFPACK directory:

    cd /usr/local/src/umfpack/UMFPACK
make

  5. Copy resulting libraries and include files:

    cp /usr/local/src/umfpack/AMD/Lib/libamd.a /usr/lib
cp /usr/local/src/umfpack/UMFPACK/Lib/libumfpack.a /usr/lib
cp /usr/local/src/umfpack/AMD/Include/amd.h /usr/include
cp /usr/local/src/umfpack/UFconfig/UFconfig.h /usr/include
cp /usr/local/src/umfpack/UMFPACK/Include/*.h /usr/include

  6. Create /usr/lib/libumfpack.so symlink:

    ln -s /usr/lib/libumfpack.a /usr/lib/libumfpack.so

Install FFTW from source

  1. Download fftw-3.2.2 from http://www.fftw.org into /usr/local/src and unpack

  2. Change into the fftw source folder:

    cd /usr/local/src/fftw-3.2.2

  3. Configure fftw (change -m64 to -m32 for 32bit):

    export CFLAGS="-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -pthread -fPIC -m64"
export FFLAGS="-g -O2 -fPIC -m64"
export CXXFLAGS="-g -O2 -fPIC -m64"
./configure --enable-sse2 --enable-threads --with-combined-threads --prefix=/usr

  4. Build fftw:

    make install

Install Scipy/Numpy from source

  1. Download numpy-1.5.0b1/scipy-0.8.0 into /usr/local/src and untar them

  2. Copy numpy_scipy_site.cfg from npinto-toolbox:

    cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/numpy-1.5.0b1/site.cfg
cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/scipy-0.8.0/site.cfg

  3. Run the configure check for numpy:

    cd /usr/local/src/numpy-1.5.0b1
python setup.py config

  4. Copy the SMCFLAGS from the ATLAS VERSION output and export CFLAGS to be the same as SMCFLAGS

  5. Copy the F77FLAGS from the ATLAS VERSION output and export F77FLAGS to be the same

  6. Install numpy:

    python setup.py install

  7. Run the following command:

    python -c "import numpy; numpy.show_config()"

  8. The output of the above command should look something like:

    In [3]: import numpy

In [4]: numpy.show_config()
atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

  9. Install scipy:

    cd /usr/local/src/scipy-0.8.0
python setup.py install

  10. Run the following command:

    python -c "import scipy; scipy.show_config()"

  11. Output of the above command should look something like:

    In [5]: import scipy

In [6]: scipy.show_config()
amd_info:
    libraries = ['amd']
    library_dirs = ['/usr/lib']
    define_macros = [('SCIPY_AMD_H', None)]
    swig_opts = ['-I/usr/include']
    include_dirs = ['/usr/include']

umfpack_info:
    libraries = ['umfpack', 'gfortran', 'amd']
    library_dirs = ['/usr/lib']
    define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]
    swig_opts = ['-I/usr/include', '-I/usr/include']
    include_dirs = ['/usr/include']

atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

  12. Make sure the following commands both output "OK":

    python -c "import numpy; numpy.test()"
python -c "import scipy; scipy.test()"
Add a custom footer
Add a custom sidebar
Clone this wiki locally