This lesson is being piloted (Beta version)

Introduction to Apptainer/Singularity

Introduction

Overview

Teaching: 10 min
Exercises: min
Questions

  • What issues motivated the creation of Apptainer/Singularity?

  • What are the differences between Docker, Apptainer and Singularity?

Objectives

  • Learn the design goals behind Apptainer/Singularity.

Working with containers

Containers are packages of software that encapsulates a system environment. An OS-level virtualization is delivered in a container, and any program running on it will use the contextualization isolated inside the container. They have the advantage that you can build a container on any system, your laptop for example, and then execute it anywhere as far as the platform compatible with the container is available.

Concepts such as reproducibility, preservation, and distribution are important in the HEP community, and the containers provide a solution totally compatible with such concepts:

What about virtual machines?

Virtual Machines (VMs) provide the same isolation and reproducibility. However, they emulate the hardware, or at least the full OS, so they are computationally heavier to run, require bigger files when distributed and are less flexible than containers, that run only what you require to be different. All containers use the same OS Kernel of the host and contain only Libraries and the App that run in User space. See this article for a more detailed comparison. And if you are curious about the Linux Kernel mechanisms that make containers possible you can check this blog pos.

Why Apptainer/Singularity?

Many solutions are available to work with containers, for example Docker, one of the most popular platforms, or the free OSS Podman. However, the enterprise-based container frameworks were motivated to provide micro-services, a solution that fits well in the models of the industry, where system administrators with root privilege start the container engine daemon and install and run applications, each in its own container. This is not so compatible with the workflow in the High-Performance Computing (HPC) and High Throughput Computing (HTC), in which usually complex applications run exhaustively using all the available resources and without any special privilege.

Apptainer/Singularity is a container platform created for the HPC/HTC use case. It allows to build and run containers with just a few steps in most of the cases, and its design presents key advantages for the scientific community:

Apptainer/Singularity

Docker and Apptainer/Singularity

As you will learn in this training module, Apptainer/Singularity can be used with Docker images. If you want to learn more about Docker or Podman, check out our podman/docker training.

Apptainer vs Singularity

In these lessons you see the name Apptainer or Apptainer/Singularity, and the command apptainer. As stated in the move and renaming announcement, “Singularity IS Apptainer”. Currently there are three products derived from the original Singularity project from 2015:

We are following Apptainer, the most adopted variation in the scientific community, so we are using the apptainer command. If you are using SingularityPro or SingularityCE, just replace the command apptainer with singularity and the APPTAINER_ and APPTAINERENV_ variable prefixes with SINGULARITY_ and SINGULARITYENV_. If you have older scripts still using the singularity command and SINGULARITY... variables, they will work also in Apptainer because it is providing the singularity alias and full compatibility with the previous Singularity environment.

Documentation

The official Apptainer documentation is available online. Contains basic and advanced usage of Apptainer/Singularity beyond the scope of this training document. Take a look and read the nice introduction, explaining the motivation behind the creation of Apptainer/Singularity.

Key Points

  • Apptainer/Singularity is a container platform designed by and for scientists.

  • Single-file based container images facilitates the distribution.

  • Secure. User inside the container = user outside.

Containers and Images

Overview

Teaching: 20 min
Exercises: 5 min
Questions

  • How to pull Apptainer images from the libraries?

  • How to run commands inside the containers?

Objectives

  • Learn to search and pull images from the Sylabs Singularity library and Docker Hub.

  • Interact with the containers using the command-line interface.

The Apptainer Command Line Interface

Apptainer provides a command-line interface (CLI) to interact with the containers. You can search, build or run containers in a single line.

You can check the version of the Apptainer or Singularity command you are using with the --version option:

singularity --version
# This works for both Singularity and Apptainer, which installs a link named `singularity` to maintain compatibility.
# In the future you may need to use `apptainer --version`

For this training we recommend Apptainer >= 1.0 or Singularity >= 3.5. Older versions may not have some of the features or behave differently. If you need to install or upgrade Apptainer/Singularity please refer to the Setup section. When asking for support please remember to include the version of Apptainer or Singularity being used, as in the output of the above command.

You can check the available options and subcommands using --help:

apptainer --help

Downloading Images

Container Images are executables that bundle together all necessary components for an application or an environment, like a template for containers. Containers are the runtime instances of images — they are images with a state. CircleCI has a nice explanation of the differences.

Apptainer/Singularity can store, search and retrieve images in registries (searchable catalogs and repositories for images and containers). Images built by other users can be accessible using the CLI, can be pulled down, and become containers at runtime.

Sylabs, the developer of one Singularity flavor, hosts a public image registry, the Singularity Container Library where many user built images are available.

Apptainer, the Linux Foundation flavor, does not point by default to the Sylab registry via the Library API as previous versions did. You can change that running these commands (documented here):

apptainer remote add --no-login SylabsCloud cloud.sycloud.io

INFO:    Remote "SylabsCloud" added.

apptainer remote use SylabsCloud

INFO:    Remote "SylabsCloud" now in use.

apptainer remote list

Cloud Services Endpoints
========================

NAME           URI                  ACTIVE  GLOBAL  EXCLUSIVE
DefaultRemote  cloud.apptainer.org  NO      YES     NO
SylabsCloud    cloud.sycloud.io     YES     NO      NO
...

Remote Endpoints, Library API and OCI Registries

Remotes are service endpoints Apptainer interacts with. These include Library API Registries, OCI Registries, and keyservers. The first two are used to search, pull and push images. The Library API, library://, was designed for SIF images, the Singularity Image Format. The Docker/ORAS API, docker://, is used for Docker Hub, and other OCI (Open Containers Initiative) registries. These include Quay.io, NVIDIA NGC, the GitHub Container Registry, AWS ECR and many more.

Once you have setup a working registry you can use search and pull. The command search lists containers of interest and shows information about users (owners or managers of stored containers) and collections (sets of containers). For example:

# this command can take around a minute to complete
apptainer search centos7

No users found for 'centos7'

Found 1 collections for 'centos7'
        library://shahzebmsiddiqui/easybuild-centos7

Found 15 containers for 'centos7'
        library://gmk/default/centos7-devel
                Tags: latest
...

Downloading an image from the Container Library is pretty straightforward:

apptainer pull library://gmk/default/centos7-devel

and the image is stored locally as a .sif file (centos7-devel_latest.sif, in this case).

Docker Images

Fortunately, Apptainer is also compatible with Docker images. There are many more registries with Docker images. Docker Hub is one of the largest libraries available, and any image hosted on the hub can be easily downloaded with the docker:// URL as reference:

apptainer pull docker://centos:centos7

Running Containers

There are several ways to interact with images and start containers. Here we will review how to initialize a shell environment and how to execute directly a command.

Initializing a shell and exiting it

The shell command initializes a new interactive shell inside the container.

apptainer shell centos7-devel_latest.sif

Apptainer>

In this case, the container works as a lightweight virtual machine in which you can execute commands. Remember, inside the container you have the same user and permissions.

Apptainer> id

uid=1001(myuser) gid=1001(myuser) groups=1001(myuser),500(myothergroup)

Now quit the container by typing

Apptainer> exit

or hitting Ctrl + D. Note that when exiting from the Apptainer image all the running processes are killed (stopped). Changes saved into bound directories are preserved. By default anything else in the container is lost (we’ll see later about writable images).

Bound directories

When an outside directory is accessible also inside Apptainer we say it is bound, or bind mounted. The path to access it may differ but anything you do to its content outside is visible inside and vice-versa. By default, Apptainer binds the home of the user, /tmp and $PWD into the container. This means your files at hostname:~/ are accessible inside the container. You can specify additional bind mounts using the --bind option. For example, let’s say /cvmfs is available in the host, and you would like to have access to CVMFS inside the container (here, host refers to the computer/server that you are running apptainer on). Then let’s do

apptainer shell --bind /cvmfs:/mnt centos7-devel_latest.sif

Here, the colon : separates the path to the directory on the host (/cvmfs/) from the mounting point (/mnt/) inside of the container. If you do not have CVMFS, you can try the command with /opt, for example. More information on binding is provided later.

Let’s check that this works:

Apptainer> ls /mnt/cms.cern.ch
bin                        etc                  SITECONF           slc7_aarch64_gcc530
bootstrap.sh               external             slc5_amd64_gcc434  slc7_aarch64_gcc700
...

URLs as input

Each of the different commands to set a container from a local .sif also accepts the URL of the image as input. For example, starting a shell with Rocky Linux 8 is as easy as

apptainer shell docker://rockylinux:8

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 7ecefaa6bd84 done
Copying config a8f7ea56a4 done
Writing manifest to image destination
Storing signatures
2024/02/24 20:32:30  info unpack layer: sha256:7ecefaa6bd84a24f90dbe7872f28a94e88520a07941d553579434034d9dca399
INFO:    Creating SIF file...
Apptainer>

Executing commands

The command exec starts the container from a specified image and executes a command inside it. Let’s use the official Docker image of ROOT to start ROOT inside a container:

apptainer exec docker://rootproject/root root -b

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
2024/02/24 20:36:21  info unpack layer: sha256:7aea3382b6b1676fdc2742fef246a9ec593b44cf8ddc81f0f7b1638f2dda6f65
INFO:    Creating SIF file...
   ------------------------------------------------------------------
  | Welcome to ROOT 6.30/04                        https://root.cern |
  | (c) 1995-2024, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jan 31 2024, 10:01:37                 |
  | From heads/master@tags/v6-30-04                                  |
  | With c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0                   |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

root [0]

And just like that, ROOT can be used in any laptop, large-scale cluster or grid system with Apptainer available.

Execute Python with PyROOT available

Using the official Docker image of ROOT, start a Python session with PyROOT available.

Solution

apptainer exec --cleanenv docker://rootproject/root python3

--cleanenv is optional but makes the command more robust (see Episode 4)

INFO:    Using cached SIF image
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> # Now you can work with PyROOT, creating a histogram for example
>>> h = ROOT.TH1F("myHistogram", "myTitle", 50, -10, 10)

Key Points

  • Use singularity --version to know what you are using and to communicate it if asking for support

  • A container can be started from a local .sif or directly with the URL of the image.

  • Apptainer is also compatible with Docker images, providing access to the large collection of images hosted by Docker Hub.

  • Get a shell inside of your container with apptainer shell <path/URL to image>

  • Execute a command inside of your container with apptainer exec <path/URL> <command>

  • Bind outside directories with --bind

Coffee break!

Overview

Teaching: 0 min
Exercises: 15 min
Questions

  • Get up, stretch out, and dance!

Objectives

  • Refresh your mind!

Key Points

Building Containers

Overview

Teaching: 20 min
Exercises: 10 min
Questions

  • How to build containers with my requirements?

Objectives

  • Download and assemble containers from available images in the repositories.

Running containers from the available public images is not the only option. In many cases, it is required to modify an image or even to create a new one from scratch. For such purposes, Apptainer provides the command build, defined in the documentation as the Swiss army knife of container creation.

The usual workflow is to prepare and test a container in a build environment (like your laptop), either with an interactive session or from a definition file, and then to deploy the container into a production environment for execution (as your institutional cluster). Interactive sessions are great to experiment and test your new container. If you want to distribute the container or use it in production, then we recommend to build it from a definition file, as described in the next episode. This ensures the greatest possibility of reproducibility and transparency.

Apptainer/Singularity usage workflow
'Apptainer/Singularity usage workflow' via Kurtzer GM, Sochat V, Bauer MW (2017) Singularity: Scientific containers for mobility of compute. PLoS ONE 12(5): e0177459. https://doi.org/10.1371/journal.pone.0177459

Build a container in an interactive session

While images contained in the .sif files are more compact and immutable objects, ideal for reproducibility, for building and testing images is more convenient the use of a sandbox, which can be easily modified.

The command build provides a flag --sandbox that will create a writable directory, myAlma9, in your work directory:

apptainer build --sandbox myAlma9 docker://almalinux:9

Notes on shared file systems like AFS

Avoid using the AFS (Andrew File System) and possibly other shared file systems as sandbox directory, as these systems can lead to permission issues. Symptoms could be warnings like “harmless EPERM on setxattr “security.capability” when building the sandbox and IO errors during commands execution, e.g. failure to install RPM packages. In particular, this applies to your home directory on lxplus and to the home and nocache directories on cmslpc. Instead, make sure to use the local file system by creating a folder in /tmp/: mkdir /tmp/$USER. Then, replace myAlma9 in the previous (and next) command with /tmp/$USER/myAlma9.

The container name is myAlma9, and it has been initialized from the official Docker image of AlmaLinux9. To initialize an interactive session use the shell command. And to write files within the sandbox directory use the --writable option. The --cleanenv option is added to make sure that the host environment is not affecting the container. Finally, the installation of new components will require superuser access inside the container, so use also the --fakeroot option, unless you are already root also outside.

apptainer shell --writable --cleanenv --fakeroot myAlma9
Apptainer> whoami

root

--cleanenv clears the environment. It has been added to make sure that the eventual setting of variables on the host is not affecting the container. Variables like PYTHONPATH or PYTHONHOME are affecting the Python execution inside the container. A corrupted Python environment could cause errors like “ModuleNotFoundError: No module named ‘encodings’”

Apptainer environment

Environment variables in Linux are dynamic values that can affect programs and containers. You can use the environment to pass variables to a container. Apptainer by default preserves most of the outside environment inside the container but has many options to control that. You can clear the environment with the --cleanenv option and you can set variables with --env. See the Apptianer manual for more options and details.

PYTHONPATH and PYTHONHOME affect the Python execution, but other variables could affect other programs so, if you don’t care about the outside environment, you can add --cleanenv all the times you start a container (apptainer shell, exec and instance start commands).

Depending on the Apptainer/Singularity installation (privileged or unprivileged) and the version, you may have some requirements, like the fakeroot utility or newuidmap and newgidmap. If you get an error when using --fakeroot have a look at the fakeroot documentation.

--fakeroot is not root

ATTENTION! --fakeroot allows you to be root inside a container that you own but is not changing who you are outside. All the outside actions and the writing on bound files and directories will happen as your outside user, even if in the container is done by root.

As an example, let’s create a container with Pythia8 available using the myAlma9 sandbox. First, we need to enable the CRB (Code Ready Builder/PowerTools) and EPEL (Extra Packages for Enterprise Linux) repositories and to install the development tools (remember that in this interactive session we are superuser):

Apptainer> dnf install 'dnf-command(config-manager)'  # this installs dnf-plugins-core
Apptainer> dnf config-manager --set-enabled crb
Apptainer> dnf install epel-release
Apptainer> dnf groupinstall 'Development Tools'
Apptainer> dnf install python3-devel

Where dnf is the package manager used in RHEL distributions (like AlmaLinux).

Now you can follow all the installation steps described in the Pythia website. Here is a summary of the commands you will need (you may need to adjust link and commands if there is a new Pythia version):

Apptainer> mkdir /opt/pythia && cd /opt/pythia
Apptainer> curl -o pythia8310.tgz https://pythia.org/download/pythia83/pythia8310.tgz
Apptainer> tar xvfz pythia8310.tgz
Apptainer> ln -s pythia8310 pythia8

To enable the Python interface, we add the flag --with-python-include during the configuration, pointing to the location of the Python headers:

Apptainer> cd pythia8310
Apptainer> ./configure --with-python-include=/usr/include/python3.9
Apptainer> make

Apptainer> exit

Installing the Pythia binary package

Pythia is now distributed as RPM in EPEL, so you can use this easier alternative to install it:

Apptainer> dnf install pythia8
Apptainer> dnf install python3-pythia8
Apptainer> dnf install pythia8-devel  # optional, if you need also the development libraries to compile

Now, open an interactive session with your user (no --fakeroot). You can use now the container with Pythia8 ready in a few steps. Let’s use the Python interface:

apptainer shell myAlma9

# These first 2 lines are necessary only if Pythia was installed from source, not if you used the binary package
Apptainer> export PYTHONPATH=/opt/pythia/pythia8/lib:$PYTHONPATH
Apptainer> export LD_LIBRARY_PATH=/opt/pythia/pythia8/lib:$LD_LIBRARY_PATH
Apptainer> python3

>>> import pythia8
>>> pythia = pythia8.Pythia()

 *------------------------------------------------------------------------------------*
 |                                                                                    |
 |  *------------------------------------------------------------------------------*  |
 |  |                                                                              |  |
 |  |                                                                              |  |
 |  |   PPP   Y   Y  TTTTT  H   H  III    A      Welcome to the Lund Monte Carlo!  |  |
 |  |   P  P   Y Y     T    H   H   I    A A     This is PYTHIA version 8.310      |  |
 |  |   PPP     Y      T    HHHHH   I   AAAAA    Last date of change: 25 Jul 2023  |  |
 |  |   P       Y      T    H   H   I   A   A                                      |  |
 |  |   P       Y      T    H   H  III  A   A    Now is 24 Feb 2024 at 22:04:44    |  |

...
 |  |   Copyright (C) 2022 Torbjorn Sjostrand                                      |  |
 |  |                                                                              |  |
 |  *------------------------------------------------------------------------------*  |
 |                                                                                    |
 *------------------------------------------------------------------------------------*

Notice that when installing from source we need to define the environment variables PYTHONPATH and LD_LIBRARY_PATH in order to use Pythia on Python. We will automate this in the next section.

Execute Python with PyROOT available

Build a container to use Uproot, a library for reading and writing ROOT files in pure Python and NumPy, in Python 3.9.

Solution

Start from the Python 3.9 Docker image and create the myPython sandbox:

apptainer build --sandbox myPython docker://python:3.9
apptainer shell myPython

Once inside the container, you can install Uproot.

Apptainer> python3 -m pip install --upgrade pip
Apptainer> python3 -m pip install uproot awkward

Exit the container and use it as you like:

apptainer exec myPython python -c "import uproot; print(uproot.__doc__)"

Uproot: ROOT I/O in pure Python and NumPy.
...

Notice how we did not need neither --writable nor --fakeroot for the installation, but everything worked fine since pip installs user packages in the user $HOME directory. In addition, Apptainer/Singularity by default mounts the user home directory as read+write, even if the container is read-only. This is why a sandbox is great to test and experiment locally, but should not be used for containers that will be shared or deployed. Manual changes and local directories are difficult to reproduce and control. Once you are happy with the content, you should use definition files, described in the next episode.

Key Points

  • The command build is the basic tool for the creation of containers.

  • A sandbox is a writable directory where containers can be built interactively.

  • Superuser permissions are required to build containers if you need to install packages or manipulate the operating system.

  • Use interactive builds only for development and tests, use definition files for production or publicly distributed containers.

Containers from definition files

Overview

Teaching: 20 min
Exercises: 20 min
Questions

  • How to easily build and deploy containers from a single definition file?

Objectives

  • Create a container from a definition file.

As shown in the previous chapter, building containers with an interactive session may take several steps, and it can become as complicated as the required setup. An Apptainer definition file provides an easy way to build and deploy containers.

Hello World Apptainer

The following recipe shows how to build a hello-world container, and run the container on your local computer.

Deleting Apptainer image

To delete the hello-world Apptainer image, simply delete the hello-world.sif file.

apptainer delete

Note that there is also a apptainer delete command, but it is to delete an image from a remote library. To learn more about using remote endpoints and pulling and pushing images from or to libraries, read Remote Endpoints and Library API Registries.

Example of a more elaborated definition file

Let’s look at the structure of the definition file with another example. Let’s prepare a container from an official Ubuntu image, but this time we will install ROOT with RooFit and Python integration.

Following the ROOT instructions to download a pre-compiled binary distribution, the definition file will look like

BootStrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install wget -y
    export DEBIAN_FRONTEND=noninteractive
    apt-get install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev \
        libxft-dev libxext-dev python libssl-dev libgsl0-dev libtiff-dev -y
    cd /opt
    wget https://root.cern/download/root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz
    tar -xzvf root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz

%environment
    export PATH=/opt/root/bin:$PATH
    export LD_LIBRARY_PATH=/opt/root/lib:$LD_LIBRARY_PATH
    export PYTHONPATH=/opt/root/lib

%runscript
    python /opt/root/tutorials/roofit/rf101_basics.py

%labels
    Author HEPTraining
    Version v0.0.1

%help
    Example container running the RooFit tutorial and producing the rf101_basics.png image.
    The container provides ROOT with RooFit and Python integration running on Ubuntu.

Let’s take a look at the definition file:

Save this definition file as rootInUbuntu.def. To build the container, just provide the definition file as argument (executing as superuser):

apptainer build rootInUbuntu.sif rootInUbuntu.def

Then, an interactive shell inside the container can be initialized with apptainer shell, or a command executed with apptainer exec. A third option is execute the actions defined inside %runscript simply by calling the container as an executable

./rootInUbuntu.sif

RooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby
                Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
                All rights reserved, please read http://roofit.sourceforge.net/license.txt
...
 PARAMETER  CORRELATION COEFFICIENTS
       NO.  GLOBAL      1      2
        1  0.02723   1.000  0.027
        2  0.02723   0.027  1.000
[#1] INFO:Minization -- RooMinimizer::optimizeConst: deactivating const optimization
RooRealVar::mean = 1.01746 +/- 0.0300144  L(-10 - 10)
RooRealVar::sigma = 2.9787 +/- 0.0219217  L(0.1 - 10)
Info in <TCanvas::Print>: png file rf101_basics.png has been created

You will find the output file rf101_basics.png in the location where the container was executed. If you don’t have a DISPLAY setup, Root may complain. Ignore the error messages, the image will be created anyway,

Here we have covered the basics with a few examples focused to HEP software. Check the Apptainer docs to see all the available options and more details related to the container creation.

A few best practices for your containers to make them more usable, portable, and secure:

  1. Always install packages, programs, data, and files into operating system locations (e.g. not /home, /tmp , or any other directories that might get commonly binded on).
  2. Document your container. If your runscript doesn’t supply help, write a %help or %apphelp section. A good container tells the user how to interact with it.
  3. If you require any special environment variables to be defined, add them to the %environment and %appenv sections of the build recipe.
  4. Files should always be owned by a system account (UID less than 500).
  5. Ensure that sensitive files like /etc/passwd, /etc/group, and /etc/shadow do not contain secrets.
  6. Build production containers from a definition file instead of a sandbox that has been manually changed. This ensures the greatest possibility of reproducibility and mitigates the “black box” effect.

Deploying your containers

Keep in mind that, while building a container may be time consuming, the execution can be immediate and anywhere your image is available. Once your container is built with the requirements of your analysis, you can deploy it in a large cluster and execute it as far as Apptainer is available on the site.

Libraries like Sylabs Cloud Library ease the distribution of images. Your institution (e.g. Fermilab or CERN) may provide an Harbor registry. GitHub has a Container Registry that Apptainer can access via the ORAS API. Organizations like OSG provide instructions to use available images and distribute custom images via CVMFS.

Be smart, and this will open endless possibilities in your workflow.

Write a definition file to build a container with Pythia8 available in Python

Following the example of the first section in which a container is built with an interactive session (see the previous episode), write a definition file to deploy a container with Pythia8 available.

Take a look at /opt/pythia/pythia8310/examples/main01.py and define the %runscript to execute it using python3.

(Tip: notice that main01.py requires Makefile.inc).

Solution

BootStrap: docker
From: almalinux:9

%post
    yum -y groupinstall 'Development Tools'
    yum -y install python3-devel
    mkdir /opt/pythia && cd /opt/pythia
    curl -o pythia8310.tgz https://pythia.org/download/pythia83/pythia8310.tgz
    tar xvfz pythia8310.tgz
    cd pythia8310
    ./configure --with-python-include=/usr/include/python3.9
    make

%environment
    export PYTHONPATH=/opt/pythia/pythia8310/lib:$PYTHONPATH
    export LD_LIBRARY_PATH=/opt/pythia/pythia8310/lib:$LD_LIBRARY_PATH

%runscript
    cp /opt/pythia/pythia8310/Makefile.inc .
    python3 /opt/pythia/pythia8310/examples/main01.py

%labels
    Author HEPTraining
    Version v0.0.2

%help
    Container providing Pythia 8.310. Execute the container to run an example.
    Open it in a shell to use the Pythia installation with Python 3.9

Build your container executing

apptainer build pythiaInAlma9.sif myPythia8.def

And finally, execute the container to run main01.py

./pythiaInAlma9.sif

This solution is building Pythia from scratch and may take several minutes to build the container. In the previous episode we saw that binary packages of Pythia are available in EPEL. Use them to build a similar container mych faster.

Key Points

  • An Apptainer definition file provides an easy way to build and deploy containers.

Coffee break!

Overview

Teaching: 0 min
Exercises: 15 min
Questions

  • Get up, stretch out, take a short break.

Objectives

  • Refresh your mind.

Key Points

Sharing files between host and container

Overview

Teaching: 30 min
Exercises: 0 min
Questions

  • How to read and write files on the host system from within the container?

Objectives

  • Map directories on your host system to directories within your container.

  • Learn about the bind paths included automatically in all containers.

One of the key features about containers is the isolation of the processes running inside them. It means, files on the host system are not accessible within the container. However, it is very common that some files on the host system are needed inside the container, or you want to write files from the container to some directory in the host.

We have already used the option --bind earlier in the module when exploring the options available to run Apptainer containers. In this chapter we will explore further options to bind directories from your host system to directories within your container.

Remember that in Apptainer, your user outside is the same inside the container (except when using fakeroot). And the same happens with permissions and ownership for files in bind directories.

Bind paths included by default

For each container executed, Apptainer binds automatically some directories by default, and other defined by the system admin in the Apptainer configuration. By default, Apptainer binds:

Let’s use for example the container built during the last chapter called rootInUbuntu.sif. Take a look at your current directory

pwd

/home/myuser/somedirectory

Open a shell inside the container and try to use pwd again

apptainer shell rootInUbuntu.sif

Apptainer> pwd

/home/myuser/somedirectory

you will notice that the files stored on the host are located inside the container! As we explained above, Apptainer mounts automatically your $HOME inside the container.

Disabling system binds

If for any reason you want to execute a container removing the default binds, the command-line option --no-mount is available. For example, to disable the bind of /tmp

run --no-mount tmp my_container.sif

Try this time with

apptainer shell --no-mount home,cwd rootInUbuntu.sif

and you will notice that $HOME is not mounted anymore

ls /home/myuser

ls: cannot access '/home/myuser': No such file or directory

Note how we disabled both home and cwd (current working directory). This because if you are running the apptainer command from your home directory, even if you use --no-mount home the home directory may still be mounted because it is also your current directory.

User-defined bind paths

Apptainer provides mechanisms to specify additional binds when executing a container via command-line or environment variables. Apptainer offers a complex set of mechanism for binds or other mounts. Here we present the main points, refer to the Bind Paths and Mounts documentation for more.

Bind with command-line options

The command-line option --bind (-B) will specify the directories that must be linked between the host and the container. It is available for run, exec, and shell (as well for instance that is not covered yet).

The syntax for using the bind option is "source:destination", and the paths must be absolute (relative paths will be rejected). For example, let’s create a directory in the host containing a constant that can be useful for your analysis

mkdir $HOME/mydata
echo "MUONMASS=105.66 MeV" > $HOME/mydata/muonMass.txt

It is very, very important in your analysis workflow to know the mass of the muon, right? It may have sense to put the data in a high-level directory within the container, like /data

apptainer shell --bind $HOME/mydata:/data rootInUbuntu.sif

This will bind the directory mydata/ from the host as /data inside the container:

ls -l /data

-rw-rw-r-- 1 myuser myuser 20 Jan  2 12:46 muonMass.txt

Now you can use the mass of the muon from a root-level directory!

If multiple directories must be available in the container, you can repeat the option or they can be defined with a comma between each pair of directories, i.e. using the syntax source1:destination1,source2:destination2.

Also. If the destination is not specified, it will be set as equal as the source. For example

apptainer shell --bind /cvmfs rootInUbuntu.sif

Will mount /cvmfs inside the container. Try it!

Binding directories with Docker-like syntax using --mount

The flag –mount provides a method to bind directories using the syntax of Docker. The bind is specified with the format type=bind,src=<source>,dst=<dest>. Currently, only type=bind is supported. Check the documentation for additional options available.

Bind with environment variables

If the environment variable $APPTAINER_BIND is defined, apptainer will bind inside ANY container the directories specified in the format source, with the destination being optional (in the same way as using --bind). For example:

export SINGULARITY_BIND="/cvmfs"

will bind CVMFS to all your Apptainer containers (/cvmfs must be available in the host, of course).

You can also bind multiple directories using commas between each source:destination.

Key Points

  • Bind mounts allow reading and writing files within the container.

  • In Apptainer, you have same owner and permissions for files inside and outside the container.

  • Some paths are mounted by default by Apptainer.

  • Additional directories to bind can be defined using the --bind option or the environment variable $SINGULARITY_BIND.

Apptainer/Singularity instances

Overview

Teaching: 40 min
Exercises: 10 min
Questions

  • How can I keep my container running in the background?

  • What are the use cases for instances?

Objectives

  • Run containers in a detached mode to keep services up.

  • Deploy instances via definition files.

As we have studied in previous chapters, commands such as run and shell allocate Apptainer/Singularity containers in the foreground, stopping any process running inside the container after logout. This behavior suits the use case of containers for executing interactive commands in a well-defined environment, but there are cases when running processes in the background is convenient. For example, when a web application like a Jupyter notebook is deployed inside a container, it is desired to keep the container up while it waits for connections from the web browser.

Apptainer provides the concept of instances to deploy services in the background. While Docker is a common choice of tool for setting services, Apptainer has the advantage of working without requiring any special permissions (like when you are working in a cluster provided by your university/laboratory). In this chapter we will learn the basics about their capabilities and some use cases as examples.

Instances from image files

To start an instance, Apptainer provides the command instance. To exemplify, let’s pull the AlmaLinux image used in previous chapters

apptainer pull docker://almalinux:9

The image must be started in the following way:

apptainer instance start almalinux_9.sif myalma9

In this example, the .sif is the image downloaded from Dockerhub, and mycentos7 is the name that we have assigned to the instance. Instead of opening a shell session or executing a command, the container is running in the background.

Confirm that the instance is running using the instance list command

apptainer instance list

INSTANCE NAME    PID      IP    IMAGE
myalma9          3277300        /tmp/myuser/almalinux_9.sif

To interact with the instance, the commands exec and shell are available. The instance must be referred as instance://name. For example, to open a shell inside the CentOS instance:

apptainer shell instance://myalma9

Remember that exiting the shell instance will not stop the container. For doing so, use instance stop:

apptainer instance stop myalma9

You can confirm the instance doesn’t exist with instance list.

Instances with bind paths

When starting an instance, the same options for bind directories between the host and the container as running an interactive session are available. For example, if you want a directory mounted inside the instance, use the --bind option:

apptainer instance start --bind /home/user/mydata:/data almalinux_9.sif myalma9

binding the directory mydata/ from the host as /data inside the instance.

A web server as an instance

One of the main purposes of the Apptainer instances is deploying services with customized environments. Before moving to more complex use cases, let’s start with a basic example: a web service showing a HTML with a message.

Let’s write a basic index.html file as:

<!DOCTYPE html>
<html>
<head>
<title>My awesome service</title>
</head>
<body>
<h1>Hello world!</h1>
<p>If you see this page, my awesome service is up and running.</p>
</body>
</html>

If you are not familiar with HTML take a quick look at the HTML Tutorial, but it is not mandatory. What really matters is having a minimal webpage that our server will show.

Now, let’s prepare a basic web server using Python http.server. Create a definition file, saved as basicServer.def, which contains:

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install -y python3.9

%files
    index.html /tmp/index.html

%startscript
   cd /tmp
   python3.9 -m http.server 8850

If you recall the chapter about definition files, this definition file will pull the official Ubuntu image from Dockerhub, and will install Python3.9. In addition, it copies index.html in /tmp inside the container. When the instance starts, commands specified on %startscript are executed. On this example, http.server will be executed, serving a page in the port 8850 (you can use any other port if 8850 is busy with another service).

Let’s build an image from the definition. Remember that building images requires either superuser permissions or using the flag --fakeroot as

apptainer build --fakeroot basicServer.sif basicServer.def

Now, let’s start an instance named myWebService with the image that we just built

apptainer instance start --no-mount tmp --cleanenv basicServer.sif myWebService

Reminder from the previous chapter: with --no-mount tmp we are asking Apptainer to NOT bind /tmp from the host to the instance (it is mounted by default), we use instead an isolated /tmp inside the instance where index.html has been copied. And with --cleanenv we clear the environment. It is not always necessary but it prevents interferences from the host environment (see ).

You can confirm in the terminal that the web service is up using curl as

curl http://localhost:8850

<!DOCTYPE html>
<html>
<head>
<title>Welcome to my service!</title>
</head>
<body>
<h1>Hello world!</h1>
<p>If you see this page, my awesome service is up and running.</p>
</body>
</html>

If you are executing Apptainer locally, try to open http://localhost:8850.

SSH tunneling

If you are deploying a service in a cluster of your institution (as LXPLUS at CERN) it is likely that you need SSH tunneling for opening pages served by your service with a web browser. A basic port forwarding can be configured as:

ssh -L <port>:localhost:<port> myuser@<server>

where is the one used by your service, and is the address of your institutional resources. For example, for connecting to LXPLUS forwarding the port 8850:

 ssh -L 8850:localhost:8850 myuser@lxplus.cern.ch

Then you can open http://localhost:8850 in your machine!

Port numbers less than 1024 are privileged ports and can be used only by root. Please consult the allowed ports and rules with your institution.

Remember to stop the instance once you are done.

Serving a Jupyter notebook with custom environment

As an example of the capabilities of instances as services, let’s extend our definition file to deploy a Jupyter notebook server with a customized environment. Jupyter Notebook is a web-based interactive computing platform. The notebook combines live code, equations, narrative text, and visualizations.

What if we provide a Jupyter notebook ready to use ROOT? If you remember our example from the definition files chapter, at this point it must be almost straightforward:

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install -y python3
    apt-get install -y python3-pip
    pip install notebook

    apt-get install wget -y
    export DEBIAN_FRONTEND=noninteractive
    apt-get install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev \
    libxft-dev libxext-dev python libssl-dev libgsl0-dev libtiff-dev -y
    cd /opt
    wget https://root.cern/download/root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz
    tar -xzvf root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz

%environment
    export PATH=/opt/root/bin:$PATH
    export LD_LIBRARY_PATH=/opt/root/lib:$LD_LIBRARY_PATH
    export PYTHONPATH=/opt/root/lib

%startscript
   jupyter notebook --port 8850

Save the definition file as jupyterWithROOT.def, and let’s build an image called jupyterWithROOT.sif

apptainer build --fakeroot jupyterWithROOT.sif jupyterWithROOT.def

Now, start an instance named mynotebook with our brand-new image. Consider using --cleanenv if needed.

apptainer instance start jupyterWithROOT.sif mynotebook

and confirm that the instance is up

apptainer instance list

INSTANCE NAME    PID      IP    IMAGE
mynotebook       10720          /home/myuser/jupyterWithROOT.sif

If you go to http://localhost:8850 (with SSH tunneling if needed), you will find out that for security reasons the Jupyter webapp will ask for an access token. Fortunately, you can get the token listing the URL of active servers using the jupyter notebook list command. To execute the command inside the instance, use sigularity exec:

apptainer exec instance://mynotebook jupyter notebook list

Currently running servers:
http://localhost:8850/?token=12asldc9b2084f9b664b39a6246022312bc9c605b :: /home/myHome

Notebook starting on a different port!

If the chosen port for the Notebook (8850 stated in the SIF file) is not available, the notebook will not error out, but will start and use the first available port after that. E.g. if you did not terminate the web server from the previous example, The above command “jupyter notebook list” will show you the correct port.

Open the URL with the token (from http to the first space), and you will be able to see the Jupyter interface. Try to open a new notebook and write in a cell to confirm that ROOT is available:

import ROOT
# Now you can work with PyROOT, creating a histogram for example
h = ROOT.TH1F("myHistogram", "myTitle", 50, -10, 10)
h.FillRandom("gaus", 10000)

c = ROOT.TCanvas("myCanvasName","The Canvas Title",800,600)
h.Draw()
c.Draw()

The bottom line: with any Jupyter notebook that you write, you can provide an Apptainer image that will set the environment required to execute the cells. It doesn’t matter if yourself or someone else comes in one, five, ten years, your code will work independently of the software available in your computer as long as Apptainer/Singularity is available!

A Jupyter notebook with Uproot available

Can you setup a Jupyter notebook server with Uproot available in Apptainer?

Hint: Uproot can be installed using pip.

Solution

Bootstrap: docker
From: ubuntu:20.04

%post
   apt-get update -y
   apt-get install -y python3
   apt-get install -y python3-pip
   pip install notebook
   pip install uproot

%startscript
  jupyter notebook --port 8850

Confirm that Uproot is available opening a notebook and executing in a cell

import uproot
print(uproot.__doc__)

Key Points

  • Instances allow to setup services via Apptainer images or definition files.

  • Code provided in Jupyter notebooks can be accompanied by a Apptainer/Singularity image with the environment needed for its execution, ensuring the reproducibility of the results.

Bonus Episode: Building and deploying an Apptainer container to GitHub Packages

Overview

Teaching: 40 min
Exercises: 0 min
Questions

  • How to build apptainer container for python packages?

  • How to share apptainer images?

Objectives

  • To be able to build an apptainer container and share it via GitHub packages

Prerequisites

The previous episode ended the Introduction to Apptainer/Singularity. This bonus episode is an optional extension mixing knowledge from different courses. For this lesson, you will also need,

Apptainer Container for python packages

Python packages can be installed using an Apptainer image. The following example illustrates how to write a definition file for building an image containing python packages.

BootStrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install wget -y
    export DEBIAN_FRONTEND=noninteractive
    apt-get install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev \
    libxft-dev libxext-dev python3 libssl-dev libgsl0-dev libtiff-dev \
    python3-pip -y
    pip3 install numpy
    pip3 install awkward
    pip3 install uproot4
    pip3 install particle
    pip3 install hepunits
    pip3 install matplotlib
    pip3 install hist
    pip3 install mplhep
    pip3 install vector
    pip3 install fastjet
    pip3 install iminuit

As we see, several packages are installed.

Publish Apptainer images with GitHub Packages and share them!

It is possible to publish apptainer images with GitHub packages. To do so, one needs to use GitHub CI/CD. A step-by-step guide is presented here.

name: Apptainer Build Deploy

on:
  pull_request:
  push:
    branches: master

jobs:
  build-test-container:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    container:
        image: quay.io/singularity/singularity:v4.1.0
        options: --privileged

    name: Build Container
    steps:

      - name: Check out code for the container builds
        uses: actions/checkout@v4

      - name: Build Container
        run: |
           singularity build container.sif Apptainer

      - name: Login and Deploy Container
        run: |
           echo ${{ secrets.GITHUB_TOKEN }} | singularity remote login -u ${{ secrets.GHCR_USERNAME }} --password-stdin oras://ghcr.io
           singularity push container.sif oras://ghcr.io/${GITHUB_REPOSITORY}:${tag}

The above script is designed to build and publish an Apptainer image with GitHub packages.

Key Points

  • Python packages can be installed in apptainer images along with ubuntu packages.

  • It is possible to publish and share apptainer images over github packages.