Adam Batten

Installing Python with Anaconda

2022-03-18T00:00:00+00:00

Table of Contents

A Guide to Installing Python with Anaconda.

A Guide to Installing Python with Anaconda.

This tutorial is a write up of the installation instructions used for my “Introduction to Python Workshops” that I ran at Swinburne University of Technology. These workshops were targeted at Year 10 work experience students to learn the basics of loading data nd plotting with Python.

Python

Python is a programming language that is very popular and is used by many astronomers to perform their research. Some of the uses of Python include reading telescope data, analysing simulations and making graphs/plots of results. If you were you were to google ‘Install Python’ you would see many different options and versions of Python to install and from dfferent locations. It isn’t particularly clear which one you should use, how to install it and how to use it one you have.

Additionally, if you are using MacOS or Linux your computer likely already has a version of Python already installed to run the operating system. So installing a new version may cause some conficting issues with your computer, particularly if when you want to install more ‘Python packages’ (we will get to this later).

However there is much easier and better way to install Python. This is through using Anaconda.

Downloading and Installing Anaconda

Anaconda is distribution of Python designed for scientific computing. By installing Python through Anaconda, it makes managing and installing Python packages/versions significantly easier

You should be able to follow these instructions to install Anaconda on your system:

Go to https://www.anaconda.com/products/individual and click on ‘Download’. It should take you to the bottom of the page to the different options (see Figure 1 below).
Choose the installer for your operating system. You will probably want to use the graphical installer since it is much easier. Windows users note: You can only use a graphical installer, but you need to check if your system is 64-bit (most likely) vs 32-bit, then choose the appropriate one. You can check which type your computer is by clicking here and following the instructions.
After the download completes, double click on the file to open the installer. It should like Figure 2 on Windows 10 or Figure 7 on MacOS Big Sur.
Then follow either the Windows 10 or MacOS installation instructions that follows.

Figure 1 - Anaconda installation options. Choose the one that is appropriate for your computer.

Windows 10 Installation

Click ‘I Agree’ to the licensing agreement (see Figure 3).
Select to install for ‘Just Me’ then click ‘Next’.
Choose location to install. The default location should be something C:\\Users\<NAME>\anaconda3. This default location is normally recommended, but you can install it elsewhere.
Tick the boxes that say ‘Add Anaconda3 to my PATH environment variable’ and ‘Register Anaconda3 as my default Python 3.8’. See Figure 4. It will likely have a red warning that says the first option is not recommended, because it ‘makes anaconda get found before previously installed software’, but that is actually what we want to happen. It might also say a different number to Python 3.8 (i.e Python 3.9, 3.10, etc..), this is also okay.
Continue to Section Anaconda Navigator.

Figure 2 - Anaconda Installer (Windows 10)

Figure 3 - Anaconda License Agreement (Windows 10)

Figure 4 - Anaconda installation options on Windows 10. You want to tick both boxes.

MacOS Installation

Click ‘Continue’ on Read Me to get to the licensing agreement.
Click ‘Continue’ then ‘I Agree’ to the licensing agreement (see Figure 7).
Click ‘Install’ on installation type. This will install anaconda in the default location, this is usually /Users/<NAME>/opt/anaconda3.
Click ‘Continue’ after the installation is complete.
Continue to Section Anaconda Navigator.

Figure 6 - Anaconda Installer (MacOS)

Figure 7 - Anaconda License Agreement (MacOS)

Anaconda Navigator

To check that Anaconda has been correctly installed we will attempt to open the Anaconda Navigator. You can load this program by doing the folowwing:

Windows: Click the home button and search for ‘anaconda navigator’, then click to open
MacOS: Click ‘Launchpad’ and search ‘anaconda-Navigator’, then click to open.

The program should look something similar to Figures 8 (Windows) and 9 (MacOS). Note that you may have different icons or tiles showing based on what is already installed on your computer. The important thing you should be able to see is something called ‘JupyterLab’ and ‘Jupyter Notebook’. If you can see both of these, then it has installed correctly and you are ready to start using Anaconda and Python.

Figure 8 - Anaconda Navigator in Windows 10

Figure 9 - Anaconda Navigator in MacOS

Setting up Python on OzSTAR

2021-02-02T00:00:00+00:00

Table of Contents

Getting Started with using Conda Environments on OzStar
Preparation
Loading Anaconda
Using Conda environments
Issues with Home Directory Quota
Python 3 vs. Python 2
Acknowledgements

Getting Started with using Conda Environments on OzStar

This is a short guide on how to set up Python on OzSTAR, the supercomputer located at the Swinburne University of Technology. This will guide you through setting up Conda enviroments and some of the pitfalls you may encounter along the way. Keep in mind this isn’t necessarily the ‘perfect’ way to use Python on OzSTAR (since it really depends on your use case), but this tutorial is a good recommendation to get started for most people.

If you are unfamiliar with Conda environments before this guide, they may feel a little different, but will make using Python on OzSTAR significantly easier in the long run.

Note: This is primarily aimed at people that will be using OzSTAR, and may not be applicable in other circumstances.

This Tutorial Assumes You Can:

Log into OzSTAR
Can edit a text file (specifically your .bashrc or .bash_profile)

Some Key Definitions:

Anaconda: A distribution of Python designed for scientific computing. By installing Python through Anaconda, it makes managing and installing Python packages/versions much easier. In particular, Anaconda has a useful feature called ‘conda environments’ which allows us to use different versions of Python or Python packages when we need to.
conda: A package and environment manger that is bundled with Anaconda. It can install packages similar to how pip can install Python packages from PyPI using conda install. It can also create environments using conda create.
.bashrc: This is a Bash shell script that runs whenever you launch an interactive shell. You can put any command(s) in a .bashrc that you would write at the command prompt. Then every time you launch a new interactive shell all the commands in the .bashrc will be executed. It is common practice to customise your .bashrc to include your own preferences (e.g. aliases for specific commands that you type often). You can find your .bashrc in your home directory (~/.bashrc).
PATH: PATH is an environment variable that specifies a collection of directories where programs are located. This list of directories are locations where programs can be started without having to type the entire path (e.g. /User/username/subfolder/otherfolder/program.exe). For example: to run Python you would normally just use the prefix python, without specifying where Python is located. This is because python is already the system PATH. If there are multiple directories in PATH that the required program can use, the one that will be used is the one that was added to the PATH the latest.

Preparation

To get started, make sure that there is absolutely nothing related to Python environments currently found anywhere in your home directory or module lists. This includes loading any of the OzSTAR Python modules (including Anaconda), like Python; NumPy; Matplotlib; etc, or having any conda installation (like Anaconda and Miniconda). Loading modules or libraries that are required by Python packages is fine.

You can ensure that all modules have been unloaded from your session use module perge.

In case you are unfamiliar, OzSTAR uses modules to manage the software it has installed. If you want to use a module you first have to load it. To load a module use the following command: `module load /`. You can search for modules using `module spider `. To view a list of all the modules you have loaded use `module ls`. </mark>

Loading Anaconda

For managing Python on OzSTAR I suggest using anaconda3 and conda environments.

Despite its name, anaconda3 is can be used for both Python 3 and Python 2 (see warning about using Python 2). However Anaconda3 has the tendency to overwrite the PATH to certain libraries that are used for Python packages. Examples of this include the HDF5-library h5py and the MPI-library mpi4py. It is a good idea to use the system modules on OzSTAR (except for the system Python) as they are compiled and optimized for the OzStar architecture. Therefore if you have a list of system modules that need to be loaded, make sure to load anaconda3 first so that the other modules will override the anaconda3 paths as they are loaded. Loading all modules after anaconda3 means that those corresponding libraries are found earlier in the system PATH than anaconda3’s libraries. This ensures that everything is found in the correct place. To do this you can execute the following:

module purge
module load anaconda3/5.0.1
<load other modules>

The reason for using module purge all loaded modules first is to make absolutely sure that no paths are being overridden by anaconda3. I highly recommend using your .bashrc or .bash_profile for this, such that you don’t have to do it manually every time you login.

The reason for using `anaconda3/5.0.1` rather than `anaconda3/5.1.0` (which is also available on OzSTAR), is because the latter version has a bug where OzSTAR will not correctly set the paths to Anaconda's commands (like `conda`; `activate`; `deactivate`). Instead, it attempts to use the ones from `anaconda3/5.0.1` whenever any environment besides the default is [activated](#activating-an-anaconda-environment). This means that it once an environment is [activated](#activating-an-anaconda-environment) using `anaconda3/5.1.0`, it becomes impossible to [deactivate](#deactivate-an-anaconda-environment) the environment again.

Example:

If every time I want to log into OzSTAR I load the modules anaconda3, openmpi, hdf5 and git, to avoid having to type these out every single time I put this code into my .bashrc file. Note that the anaconda3/5.0.1 comes directly after module purge.

module purge
module load anaconda3/5.0.1
module load openmpi/3.0.0
module load hdf5/1.10.1
module load git/2.16.0

Using Conda environments

TL;DR: Create custom environments using `conda create -n python==`, activate with `source activate ` and deactivate with `source deactivate`. </mark>

Why use a Conda Environment?

The main purpose to using conda environments is to create an isolated environment for your project. This means you don’t have to worry about if different projects you are working on require different versions of software.

Say for example that you are writing some code for one of your projects. This project requires Python > 3.5. However, in a different project at the same time you are using someone else’s software that is only compatible with Python 2.7, you really have five choices:

Uninstall and reinstall Python every single time you swap between the different software (tedious and not fun).
Re-write the other software to be Python 3 compatible (also not fun).
Re-write all your software to be compatiblePython 2.7compatible (not fun and will be more limited in features and support).
Give up entirely (really really not fun).
Use different conda environments to swap betweenPython 2.7and Python 3 easily.

Creating a Conda Environment

The first rule about using a conda environment is NEVER use the base/default conda environment (i.e. the one that is activate with source activate with no additional arguments). This environment is available to everyone that uses the same anaconda3 module, and therefore cannot be used in a personal manner. It also does not allow for packages to be installed globally (within the environment), which certain Python packages cannot deal with. This is generally a good idea to avoid as well on personal computers.

So for this reason you will have to first create an new custom environment. A new environment can be created as follows:

conda create -n <environment_name> python==<python_version>

This will create an environment with the name <environment_name> using Python version <python_version>. This environment will be stored in your ~/.conda/envs directory. You can create multiple environments for the different versions of Python required. I personally have a Python 3.8 environment for 99.9% of my work and Python 2.7 environment to run software that hasn’t been update to Python 3 yet.

If you don’t know what to name your environment, I would suggest naming your environment after the Python version. i.e If you are going to be using Python 3.8, name the environment py38, or you are using Python 2.7 name it py27.

Activate a Conda Environment

After the environment has been made, it can be activated with source activate <environment_name> (or . activate <environment_name>, either one is fine). You will be able to see that your environment is activated because the <environment_name> will appear on the left hand side of your prompt as shown below.

[~] conda activate py38
(py38) [~]

After you activate your environment, packages can be installed using pip or conda as normal without having to use the --user flag (which installs a package locally, but as environments are isolated, that is not necessary).

It’s also convenient to include the source activate <environment_name> in your .bashrc or .bash_profile so that your environment is automatically activated when you log into OzSTAR. For example, I have an environment called py38 and my .bashrc contains this:

module purge
module load anaconda3/5.0.1
source activate py38

module load openmpi/3.0.0
module load hdf5/1.10.1
module load git/2.16.0

Deactivate a Conda Environment

To deactivate an active environment use source deactivate ( or . deactivate).

Keep a note of the issue with deactivating environments if you use anaconda3/5.1.0.

Issues with Home Directory Quota

TL;DR: If you are running out of inode space in your home directory, you can move your `.conda` to Lustre with `mv /home//.conda /fred/oz/` and make a symbolic link with `ln -s /fred/oz//.conda /home//.conda` </mark>

The main downside with using this approach to use Python on OzSTAR is that conda environments produce a large number of small files. If you have many environments or large ones, it is possible that you will hit the file limit (aka inode limit) in your home directory (which is set to 100k).

A solution to this is to move the .conda folder to a project on Lustre (/fred/) and making a symbolic link between its location and your home directory.

mv /home/<username>/.conda /fred/oz<group_number>/<username>
ln -s /fred/oz<group_number>/<username>/.conda /home/<username>/.conda

This will allow you to use your environments anywhere as if they were still located in your home directory. However keep in mind that Lustre is designed for long-term, big-file storage and therefore is not nearly as fast with reading as the home directories are. This means that activating an environment or executing a script can take a while to start up (often about 30 seconds to 1 minute). Once started, there are no speed differences as all required scripts and modules have now been loaded. So in effect, the first Python script you run in your session will take about a minute longer to run, but the rest will be exactly the same.

Python 3 vs. Python 2

It is advised that you should be using Python 3 for almost all of your projects. As of the 1st of January 2020, Python 2 is no longer supported. Meaning that packages like NumPy, Matplotlib, SciPy and Astropy (plus more) will no longer update or fix issues with the Python 2 version of the software. If you absolutely need to use Python 2 for a specific project (e.g. legacy code that hasn’t been ported into Python 3 yet), you can set up aPython 2.7environment to run that specific piece of software and a Python 3 environment for everything else.

Acknowledgements

A big thank to Ellert van der Velden (@1313e) who wrote the backbone of this tutorial in a text document one day on Slack and for useful feedback on my interpretation of the document that I have presented in this article.

Submitting a Job on a Supercomputer

2020-09-08T00:00:00+00:00

Table of Contents

A beginners guide to writing and submitting Slurm jobs on OzSTAR.
Writing and Submitting A Job
Querying the State of a Job
Canceling a Job
Examples

A beginners guide to writing and submitting Slurm jobs on OzSTAR.

Many students in the Centre of Astrophysics and Supercomputing at the Swinburne University of Technology have found it challenging to determine exactly what they need to do when they want to turn a program they have written into a job to OzSTAR. This tutorial aims to resolve this issue by giving detailed instructions so by the end, even complete OzSTAR beginners should understand how to write and submit their own jobs.

This tutorial for submitting jobs is primarily aimed at people who will be using the OzSTAR supercomputer located at Swinburne. So keep in mind that some of this tutorial may not be applicable to all systems that are using a Slurm based job scheduler. This tutorial is based on the OzSTAR/Slurm documentation found here.

This Tutorial Covers:

Writing a job submission script.
- The Shebang.
- #SBATCH.
- Loading Modules.
- Job Step.
Submitting a job submission script.
Querying the state of a job.
Canceling a job.
Example Job Submission Scripts.

This Tutorial Assumes You Can:

Log into OzSTAR
Write a shell script

Some Key Definitions:

Slurm: A job scheduler. When you submit a job, you are submitting it to Slurm. Slurm works to organise and schedule when all of the jobs will run after they have been submitted. Slurm determines the optimal way to allocate CPUs, time and memory to jobs based on what has been requested.

Writing and Submitting A Job

Submitting a job on OzSTAR is a two step process:

1) First you have to request resources from OzSTAR. This will normally include the amount of memory you need, the number of CPUs you want to use and how long you expect to be using the resources. This step is called a ‘‘resource request”. An example resource request using english could be:

"I want to use 200 MB of memory on 1 CPU core for 2 hours".

2) The second step is where you say what you want these resources to be doing. This includes specifying what software is needed, which scripts you want to run, and how to execute them. This step is called the ‘job step’. An example job step could be:

"I want to load python and then execute a python script".

Luckily, you can do both of these steps at the same time using a “submission script”.

Writing a Job Submission Script

A submission script is just a shell script that is formatted in a specific way such that it contains both your resource request and your job step. Here is an example of a simple submission script called my_slurm_job.sh. This submission script will run a python script called example_python_job.py.

#!/bin/bash

#SBATCH --ntasks=1
#SBATCH --mem=100MB
#SBATCH --time=00:30:00

module purge
module load anaconda3/5.0.1

source activate py3

python example_python_job.py

There are four main components to a submission script: The Shebang, #SBATCH, Loading Modules and The Job Step.

1 The Shebang

#!/bin/bash

The first component is the very first line: #!/bin/bash. This line has many names including: ‘shebang’, ‘shabang’, ‘hashbang’, ‘pound-bang’, ‘hash-pling’ and probably lots of other ridiculous sounding names. This line serves the purpose of defining which ‘interpreter’ to use when executing the script.

In this case we are effectively saying: “This is a bash (Bourne-Again Shell) script, so execute this script with a bash shell”.

Since all of your submission scripts will be likely be a bash script, this line will be the same for every submission script you write.

2 #SBATCH

#SBATCH --ntasks=1
#SBATCH --mem=100MB
#SBATCH --time=00:30:00

The second component is where you request resources and it includes all the lines that start with #SBATCH. Any line in your submission script that begins with #SBATCH is understood by Slurm to be a resource request or an additional option related to the job.

Notice the # in front. This can be easy to forget but is important.

In this example script we are requesting: 1 CPU (#SBATCH --ntasks=1) and 100 MB of RAM (#SBATCH --mem=100MB) for 30 minutes (#SBATCH --time=00:30:00). Similarly if you wanted request 2 GB of RAM on a single CPU for 12 hours your resource request would look like this:

Example

#SBATCH --ntasks=1
#SBATCH --mem=2GB
#SBATCH --time=12:00:00

There are lots of additional options that you include here, for instance you can include the option to have SLURM send you an email when your job starts and ends.

Example:

#SBATCH --mail-user=name@swin.edu.au
#SBATCH --mail-type=ALL

You can see a complete list of parameters using man sbatch. I have also listed more examples of options that may be useful at the end of this tutorial.

A good rule of thumb is: The more resources you request, the longer it will take for your job to start. This means that asking for way more time and RAM than you actually need is not a good idea. Unfortunately, this tutorial can not tell you how much resources your job will require. You will have have to determine that for yourself. If you are using python, you can look into using packages such as memory_profiler and line_profiler to help estimate the memory usage and timing of a script.

3 Loading Modules

module purge
module load anaconda3/5.0.1

source activate py3

The third component is where you load all the modules that are necessary to run your script. OzSTAR uses modules to manage the software it has installed. If you want to use a module you first have to load it. To load a module use the following command: module load <module_name>/<software_version>. You can search for modules using module spider <module_name>. We will be executing a python script using the example submission script, hence we have to load the modules for anaconda and the Conda environment that all the packages are install in.

module purge unloads all loaded modules.
module load anaconda3/5.0.1 loads Anaconda. I recommend using the anaconda3/5.0.1 version of instead of anaconda3/5.1.0 because there is a significant bug in the later version on OzSTAR related to Conda environments.
source activate py3 loads a Conda environment that I have called py3. In this environment I have installed all of the necessary packages required to execute example_python_job.py.

Here is a list of a few common modules that you might need to load: gcc/7.3.0, hdf5/1.10.1, openmpi/3.0.0.

Loading modules is not necessary if they are already loaded in your session. You can load modules in the .bashrc file in your home directory the same way you would load them in a bash script. If you load modules in your .bashrc those modules will automatically get loaded into your environment every time you log into Ozstar. This means you only have to write load module anaconda3/5.0.1 once in your .bashrc and then forget about it.

If you are using python I highly recommend that you use Conda environments when working on OzSTAR. Anaconda environments will save you lots of time, stress and effort in the long run. I have a tutorial for setting up python on OzSTAR that details how to set up Conda environments.

4 Job Step

python example_python_job.py

This final component is where you say what you want to actually run on the requested resources. This is the job step. In the example, we have a single line python example_python_job.py indicating that our job will run the example_python_job.py script. It is also possible to list multiple job steps in a submission script and they will be performed one after another.

Example:

python example_python_job.py
python example_python_job2.py
python_example_python_job3.py

In this case, after the first job step is completed the second job step (example_python_job2.py) will begin. In your resource request you will have to allow for the time and memory requirements for all of the job steps.

Submitting a Job Submission Script

After you have finished writing a submission script, you can submit it to Slurm with the sbatch command.

>>> sbatch my_slurm_job.sh
sbatch: Submitted batch job 99999999

If the job is successfully submitted, it will respond with the jobid that was assigned to the job. In this example the jobid is 99999999.

You can also submit a job to the queue from within a submission script. This can be useful for automating a pipeline of scripts that need to be completed in a sequence.

You can find more documentaion related to the sbatch command on the official Slurm website here.

Querying the State of a Job

You can find out about the state of your job and all other jobs in the queue using the command squeue.

>>> squeue --user=yourusername

This will give you the status of all your running and submitted jobs. You can also neglect the --user=yourusername to view the entire job queue for all users.

If you are using OzSTAR then the output of squeue --user=yourusername should look similar to the figure below.

The columns of the output are as follows:

JOBID: The JOBID that is given to the job. This ID is unique amongst all jobs past, present and future.
PARTITION: The type of ‘queue’ that the job is in. This is usually given by the name of the type of CPU that will be running the job.
NAME: The name of the job.
USER: The username of the person that submitted the job.
ST: The status of the job.
- R: Currently Running
- PD: Waiting for Resources (Pending)
TIME: The length of time the job has been running. If the job is pending (ST = PD) it will say 0:00.
NODES: The number of ‘nodes’ that the job has requested. A ‘node’ is a collection of many CPUs. OzSTAR has a few different types of nodes with different amounts of CPUs on each. For example the john (PARTITION = skylake) nodes have 32 CPUs each.
NODELIST(REASON): If the job is currently running (ST = R) this is the list of nodes that the job is using. If the job is pending (ST = PD) this is why the job is pending.

You can also use the OzSTAR Job Monitor Website for a graphical view of all the jobs that are running and in the queue.

Canceling a Job

Sometimes you will have a job that you need to cancel for some reason. You can cancel a running or submitted job at any time with scancel jobid.

>>> scancel 99999999

You can also cancel all of your jobs with scancel --user=yourusername or you can only cancel your “Pending” jobs with scancel -t PD.

Examples

Example 0

#!/bin/bash

#SBATCH --ntasks=1
#SBATCH --mem=100MB
#SBATCH --time=00:30:00

module purge
module load anaconda3/5.0.1

source activate py3

python example_python_job.py

This is the same example as shown throughout this tutorial.

Example 1

#!/bin/bash

#SBATCH --ntasks=1
#SBATCH --mem=100MB
#SBATCH --time=00:30:00

#SBATCH --job-name=Calculate_Mean
#SBATCH --output=slurm_output.txt

#SBATCH --mail-user=name@swin.edu.au
#SBATCH --mail-type=ALL
#SBATCH --account=oz999

module purge
module load anaconda3/5.0.1

source activate py3

python example_python_job.py

This is essentially the same as Example 0 but with a few additional parameters.

--job-name is being used to give a more meaningful name to the job. This is the name that will show up as NAME when using squeue.
--output is defining the file that all of the Slurm output (i.e. print statements) will be directed.
--mail-user and --main-type makes Slurm send an email when your job starts and completes.
--account is setting which group account this job belongs is associated with.

Summary

This tutorial is not meant as a comprehensive article covering all there is to know about using Slurm. Still, hopefully by now you feel confident enough to be able to write you own bash scripts and get jobs running on OzSTAR.

Acknowledgements

Thank you to Ellert van der Velden (@1313e) for useful feedback on the draft of this article.