Supercomputing

Overview

What isn’t a supercomputer

A super fast computer that just runs your program faster.

What is a supercomputer

Nibi (latest SHARCNET supercomputer)

752 computers (nodes)

140,928 cores

662 terrabytes (TB) of RAM (most 4 gigabyte (GB)/core)

25 petabytes (PB) of SSD disk storage (home, project, and scratch)

80 petabytes (PB) of transparent tape storage (nearline)

288 NVIDIA GPUs (H100 Hopper) and 25 AMD APUs (MI300A)

200/400 gigabit (G) ethernet

How do you use a supercomputer

serial

(easy) many computers means you can run many programs indepedently at the same to solve many independent problems

parallel

(hard) many computers can (sometimes) be programmed to all collaborate and solve a single problem together

cloud

virtual computer on the internet (like Amazon’s Elastic Compute Cloud, Microsoft’s Azure cloud, or Google’s Compute Engine)

Canada

The players

National

Digital Research Alliannce of Canada

Regional

Compute Ontario, Calcul Québec, ACENET, BC DRI Group, and Prairies DRI

Ontario

Center for Advanced Computing (CAC), HPC4Health, SciNet, and SHARCNET

Getting an accounts

Cost

no cost

Who

faculty or those sponsored by faculty that have an account

How

https://ccdb.alliancecan.ca

apply for a Digital Research Alliance account

pick the desired regional consortia accounts

What software is available

Operating system

Linux (Rocky/AlmaLinux 9)

Programming languages

C/C++, Fortran, MATLAB/Octave, Python, R, Julia, etc.

Parallel development support

pthreads, MPI, OpenMP, CUDA, OpenACC, OpenCL

Other

common open source and commerical packages (e.g., OpenFOAM, Fluent, and STAR-CCM+)

How to use

resources are scheduled to avoid collisions and ensure fair access

access from anywhere on the internet using secure shell (SSH) to enter commands and a secure file transfer (SFTP) to transfer files

tell the supercomputer what progam (command) you want it to run and then do something else till it does it

Typical workflow

transfer your data and/or programs to the supercomputer using SFTP

login (bring up a window in which you can enter commands) to the supercomputer using SSH

enter commands to tell the scheduler what you want it to run when the required resources are avaialable

do something else until you get notification that your commands have completed running

transfer the resulting data from the supercomputer to your computer

Using

Required Software

Clusters are accessed via the secure shell programs

ssh

used to run commands (secure shell)

scp

used to copy files (secure copy)

sftp

alternative to copy files (secure file transfer protocol)

from anywhere on the internet using the address CLUSTER.alliancecan.ca (capitalized terms indicate you should substitute something appropriate).

Windows

Most common options are the GUI

mobaxterm [https://mobaxterm.mobatek.net/] is a good freeware GUI choice that provides everything in one package including a graphical sftp, X11 server, and local shell

PuTTY [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html] is a basic open-source ssh-only option

Windows 10 has native versions of the command-line tools too

Windows subsystem for Linux [https://docs.microsoft.com/en-us/windows/wsl/] provides full linux text environment including secure shell

OpenSSH installable component [https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse] provides secure shell inside Windows command or power shell

MacOS X and Linux

Command-line versions are builtin and available in the shell

search for and start a terminal application

Browser

Open OnDemand (OOD) web based interface is available for some clusters

https://ondemand.sharcnet.ca

https://ondemand.scinet.utoronto.ca

while others have JupyterHub (JH)

https://jupyterhub.rorqual.alliancecan.ca

https://jupyterhub.fir.alliancecan.ca

Accessing

Logging in

ssh USER@CLUSTER.alliancecan.ca

login to CLUSTER as USER and start a remote shell session

for GUI ones (e.g., mobaxterm, putty, etc.), enter USER into username field, CLUSTER.alliancecan.ca into host name field, and connect

for web-based OOD login on web interface and choose Shell Access from the menus at the top

Copying files

scp USER@CLUSTER.alliancecan.ca:REMOTE LOCAL

login to CLUSTER as USER and copy REMOTE file to LOCAL file

scp LOCAL USER@CLUSTER.alliancecan.ca:REMOTE

login to CLUSTER as USER and copy LOCAL file to REMOTE file

for GUI ones (e.g., mobaxterm, etc.) enter USER and CLUSTER, connect, and then drag and drop files to transfer

for web-based OOD login on web interface, choose home directory, and use the upload/download button

Shell

Commands, entered interactively or ran from a file, tell computer what to do

a bit of an initial learning curve, but

much easier to automate, document, and share

Basic command

PROGRAM ARGUMENTS...

arguments usually options (switches) followed by strings (e.g., file names, etc.)

short options are a single dashes followed by letter for each switch (e.g., rm -fr mydir)

long options are a double dash followed by a descriptive string (e.g., rm --force --recurse mydir)

Help

help COMMAND

show help for builtin command (like “cd”)

man COMMAND

show the manual page for command (‘q’ to quit)

Getting around

ls

list files

pwd

what folders you are in

cd DIRECTORY

change to directory named directory (use “..” for previous one)

Copying/renaming/moving/removing files and directories

cp SRC DEST

copy file/directory (add -r for directory)

mv SRC DEST

move/rename file/directory

rm PATH

remove file (add -r for directory)

Editing files

nano PATH

edit file named file

Digital Research Alliance of Canada software (scientific packages)

module list

show what software is load environment

module avail

show what software you can load

module spider PACKAGE

search for package

module load PACKAGE

load package

module unload PACKAGE

unload the package

Running/submitting, viewing, and killing jobs

salloc

start interactive job

sbatch SCRIPT

queue batch job

squeue

list jobs

skill JOBID

kill job

Filesystems

Storage Folder

Type

Quota (size)

(files)

Surge (size)

(files)

(days)

/home/USER

User

50GB

500K

70GB

700K

60

/scratch/USER

User

1,000GB

1,000K

20,000GB

1,000K

60

/project/PROJECT

Group

1,000GB

500K

NA

NA

NA

nibi scratch has no limit, but requires you to drop to 1TB every 60 days

quick scratch, project (default), and projects (all) links under home

Jobs

Check documentation page for desired software on docs.alliancecan.ca [https://www.docs.alliancecan.ca]

program specifc instructions

licensing details for commerical software

All significant computations (> few minutes) run as SLURM jobs

only way to access the 752 compute nodes

initial interative type-in-commands test runs with salloc

final non-interactive put-commands-in-file runs with sbatch

Batch file for sbatch have the general format

##!/bin/bash
##SBATCH OPTION1
##SBATCH OPTION2
...

COMMANDS TO RUN
...

Job options

Specify the resources and accounting required to run a job

as options to salloc and sbatch commands, and/or

in start of job files using special #SBATCH ... lines

Basic options

--account def-SPONSOR

account to allocate usage to

--time D-HH:MM:SS

maximum runtime before killing (add a bit of buffer)

--mem-per-cpu N

memory per CPU (suffix with K, M, G, or T for units)

Multi-threaded jobs

limited to maximum number of CPUs in one computer (generally 32)

may need export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK in job script

--cpus-per-task N

number of threads per process (threaded)

Multi-processor jobs (MPI)

need to start the program with the mpirun or srun commands

--ntasks

number of processes (MPI)

GPU jobs

--gpus-per-task TYPE:N

number of GPUs per process

Breakpoints

partition_stats

show paritioning of compute nodes eligible for jobs

Exceeding these resources decreases the number of machines that will run a job

runtime > 3, 12, 24, 72, 168, and 672hrs

memory/cpu > 12MB

OpenFOAM

Enabling

Search through the available modules to see what OpenFOAM ones exist

$ module avail openfoam

This only shows versions that are compatible with other loaded modules. All versions can be found with the spider command along with directions on what else needs to be loaded to load them

$ module spider openfoam
$ module spider openfoam/13

Note that there two releases of OpenFOAM [https://en.wikipedia.org/wiki/OpenFOAM#History]. One is openfoam.com [https://www.openfoam.com/] and uses vYYMM (year and month) version numbering. The other is openfoam.org [https://www.openfoam.org/] and uses X and X.Y version numbering. With the exception of a some specific features, the two are largely similar.

Load the openfoam module (not specifying a version uses the default). Note that spider may show additional modules that you need to load first.

$ module load openfoam

Can use the show command to see what this does. The environment variables that it sets can be accessed using the $VARIABLE syntax (substuting VARIABLE for the name of the variable).

$ module show openfoam

Currently the vYYMM versions cannot be unloaded/purged without beaking your environment. You can logout and back in instead to reset your environment.

Create the top-level user OpenFOAM directory and change to it (not strictly required, but recommended by the tutorials)

$ mkdir -p $FOAM_RUN
$ cd $FOAM_RUN

Tutorial

OpenFOAM comes with several tutorials in the $FOAM_TUTORIALS directory. This includes the ones in the user manual [https://doc.cfd.direct/openfoam/user-guide-v13/tutorials].

$ ls $FOAM_TUTORIALS

Until version 11, the standard first tutorial was a cavity flow problem [https://doc.cfd.direct/openfoam/user-guide-v10/cavity]. It is still available in the legacy directory though, and it is one of the simplest examples, so we will use it.

$ cp -r $FOAM_TUTORIALS/legacy/incompressible/icoFoam/cavity/cavity .
$ cd cavity

Generate mesh

Generate the mesh data files in constant/polyMesh from the higher-level description file system/blockMeshDict

$ blockMesh

Interactive run

Run the simulation (incompressible flow using the PISO algorithm) in an interactive session to test

$ salloc --mem-per-cpu=500M --ntasks=1 --account=def-SPONSOR

$ icoFoam
$ exit

Batch run

Create a run.sh (name not important) using nano to do the same

$ nano run.sh

#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --mem-per-cpu=500M
#SBATCH --ntasks=1
#SBATCH --account=def-SPONSOR
#SBATCH --output=icoFoam-%J.log
icoFoam

Submit the batch file to the cluster to run

$ sbatch run.sh

New case

Make a new cavityMPI case based on the cavity case setup

$ cd ..
$ foamCloneCase cavity cavityMPI
$ cd cavityMPI

Run in parallel

Use foamGet to add the sample system/decomposeParDict dictionary (pick the non-annotated one as suggested)

$ foamGet decomposeParDict

Edit it to use four processors and the fully automatic scotch method

...
numberOfSubdomains  4;
...
method              scotch;
...

Decompose the cavity

$ decomposePar

Run the simulation in parallel using four processors

$ salloc --mem-per-cpu=500M --ntasks=4 --account=def-SPONSOR

$ mpirun icoFoam -parallel
$ exit

Collect the output back into a single file

$ reconstructPar

Prep for ParaView

Create a NAME.foam file to give something to open in ParaView (name doesn’t matter – just identifies it as an OpenFOAM directory)

$ touch cavity.foam

Can also generate VTK input files

$ foamToVtk

ParaView

Availability

Personal

Copy data to personal machine for analysis.

Windows and OS X

Download latest version from ParaView website [https://www.paraview.org/download/]

older version may work if newer fails on older OS or graphics card

Linux

Install with your package manager (can also download as above)

Debian and Ubuntu: sudo apt-get install paraview

Fedora: sudo yum install paraview

Clusters

Remote desktop to avoid having to transfer data

Open OnDemand (OOD) (nibi, trillium)

JupyterHub (JH) (rorqual, fir)

Some general hints for picking job options

reservation: None unless taking part in a course (not this one) with a reservations,

account: ending in gpu if using a GPU (including hardware OpenGL) and cpu otherwise,

time: 8hrs or less (runs in the interactive queue) for faster start,

cores: enough cores so memory/core is 12G or less (regular memory queue) for faster start,

memory: at least 8GB for a desktop session (see cores note about keeping memory/core 12G or less),

oversubscription: select if possible (shares CPUs with other users) for faster start,

gpu configuration: if using a GPU pick type cluster has most of for faster start, and

user interface: JupyterLab (Jupter Notebook is the older version before the renaming).

OOD

Login and pick Compute Node -> Desktop (nibi) or Apps -> Trillium Desktop (trillium)

https://ondemand.sharcnet.ca

https://ondemand.scinet.utoronto.ca

JupyterHub

Login, start job, pick Mate Desktop or XFCE4 Desktop

https://jupyterhub.rorqual.alliancecan.ca

https://jupyterhub.fir.alliancecan.ca

If the browser fails to connect to the desktop, trying waiting around thirty seconds and then refresh the failed connection page (press the F5 key).

If you see … relocation error: …/rebind.so … in your terminal, you have run into an issue caused by the JupyterHub desktop websockify LD_PRELOAD. Disable this by running

$ unset LD_PRELOAD

If you see warning messages about ipykernel when loading modules, you have run into issues caused by JupyterHub pre-loading the ipython-kernel. Unload this by running

$ module --force unload ipython-kernel python

Tutorial

Working through the ParaView tutorial [https://www.paraview.org/Wiki/The_ParaView_Tutorial] is one of the quickest and easiest ways to get up-to-speed on ParaView.

basic usage (what we will be going over)

batch python scripting

visualizing large models

$ cp -r /home/tyson/ParaView .

Basis of visualization

map raw data to visual data

spacial and temporal data

topology and types of grids

User interface

menu bar

tool bars

pipeline browser

properties panel

view

Basic interface

creating a source

interacting with a 3d view

modifying visualization paramaters (filter, display, view)

undo and redo (regular vs camera)

Loading data

opening file (selecting which variables to load)

representation and field coloring

scaling

Filters

selecting filters (toolbar, menu, and search)

applying a filter (contours, slices)

Multiview

creating a multiple views

linking cameras

re-arranging the views

Vector visualization

streamlines

tubes and glyphs

surface LIC

Volume rendering

Enabling

Transfer function

Plotting

histogram plot

plot over a line in space

plot over a curve

plot over time

Selections

Query and view based selections

Data vs spatial selections

Selection labels

Extract selection and spreadsheet

Ansys

Licensing

Types:

CMC/SHARCNET/Campus

Server/Floating

License File

Purchase/Renewal

Wiki Documentation

Abaqus

Ansys

Comsol

Starccm

https://docs.alliancecan.ca/

Ansys Online Help

Release Notes

Users Guides

Analysis Guides

Tutorials & Videos

Optimization & Performance Guides

Material Reference

Command Reference

https://docs.alliancecan.ca/wiki/Ansys#Online_documentation

Ansys Electronics Desktop (AEDT)

Ansys Help - HFSS (high-frequency simulation software) - Videos:

The ANSYS Electronics Desktop Environment [https://www.youtube.com/watch?v=oOH6hVP6vcA]

ANSYS HFSS: Overview of Antenna Simulation - Part 1 [https://www.youtube.com/watch?v=oOH6hVP6vcA]

Demo Interactive (salloc) AND Queue (sbatch)

ANSYS Tutorials

Meshing - Meshing Tutorial Guide

Can Combustor

Fluent - Fluent Tutorials

Fluid Flow in an Exhaust Manifold

Fluent - Fluent Workbench Tutorial

Introduction to Using Ansys Fluent in Ansys Workbench: Fluid Flow and Heat Transfer in a Mixing Elbow

Mechanical APDL - Introductory Tutorials

Structural Tutorial - Static Analysis of a Corner Bracket

Mechanical Application - Mechanical Tutorials

Rigid Dynamics Analysis of an Actuator Mechanism using Joints and a Spring

Index

Table of Contents

Super Computing

Overview

What isn’t a supercomputer

What is a supercomputer

How do you use a supercomputer

Canada

The players

Getting an accounts

What software is available

How to use

Typical workflow

Using

Required Software

Windows

MacOS X and Linux

Accessing

Logging in

Copying files

Shell

Basic command

Filesystems

Jobs

Job options

Breakpoints

OpenFOAM

OpenFOAM

Enabling

Tutorial

Generate mesh

Interactive run

Batch run

New case

Run in parallel

Prep for ParaView

ParaView

VDI nodes

Accessing

Software

ParaView

Personal computer

VDI

Tutorial

Basis of visualization

User interface

Basic interface

Loading data

Filters

Multiview

Vector visualization

Volume rendering

Plotting

Selections

Ansys

Licence

Fluids

Mechanical

Electrical

Workbench

Documentation

Exercises

OpenFOAM

Cavity

Dam break

ParaView

OpenFOAM

Cavity

double the resolution

Dam break

copy the dam break tutorial and run it on the cluster (up to post-processing)

https://cfd.direct/openfoam/user-guide/v6-dambreak/

do a parallel run (running in parallel section – increasing resolution is optional)

ParaView

extracting just girder

extracting a plane

glyphs

graphs (multiple displays)

plot over line

plot over time (sample data)

enabling LIC plugin and showing on plane

clipping off the sides

enabling volume rendering for low pressure regions

stream tracers, tubes, and glyphs