Using Psi4 on a High Performance Cluster

This blog post is advanced but short. Psi4 is an open source quantum mechanics package that is built on CFOUR used for managing calculation of integrals and parallelizing them on mass scale. Depending on the size and complexity of the orbitals, it can be pretty expensive to run.

Our cluster is using the Sun Grid Engine like others with shared-memory processing. Each node has different amount of cores available that we can request.

The SGE header should have this flag with the smp:

#$ -S /bin/bash
#$ -cwd 
#$ -V
#S -pe smp 64

For now, I want to maximize the core allocation where I control all of them on one node. I have a reason for this. Next in your bash script set your MKL and OMP flags which tells the computer okay we want to multiprocess the math, MKL , and it’s processing architecture, OMP, where all processes share memory and data.

export MKL_NUM_THREADS=64
export OMP_NUM_THREADS=64

In your python script when you are running your psi4, each core runs 1 thread with a max size of 1GB of memory:

import psi4

psi4.core.set_num_threads(1)
psi4.core.set_memory('1000mb')

Now what happens, is that this is how much is available to the system.

Now what is going on here is that the system I was running wasn’t too intensive so psi4 didn’t utilize the full space of the 64 cores. You can see that the running average was 6. Oddly enough, I see these spikes on the server and where it ocassionally it asks for the 64 but doesn’t shoot over with the redline bar.

For now, I can get my data but this is something to note in the future.