Using the Rocks cluster
Overview of PHYSPLANCK Cluster
physplanck.canterbury.ac.nz is a computing cluster consisting of one frontend node and 7 dedicated compute nodes. The cluster runs Rocks Cluster 6.0, which is a collection of cluster management tools build on top of CentOS 6.2 (a standard Redhat Enterprise rebuild, like Scientific Linux). Job scheduling is through Sun Grid Engine (SGE).
Cluster hardware
Headnode:
Intel Core i7-3770 3.40GHz CPU per node
16 GB of RAM per node
500GB HDD plus 1TB HDD data disk
7 Compute nodes:
Intel Core i7-3770 3.40GHz CPU per node
16 GB of RAM per node
500GB HDD per node
7 additional compute nodes:
Intel Xeon E3-1240v3 3.40GHz CPU per node
32 GB of RAM per node
500GB HDD per node
Storage node:
4TB of RAID1 storage via NFS.
Only the frontend is directly connected to the outside network. The compute nodes are interconnected with a dedicated Gigabit ethernet network.
Cluster Storage Configuration:
/home -- The user home directories reside on the headnode 500GB disk and are NFS exported to all the compute nodes.
/home/data -- The headnode's 1TB disk is NFS exported to the compute notes at /home/data. Users can create subdirectories under /home/data as on a standard scratch and the contents will be accessible from any compute node.
/home/data1 -- The storage node nas-0-0 4TB disk array is NFS export to all the nodes.
/home/data2 -- The storage NAS 18TB disk array is NFS exported to all the nodes, and is also available via our other linux servers as /home/phys/planck
/state/partition1 -- Each compute nodes has a local working area mounted at /state/partition1. These areas are local to each node and are not visible to the other nodes. If your job is disk intensive, you may wish to have your jobscript copy files onto the local work space at the beginning of the job. If your job writes files into the local work space, be sure to have your jobscript move them out when it is done. This is for two reasons. First you will not necessarily know what node your job ran on, so it may be difficult to find the files later. Secondly, it leaves the node clean for the next user.
Running Jobs and Job Control
Interactive Jobs
You may run interactive jobs on the frontend. Provided the job is not too CPU intensive it should not disrupt the cluster. Tasks such as editing, building and debugging code are most simply accomplished by logging into physplanck.canterbury.ac.nz and doing things there.
If you have a CPU intensive interactive job, you can use the command qlogin. This will select a free compute node and log you into it. This is perfect for running interactive jobs that require occassional user input.
If you must subvert the batch system selection provided by qlogin, it is possible to SSH directly into the compute nodes. This is not recommended unless you have a good reason.
Batch Jobs
You must first create an SGE job script file in order to tell SGE how and what to execute on the nodes.
job.sh
#!/bin/bash
#$ -cwd
#$ -S /bin/bash
#
# job under the Sun Grid Engine
#
# use qsub job.sh to submit the job.
# Note, this shell script must be executable
# (i.e. chmod a+rx job.sh)
#
# The #$ statements are Sun Grid Engine Directives
# which SGE uses to decide when and where to run the job
#
# Name the job mymatlab_job
#$ -N mymatlab_job
#
# tell SGE to put a "hard" cpu
# limit of 10 hours on the job
#
#$ -l h_cpu=10:0:0
#
# Run the job ... in this case locally on the compute node using local space and cleaning up after completion. Test throughly before bulk use!!!
export TMP=/state/partition1/{usercode}
export TMP_CURR=$TMP/curr
export HOME_CURR=$PWD
mkdir -p $TMP
mkdir -p $TMP_CCURR
cp $HOME_CURR/* $TMP_CURR
cd $TMP_CURR
/share/apps/matlab/bin/matlab -nojvm -nodisplay -singleCompThread -r mymatlab_job
cp $TMP_CURR/mymatlab_job.* $HOME_CURR
rm -r $TMP_CURR
mymatlab_job.m
for k = 1:10
b2sum=0.0
bsum = 0.;
for i = 1:1000
x =[1;2;3;4;5;6;7;8;9;10];
x= [x;x];
x= [x;x];
x=[x;x] ;
x=[x;x];
y = x*33. + random('norm', 0 ,3.);
b=inv((x'*x))*(x'*y);
bsum = bsum +b;
b2sum = b2sum + b*b;
end
bav = bsum /1000.
bvar=(b2sum-(bsum*bsum/1000.))/1000.
bstd=sqrt(bvar)
end
To run the program use the following command
qsub job.sh
to check the job has been submitted
qstat
or
qstat –f
To delete a job
qdel job.sh
Futher details can be seen in the userguides at http://physplanck.canterbury.ac.nz
SGE Paramters explained:
These are just a few parameter options, for more type man qsub while logged into the cluster.
- #$ -cwd tells SGE to run the script from the current directory. This defaults to your home directory (/home/username). This could be problematic if the path on the head node is different than the path on the slave node.
- #$ -j y parameter tells SGE to join the errors and output streams together into one file. This file will be created in the directory described by #$ -cwd and will be named in this case test.sh.o# where # is the job number assigned by the SGE.
- #$ -S /bin/bash parameter tells the SGE that the program will be using bash for its interpreter. Required.
- #$ -N testJob3 parameter tells the SGE to name the job testJob3.
- #$ -M username@canterbury.ac.nz parameter tells the SGE to send an e-mail to username@canterbury.ac.nz when the job is done.
- #$ -m abe parameter tells the SGE to send an e-mail to the e-mail defined using the above -M parameter when a job is aborted,begins or ends.
- #$ -l h_data=1024M parameter tells the SGE to reserve 1024 Megabytes of RAM per core for this job.'
- #$ -q all.q@compute-0-5.local parameter tells the SGE to run the job on compute node compute-0-5.
- #$ -pe orte 8 parameter tells the SGE to run the job distributed in parallel on the number of cores specified (8 in this case). Other options include smp, mpi, mpich. See output of "qconf -spl" for available options.
- #$ -o output.log parameter tells SGE to send the output stream to this file.
- #$ -e output.err parameter tells SGE to send the error stream to this file.