INSTALLING TORQUE RESOURCE MANAGER Compiled and written by Andrew O’Hara, Haverford Colleg

INSTALLING TORQUE RESOURCE MANAGERCompiled and written by Andrew O’Hara,Haverford College’09.March11,2007TORQUE is an OpenSource batch system and resource manager based on OpenPBS2.3.12 that provides several improvements.Some of these improvements include GUI support,mas-sive scalability of both cluster size and job size,and better usability.Because TORQUE is mainly just a resource manager,rather then using the built in scheduler,it is commonly used with a more advanced scheduler such as Maui.This guide will help install TORQUE, but you should then follow the guide for installing and configuring Maui afterwards.Prior to the installation of TORQUE,password-less ssh across the cluster needs to be working.In addition to being able to access all compute nodes from the headnode’s root account,it is recommended that each compute node can then ssh back password-lessly.1.First,download the tar ball:/downloads/torque/torque-2.1.8.tar.gz(note that you may want to look in their torque directory to see if a later version is available).Untar it by running‘tar-zxvf torque-2.1.6.tar.gz’2.Change into the directory with‘cd torque-2.1.8’.During the configuration step,there are several settings we can pass to the script that will be incorporated to the build.For simplic-ity,we will use the default directory for installation(/var/spool/torque),however for larger (several100-1000nodes)clusters in which you are sure all nodes will always be configured with the same architecture and Linux distribution,using an nfs is more efficient and useful.If you want to change the directory use the option tag‘−−with-server-home=$TORQUECFG’, where$TORQUECFG is already defined(‘export TORQUECFG=<directory>’).Also,if using a64-bit system,add the tag‘−−libdir=/usr/lib64’to help with linking to the proper libraries.Assuming the configuration runs properly,we will then install the package.master$./configure−−enable-server−−enable-clients−−with-scp\−−with-default-server=<master hostname>master$makemaster$make installAlso,check that‘/usr/local/bin’and‘/usr/local/sbin’are set in the$PATH and if not,then update it.3.Next the PBS server needs to be configured.First,execute‘pbs server-t create’and then run‘qmgr’to create and edit the configurations database.This will produce a special shell prompt that will let us change some configuration options.For example,we’ll want to set the server operators and create the default queue.(Note:If the master hostname doesn’t work below,it is probably caused by a DNS resolving error.If you don’t have anyother PBS servers,the easistfix is to just use‘localhost’in its place.Qmgr$set server operators=root@<master name>Qmgr$create queue batch1Qmgr$set queue batch queue type=ExecutionQmgr$set queue batch started=TrueQmgr$set queue batch enabled=TrueQmgr$set server default queue=batchQmgr$set server resources default.nodes=1Qmgr$set server scheduling=TrueQmgr$quitAlternatively,you can use the prepackaged script for automated setup by executing‘torque.setup root’.4.Now edit thefile‘/var/spool/torque/server priv/nodes’and list out either the host names or the ip addresses of the compute nodes.If there are multiple processors on a node,add ‘np=<#>’after the host name/ip address of the compute node.For example each line should have a syntax similar to:‘node01np=2’.5.Create the client packages and put them on each node.For a cluster with64-bit ar-chitecture,replace i686with ia64.master$make packagesmaster$scp torque-package-mom-linux-i686.sh<node>:torque-package-mom-linux-i686.shmaster$scp torque-package-clients-linux-i686.sh<node>:torque-package-clients-linux-i686.sh master$scp torque-package-clients-linux-i686.sh<node>:torque-package-devel-linux-i686.shmaster$scp torque-package-clients-linux-i686.sh<node>:torque-package-doc-linux-i686.shmaster$dsh-a−−‘./torque-package-mom-linux-i686.sh−−install’master$dsh-a−−‘./torque-package-clients-linux-i686.sh−−install’master$dsh-a−−‘./torque-package-devel-linux-i686.sh−−install’master$dsh-a−−‘./torque-package-doc-linux-i686.sh−−install’Now,edit the configurationfile‘/var/spool/torque/mom priv/config’on each node(change the arch and opsys if necessary):arch x86#or ia64or x8664opsys Ubuntu#or whatever dist.you’re using$pbsserver<master hostname>$logevent255$usecp<master hostname>:/home/nfs/users/n/home#tells torque to use nfs mounting of users home directoryYou can edit a copy of this on the head node and push it out to each of the nodes using scp.Start the PBS client side on each node by executing‘pbs mom’.Also create thefile ‘/var/spool/torque/server name’and place in it the hostname of the master node.26.After setting up the TORQUE server,we need to test it to make sure everything is running properly.The following test procedure is taken directly from the TORQUE admin-istrator manual:master$qterm-t quick#this command shuts down the servermaster$pbs server#starts the server againmaster$qstat-q#verify all queues are properly configuredmaster$qmgr-c‘p s’#view additional server configurationmaster$pbsnodes-a#verify all nodes are correctly reportingmaster$echo“sleep30”|qsub#submit a basic jobmaster$qstat#verify jobs displayThefinal output at this stage should show the job state to be in the“Q”state.The job won’t actually run because there is no scheduler running.You can either activate the native scheduler‘pbs sched’or install a more advanced schedular like Maui.7.In order for each user to be able to submit jobs and run them(and get their results), each user name needs to be added to all nodes.The only account that will actually require a password will be the one the user logs into on the headnode(the rest are used by TORQUE to run thejobs).It is imperative that each username have the same uid across the network. Also,for each user that will be using the cluster,in order for the jobs to run without causing an exit error the following needs to be executed for each user.It is basically a modified passwordless ssh to allow the user’s account to communicate with‘pbs server’.root$useradd-s/bin/bash-d/n/home/<new user>-m<new user>root$passwd<new user>#so you can assign a temp password for userroot$dsh-a−−‘useradd-d/n/home/<new user><new user>’#this creates the user accounts without general login rights on nodesroot$cd/n/home/<new user>root$mkdir.sshroot$ssh-keygen-t rsa-q-N“”-f/n/home/<new user>/.ssh/id rsaroot$mv/n/home/<new user>/.ssh/id rsa.pub/n/home/<new user>/.ssh/authorized keysroot$chown-Rf<new user>/n/home/<new user>/.sshroot$chmod-Rf700/n/home/<new user>/.ssh/*root$su-<new user><new user>$ssh localhost#enter yesNow when you have your scheduler up and running,qsub submissions should runfine(as-suming everything else is working and your script is properly set up).References:‘Paragao’.HOWTO Torque/Maui-grid scheduler and resource manager.21Oct2006.3Gentoo Linux Wiki.Accessed:23Dec2006.</HOWTO Torque/Maui-grid scheduler and resource manager>. TORQUE v2.0Administrator Manual.19Sept2006.torque:torque wiki,Cluster Resources Inc.Accessed:02Jan2007.</wiki/doku.php?id=torque:torque wiki>‘xman’.Installing Torque+Maui.18Sept2006.Xman Drupal.Accessed:10Mar2007.</dp/node/22>Also thanks to Donald Tripp,HPC Systems Administrator for University of Hawai’i at Hilo for helping with step number7.4。