STEP-BY-STEP INSTRUCTION TO SET
UP A ROCKS LINUX CLUSTER with DUAL BOOT (Windows + Rocks) NODES
================================================================================
We have built two 20-PC cluster using Rocks 6.1.1 (Sand Boa) in the
School of Physics, Universiti Sains Malaysia (USM), Penang. They are
comsics.usm.my and anicca.usm.my. This document serves to record all
the steps involved while we build our cluster.
First, some useful IP setting information we used when installing the
Rocks Linux cluster. USMNET is the network of USM. The IP mentioned
below is that defined within USMNET.
Primary DNS: 10.202.1.27
Secondary DNS: 10.202.1.6
Gateway: 10.205.19.254
Public Netmask 255.255.255.0 (alternatively 255.255.254.0, or that
auto-suggested by Rocks installation DVD.)
Private IP: 10.1.1.1
Private netmask: 255.255.0.0.
Public netmask: 255.255.255.0 (alternatively 255.255.254.0, or that
auto-suggested by Rocks installation DVD.)
Note that these private netmask are default values suggested by the
Rocks installation disk.
IP names and address requirement
================================
You got to have an IP address for the cluster you are setting up, and
register the name of the cluster in your institute network. For
example, our comsics's IP address is 10.205.19.208, and the formal name
of the cluster, www.comsics.usm.my, is registered in USMNET so that the
IP 10.205.19.208 is associated with the name www.comsics.usm.my. Then
ths comsics cluster can be accessed either via comsics.usm.my or the IP
10.205.19.208. We have to request USM PPKT (the network administrator
of USM) to have the domain name comsics.usm.my registered under the IP
10.205.19.208. In USM, we request for Server Registration online at
http://infodesk.usm.my/infodesk/login.php
You may have to request for a named DNS and IP address from the network
administrator of your institute.
Installation procedure
=======================
0. Assures that the boot sequence of ALL PCs are set such that
CD/DVDROM is in the first boot up sequence.
1. A PC used as a frontend must have two network cards: one built-in
and another plug-in with a spec of 10/100/1000 Mb/s. Connect from
internet line a LAN wire into the built-in LAN port. This is identified
a the eht1 port. The eth0 port (plug-in LAN card) is connected to a
switch that must support 10/100/1000 Mb/s. Any other PCs used as
compute-nodes must also have LAN cards of 10/100/1000 Mb/s speed
(usually plug-in ones). All these plug-in LAN cards must also be
connected to the switch. No LAN wire shall be connected from a compute
node directly to the internet network. All LAN cable in a compute node
must only be connected to the switch.
2. Initially, switch on only the frontend PC. Leave the compute node
power off. The LAN switch must be on and assure that the LAN cables are
connected into the frontend and the switch in the way mentioned in 2.
Slot in the Rocks Cluster DVD installation disk, and type 'build' when
prompted by the screen.
3. When prompted, fill in the name, some miscellaneous info of the
cluster and the IP details as mentioned above. Choose automatic
partition if do not wish to customise the partition. If customised
partitioning is desired, suggest to use the following allocation:
SWAP: 10GB (make it large so that it can serve as temporary
life-saving RAM in case of critical situation during intensive
calculation)
/var: 20 GB (you may not know when /var is going to blow
up, which happen in cases of perpertual warning messages being
generated by the apache server or whatever protocal in the server due
to don't-now-what error triggered unexpectedly).
/boot: 100 MB
/: 24 GB (or larger if wished)
/state/partition1: Maximum
The installation will take place automatically once partitioning
begins.
4. Make sure to take out the DVD when it is ejected after about 20 - 30
min when Rocks is first successfully installed. Failure to retrieve the
installation disk from the CD drive renders repeated installation of
infinite times.
5. Rocks will reboot when finish installing for the the first time. The
first screen may see black screen with some warning sign if the
frontend PC does not has NVDIA GPU installed. Simply press 'enter' when
prompted so that the frontend fixes the problem automatically by
installing a generic display driver. A GUI shall be displayed after a
step times pressing enter.
Post-installation of the Rocks Frontend (Stage 1)
=================================================
Once the Rocks frontend is up and running, do the following:
Download two *.xml files from the www2 server. These *.xml files
contain the instruction of how the hardisks of the nodes are to be
partitioned. The *.xml files first be stored in /share/apps/configrepo.
The directory /share/apps/configrepo will be used to keep all other
files for post-installation purposes.
Now, instruct the frontend to partition the nodes so that they could be
used to host a dual-boot OS (Windows + Rocks). To do so, execute the
following command lines in the frontend terminal:
#####################################
mkdir /share/apps/configrepo
cd /share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_noformat_windows
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_format_all
cd
/export/rocks/install/site-profiles/6.1.1/nodes
mv
replace-partition.xml
replace-partition.xml.orig
cp
/share/apps/configrepo/replace-partition.xml_noformat_windows .
cp
/share/apps/configrepo/replace-partition.xml_format_all .
cp
replace-partition.xml_format_all replace-partition.xml
cd /export/rocks/install
rocks create distro
#####################################
In the above code, the replace-partition.xml (which is copied from
replace-partition.xml_format_all) file will provide instruction (during
the execution of the insert-ethers Compute procedure, see the following
section) of how the node should be formatted. Note that for
replace-partition.xml node-formatting instruction to take effect, you
must do the following:
cd /export/rocks/install
rocks create distro
If replace-partition.xml_format_all is used as replace-partition.xml,
all content of the node's hard disk will be formatted. After
formatting, each node will have a 40GB of space in the first partition
(to be used for Windows installation), a small /state/partition2 (to be
used for KDE partition manager) and other default partitions.
In the event where you are installing a fresh frontend node where all
the compute nodes have been partitioned according to
replace-partition.xml_format_all (say by an earlier installation), you
should use replace-partition.xml_noformat_windows in place of
replace-partition.xml_format_all as replace-partition.xml (the xml file
to be 'rocks create distro'). replace-partition.xml_noformat_windows
instructs that the all partitions in the nodes shall NOT be formatted
except the partition of root directory '/' in the nodes.
Installing compute nodes
========================
Once 'rocks create distro' is done, you can proceed to the next step:
Installing the compute nodes.
1. First and foremost, issue the following commands as root in terminal:
rocks set appliance attr
compute x11 true
After the above command, the nodes will be equipped with GUI (Gnome
desktop) when they are installed via insert-ethers subsequently (see
later).
2. Connect the frontend and all the compute nodes to the LAN switch via
their eth0 ports. Of course, assure the power supply to the LAN switch
is ON. In the frontend's terminal, issue the command as root in the
frontend terminal:
insert-ethers
When prompted, choose `Compute'. Manually slot a Rocks Cluster
installation DVD into the individual PC. They will be detected if the
LAN wires are connected properly into the LAN switch via eth0. Warning:
insert-ethers ONLY after the command 'rocks set appliance attr compute
x11 true' is issued.
3. Boot up the nodes using the Rocks installation disk. These nodes
shall be partitioned according to the spec as specified in
replace-partition.xml. A node usually takes about 15 - 20 mins (or
less) to be configured. YOU MUST RETRIEVE THE INSTALLATION DISK FROM A
NODE ONCE IT FINISHES TO PREVENT REPEATED INSTALLATION. Once a node
completes its installation, it will reboot into a usual-looking CENTOS
log-in screen.
Post-installation of the Rocks Frontend (Stage 2)
=================================================
We will put some necessary files into the Rocks frontend. This is done
via wget and scp into a private server directory from the server
www2.fizik.usm.my (password required in a scp fetch).
The wget part:
cd /share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/
cd /share/apps/configrepo
wget http://www2.fizik.usm.my/tlyoon/Downloads/Apache_OpenOffice_4.0.1_Linux_x86-64_install-rpm_en-US.tar.gz
tar
-xzvf Apache_OpenOffice_4.0.1_Linux_x86-64_install-rpm_en-US.tar.gz
The scp part:
scp -r
tlyoon@www2.fizik.usm.my/home/tlyoon/repo/configrepo/rocks_611/*
.
chmod +x *.sh *.conf *. *.run
Add users to the Rocks cluster
sh
/share/apps/configrepo/useradd.sh
sh
/share/apps/configrepo/useraddc.sh
sh
/share/apps/configrepo/useradd_human.sh
rocks sync users
cd /share/apps/configrepo
Install openoffice:
sh inst_openoffice.sh
Install some necessary applications in the frontend
yum install -y gftp.x86_64
yum install -y
compat-libstdc++-33.i686
Copy virtual Windows.XP' vdi file to VBtemplate and carbon capy one to
VBlocal. This is quite a lengthy process, may take about 30 mins (or
less)
#########################
mkdir /share/apps/VBtemplate
mv
/share/apps/configrepo/windowsXP.vdi /share/apps/VBtemplate
mkdir
/state/partition1/VBlocal
cp
/share/apps/VBtemplate/windowsXP.vdi /state/partition1/VBlocal
chmod 777
/state/partition1/VBlocal/windowsXP.vdi
cd /share/apps/configrepo
#########################
Install mathematica (manual attention required)
sh
/share/apps/configrepo/mathematica1.conf
Install mathematica license manager (manual attention required. Read
the instruction as contained in addmathlm)
sh /share/apps/configrepo/addmathlm
To activate mathlm, you may need the following URLs:
https://user.wolfram.com/portal/requestAK/506f8a2585f11524c0d64de6d0589e4f427ba1af
https://user.wolfram.com/portal/passwordrequest.html
Installation of compilers and libraries in /share/apps in the frontend
(in sequence)
====================================================================================
List of compilers and libraries to
be installed in /share/apps in the frontend (icc must be installed
first)
============================================================================================================
icc
http://www2.fizik.usm.my/configrepo/howto/intel/inst_icc_11.0.081_sa
ifftw
http://www2.fizik.usm.my/configrepo/howto/intel/inst_ifftw_sa
ifort
http://www2.fizik.usm.my/configrepo/howto/intel/inst_ifort_111072_sa
impi 4.1.0.024 (* may not wrok without valid lisence file*)
http://www2.fizik.usm.my/configrepo/howto/intel/inst_impi_410024_sa
lapack, blas (Netlib,
non-optimised)
http://www2.fizik.usm.my/configrepo/howto/BLAS_and_LAPACK/inst_blas
http://www2.fizik.usm.my/configrepo/howto/BLAS_and_LAPACK/inst_lapack
mpich2
http://www2.fizik.usm.my/configrepo/howto/mpich2/inst_mpich2_314_intel_sa
fftw3
http://www2.fizik.usm.my/configrepo/howto/fftw/inst_fftw334_sa
gcc-4.8.4
http://www2.fizik.usm.my/configrepo/howto/gcc/inst_gcc_4.8.4
openmpi
ATLAS (tuned and optimised lapack and blas)
install hdf5
install netcdf
install PnetCDF
List of physics software to be
installed in /share/apps in the frontend
=======================================================================
g09
http://www2.fizik.usm.my/configrepo/howto/gaussian/inst_g09
dftb+
http://www2.fizik.usm.my/configrepo/howto/dftb+/inst_dftb+_sa
pthbhga
mkdir /share/apps/ptmbhga
cd /share/apps/ptmbhga
scp -r
tlyoon@www2.fizik.usm.my:/home/tlyoon/repo/ptmbhga/ptmbhga_anicca/* .
cd VERX
Modify
Makefile (locate -llapack and set mpif90 gnu version)
make
runopt
cd
/share/apps
scp -r
tlyoon@www2.fizik.usm.my:/home/tlyoon/repo/runopt_VER2.0
lammps, gnu version
http://www2.fizik.usm.my/configrepo/howto/mylammps/gnu/inst_lammps_gnu_Rocks611
lammps,intel version
http://www2.fizik.usm.my/configrepo/howto/mylammps/intel/inst_lammps_intel_Rocks611_generic
TestHsu3
wien2k
http://www2.fizik.usm.my/configrepo/howto/wien2k/ver_140611/inst_wien2k_140611_parallel_sa
ptmbhga_lammps
ptmbhga_dftb+
ptmbhga_g09
deMon2k
VASP
Mathlab
USPECK
Post-installation of Rocks compute nodes
=========================================
Right after all nodes are up and running:
1. Install some necessary packages in the nodes
rocks run host 'nohup yum
install -y compat-libstdc++-33.x86_64 compat-libstdc++-33.i686
brasero.x86_64 gftp.x86_64 &'
rocks run host 'nohup sh
/share/apps/configrepo/inst_virtualbox &'
rocks run host 'nohup sh
/share/apps/configrepo/inst_openoffice &'
rocks run host 'mkdir
/state/partition1/VBlocal/'
3. The following line cp *.vdi to local hard disk. This process takes a
long time, a few hours at least.
rocks run host 'nohup cp
/share/apps/VBtemplate/windowsXP.vdi /state/partition1/VBlocal/ &'
4. Installation of mathematica has to be done locally in each node. It
can't be done via rocks run host
sh
/share/apps/configrepo/mathematica1.conf
Setting up dual boot in the compute nodes
=================================================
This stage is to be carried out only when everything described above
are completed. Please proceed to
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/howto_set_up_dual_boot_for_nodes.txt
Maintenance
===========
At this stage the Rocks cluster should have already up and running.
However at times some possible glitches may occur to the cluster
involving frontend and nodes alike. We shall address some possibilities
in the following, and their remedy.
In the event of electric blackout or unintended hardware failure, a few
possibilities may occur.
1. The frontend and all the nodes reboot automatically as normal,
without the need of any manual intervention. The cluster as a whole
appears normal and working as usual.
2. Auto re-installation of the nodes may be triggered. The
auto-reinstallation may just proceed automatically and seamlessly
without the need of manual intervention. Upon completion of the
re-installation, the nodes appear normal and working as usual. All
previous applications and software existed in them are not lost.
3. Auto re-installation of the nodes are triggered. The auto
re-installation may just proceed automatically and seamlessly without
the need of manual intervention. Upon completion of the
re-installation, the nodes appear normal and working as usual, BUT all
previous applications and software previously existed in some of the
nodes are lost. In this case, you have to reinstall all these lost
applications and software in these lost nodes.
5. The worst case scenario will be: the frontend just get corrupted and
the Rocks cluster as a whole could not resume. Re-installation of the
frontend (hence the Rocks cluster) is necessary.
Ideally, the customization of the frontend and nodes should be as
painless as possible. It is particularly more so in the case where the
re-installation of nodes or frontend are forced to take place. To make
the customization as easy as it could possibly be, the strategy is to
prepare a list of installation scripts (written in bash script) that,
once executed, install the intended program/software package
seamlessly. These one-click installation scripts are to be found in
/share/apps/configrepo, and are usually named in the format of
inst_XXX, e.g., inst_openoffice, inst_virtualbox, etc.
To install a program across all nodes with a one-liner script, issue
rocks run host "sh
/share/apps/configrepo/inst_XXX"
where XXX is the neme of the program to install. To install the program
XXX using the inst_XXX script in a compute node X,
ssh -X -Y compute-0-X
sh
/share/apps/configrepo/inst_XXX,
The script /share/apps/configrepo/mathematica1.conf will install
Mathematica (but you have to standby to interact manually during the
installation process).
Re-installation of the frontend
==============================
This worst-case scenario could be forced upon us in the event of hard
disk breakdown, which is not entirely unlikely. In this case, the
re-installation process have to be repeated by following through the
instructions and procedures as described in this document. However,
since the nodes may have already been pre-installed with Windows and
KDE partition earlier, you don't want the re-installation to wipe them
off. So, make sure that when you re-install the frontend, mind to
assure that you are using the right replace-partition.xml, i.e, use the
replace-partition.xml_noformat_windows as the replace-partition.xml
when performing the 'rocks create distro' act.
#####################################
mkdir /share/apps/configrepo
cd /share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_noformat_windows
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_format_all
cd
/export/rocks/install/site-profiles/6.1.1/nodes
mv
replace-partition.xml
replace-partition.xml.orig
cp
/share/apps/configrepo/replace-partition.xml_noformat_windows .
cp
/share/apps/configrepo/replace-partition.xml_format_all .
cp
replace-partition.xml_noformat_windows replace-partition.xml
cd /export/rocks/install
rocks create distro
#####################################
Theoretically, the re-installation of frontend and the cluster as a
whole should be a seamless process and much easier than in the first
instance of installation. Inst_XXX scripts in /share/apps/configrepo
make the installation of software and applications in the frontend and
nodes an easy task. You may not need to copy the virtual Windows.XP's
dvi file into the nodes anymore since, as a legacy from previous
installation, they are already sitting in the nodes's
/state/partition1. Also due to legacy, there is no need to reinstall
the Windows and KDE partition manager when you are doing the
re-installation.
By Yoon Tiem Leong
School of Physics
Universiti Sains Malaysia (USM)
11800 USM
Penang, Malaysia
14 May 2015