BUILDING YOU OWN CENTOS COMPUTER CLUSTER This document contains the instruction to build a cluster of PC running CENTOS Linux system that has network sharing file (NSF). The modus operandi of the cluster mimics that of the Rocks Cluster (see http://www.rocksclusters.org/). The cluster contains a frontend node pluse a number of compute nodes. All nodes are connected via a 100/1000 MB/s switch. In principle this document can also be applied on other Linux OS with or without modification. The 'cluster' is not one that is defined in a strict sense (like the Rocks Cluster), as our's much less in complexity. If you want to use this manual to set up your own CENTOS cluster in your institution, you need to modify only the following information which is network- and institution-specific. The default values used in this document are as below. They probably would not work if you do not specify your own DOMAIN, IPADDR, GATEWAY, DNS1 and DNS2. Ask your network admin for these information. HOSTNAME=chakra IPADDR=10.205.18.133 DOMAIN=local usm.my GATEWAY=10.205.19.254 DNS1=10.202.1.27 DNS2=10.202.1.6 Hardwar requirement: Two hardisks on frontend (one small capacity and the other large, e.g., 500 GiB + 1 TiB). One frontend PC + at least one compute node PC; LAN cables, a 100/1000 MB/s switch. The frontend node has to be equipped with two LAN cards, a built-in one and an externally plug-in one. All compute nodes must be equipped with a minimum of one LAN card (either built-in or external). Naming convention: We shall name these PCs respectively: $HOSTNAME (frontend node), node1, node2, node3, .... For the frontend node, we shall denote the built-in LAN card eth0 while the external LAN card eth1. Important IPs to take note: The IP for the frontend at eth0 is by default set to: 192.168.1.10 The IPs for node1, node2, node3 etc. at their respective eth0 are by default set to (in sequential order): 192.168.1.21, 192.168.1.22, 192.168.1.23, ... To build up a CentOS cluster, follow the following procedure step-by-step in sequential order (0) Use Rufus or other software to burn the lastest CentOS iso into a bootable thumb drive. This manuscript is prepared based on CentOS-7-x86_64-DVD-1804, but in principle it should work also for other version of CENTOS (version 6 or above). (1) Connect all eth0 ports of all the PCs in the cluster to a 100/1000 MB/s switch. eth1 of the frontend is to be connected to the internet network (e.g., in USM, it is the usm.my network). (2) Install CENTOS using the bootable thumb into the frontend's smaller hardisk. Leave the larger hardisk as it for the moment. Install CENTOS in all other compute nodes. All nodes (including the frontend) should use the installation option of 'Development and Creative Workstation'. Choose to add all software packages offered at the installation prompt. Use a common root password for the frontend as well as all compute nodes. (2.5) Check out the label for the hard disks in the frontend using fdisk -l. Say the larger hard disk is labelled /dev/sdb. Mount this hard disk in the folder /export in the root directory in the frontend CENTOSby typing the followinng command line (as us) in the terminal: mkdir /export chmod -R 777 /export mount -t xfs /dev/sdb /export cp /etc/fstab /etc/fstab.orig Mount the 1 TiB hardisk permanantlty by adding the line '/dev/sdb /export xfs rw 2 2' to /etc/fstab. This permanant mount of /dev/sdb to /export will take place after a reboot In case the 1 TiB hard disk is not formatted properly, it may refuse to be mounted. The hard disk can be formatted using 'mkfs.xfs -f /dev/sdb' to forcefully format it into XFS format. The task of formatting and mouting the external hard disk to the Centos root directory can be executed by running the script sh mount_export.txt The script mount_export.txt is downloadable from http://comsics.usm.my/tlyoon/configrepo/howto/customise_centos/Centos_cluster/ (3) Install CENTOS in all nodes using the same option as that used in setting up the frontend. (4) wget http://comsics.usm.my/tlyoon/configrepo/howto/customise_centos/Centos_cluster/centos_frontend-1of2.txt. Run sh centos_frontend-1of2.txt in the frontend as su. No internet connection is established at this stage. centos_frontend-1of2.txt will do the necessary preparatory configuration for the frontned. These include (i) setting the SELINUX to permissive in /etc/sysconfig/selinux, (ii) activating sshd.service so that ssh can work, and (iii) setting the IP of the frontend by making changes to its network configuration such as assigning IP address, setting DNS server, etc. (The instruction is based on http://www.techkaki.com/2011/08/how-to-configure-static-ip-address-on-centos-6/). The frontend will reboot at the end of centos6_frontend1of2. (5) After rebooting of frontend, log in as root. Check the frontend to see if the following functions are configured successfully by centos_frontend-1of2.txt. (a) Check the mode of SELINUX using the two command getenforce sestatus The SELINUX should be set to permissive. This can be confirmed manually by checking if the /etc/sysconfig/selinux has a line stating 'SELINUX=permissive'. (b) Check manually that ssh is enabled. (c) Check manually if both eth0 and eth1 are connected to the switch and internet respectively. This can be done by issuing the following commands one-by-one (i) ping google.com (ii) ssh into the frontend from a third party terminal (ii) ifconfig (iii) cat /etc/sysconfig/network-scripts/ifcfg-$nc (iv) cat /etc/sysconfig/network-scripts/ifcfg-$nc1 where nc0=$(lshw -class network | grep -E 'Ethernet interface|logical name' | grep 'logical name' | awk '{print $3}' | awk '!/vir/' | tail -n1) nc1=$(lshw -class network | grep -E 'Ethernet interface|logical name' | grep 'logical name' | awk '{print $3}' | awk '!/vir/' | awk 'NR==1{print}') (6) Check if the HWADDR for both $nc0 and $nc1 are explicitly specified. Assure that (a) DOMAIN=local, DNS1=127.0.0.1 for $nc0 (b) DOMAIN=usm.my, DNS1=10.202.1.27, DNS2=10.202.1.6 for $nc1 The setting of the network configuration required (as stated above) can be manually configured, if needed, by tweaking the NetworkManager sitting in the upper right hand side of the screen. In case the NetworkManager icon fails to appear, type 'systemctl NetworkManager restart'. (7) Alternatively, the setting of the network configuration required (as stated above) can be manually configured, if needed, by editing the files /etc/sysconfig/network-scripts/ifcfg-$nc0 /etc/sysconfig/network-scripts/ifcfg-$nc1 (8) In anycase, both network cards must be active and connected before proceeding to the next step. Reboot the frontend if you have done some manual configurations. Often both cards will be connected after rebooting. (9) Run centos_frontend-2of2.txt in the frontend as root. It will reboot at the end of the script. All the steps described until this stage must be completed before initiating the following steps. (10) Configure the compute nodes by running the script centos_node-1of2.txt in each node separately. Make sure that each of the cmopute node is connected to the swtich via the existing LAN port of the compute node. You must manually edit the correct value for IPADDR=192.168.1.XX in centos_node-1of2.txt, for XX=21,22,23... centos_node-1of2.txt is downloadable from http://comsics.usm.my/tlyoon/configrepo/howto/customise_centos/Centos_cluster/. You may have to save centos_node-1of2.txt into a pendrive and plug it into each node manually as at the present stage the nodes may be still unable to go online. (11) After rebooting a node, log in again as root to execute the file centos_node-2of2.txt. (12) To add any new user, run useradd_recursive.txt from the frontend as su (download from http://comsics.usm.my/tlyoon/configrepo/howto/customise_centos/Centos_cluster/useradd_recursive.txt). (13) After any new user has been added using useradd_recursive.txt, it must be followed immediately by manually executing the script pwlssh_recursive.txt as the new user from the frontend. This script will render all nodes and the frontend to be connected via passwordless ssh for the user who executed the script. (pwlssh_recursive.txt is available from http://comsics.usm.my/tlyoon/configrepo/howto/customise_centos/Centos_cluster/pwlssh_recursive.txt). Yoon Tiem Leong Universiti Sains Malaysia 11800 USM Penang, Malaysia Date: 16 May 2019