I. OS
We installed RedHat 8.0, mainly because we planned to run a production software that is supported on RedHat. Installation was very easy. For anybody with some experience only Custom installation is what I recommend. The following should be installed: Ethernet and Internet support, rsh/ssh/ftp including corresponding daemons; NIS and NFS. X-Windows is probably not needed for most scientific users, although it is probably worth of having to run diagnostics and some system utilities for setting the machines up. We installed OS onto two machines, one for the front end machine ("master") with lots of packages and much smaller set for the rest (slaves).

II. Cloning
We chose to clone the slaves rather than use BOOTP. For this an external "cloning machine" had to be used because Dell 4500C has no space for two HDDs. There is a nice reference on cloning, so we did as it said, except that MBR did not seem to be copied with only 446 bytes, so we copied 64K. We did not use "nc" because Dell 4500C comes without a floppy drive and we did not bother with creating a bootable CDROM and using "nc" (perhaps it would have been easier). So we took the installed disk, connected it as secondary IDE slave (hdd) and attached a fresh disk as secondary IDE master (hdc) (Western Digital was a bit mean in not putting jumper info on the disks so it had to be found on their homepage, thanks WD at least for that). hdc had to be changed 22 times (I scratched my fingers and lost some blood when doing it). Finally, /etc/fstab had an entry LABEL=/work rather then /dev/hda2. This means /work was also a part of the extended partition. After cloning /work did not seem to work even after being formatted, so I simply replaced LABEL=/work by /dev/hda2 and there was no further problem.
How to clone two identical disks

• 1. dd if=/dev/hdd of=/dev/hdc bs=65536 count=1
• 2. reboot to reread MBR (I don't know how to reread without it)
• 3. dd if=/dev/hdd1 of=/dev/hdc1 (copy /)
• 4. dd if=/dev/hdd4 of=/dev/hdc4 (copy external partition table)
• 5. reboot to reread the external partition table
• 6. dd if=/dev/hdd5 of=/dev/hdc5 (/usr that was the extended partition)

• mkswap /dev/hda3 (1G swap)
• mkfs -t ext3 /dev/hda2 (create /work area)

III. System set-up (some part can be done before cloning and the rest individually).

• Ethernet:
  /etc/resolve.conf
  /etc/hosts, /etc/hosts.allow, /etc/hosts.deny
  These were set to allow any IP connexion between cluster nodes and only ssh from outside.
  /etc/sysconfig/network, /etc/sysconfig/network-scripts/ifcfg-eth0
  On the master /etc/sysconfig/network-scripts/ifcfg-eth1 had to be set for the second card. (Stupidly it took me a while to figure out which card is eth0 and which is eth1, system response was rather weird with the wrong choice). 10/100 card was hardwired to a fixed IP obtained at the local network office. All Gigabit cards were set to 192.168.1.11,192.168.1.12 etc addresses. In /etc/hosts the fixed IP and the local IP belonging to two different Ethernet cards on the same master machine were given two different alpha (that is, text) addresses, so that production parallel software would run only on Gigabit. The switch itself needed no set up.
• Passwords and user files
  NIS was loaded during the Custom isntallation and it needed only
  /usr/lib/yp/ypinit -m
  (Do not forget to do "cd /var/yp; make" if more users are added later). NFS was also loaded during the installation, it needed to have /etc/exports edited and "/etc/rc.d/init.d/nfs restart" to reset it. Beside the user directories, /usr from the master was also mounted on all slaves.
• Finally, by running "setup" I removed many daemons to have faster reboots (lpd, kudzu, sendmail, xfs) and more memory. kudzu would have to be reactivated if any hardware is added (I kept it on the master).
• If you plan to do remote power management, set it up BEFORE cloning.
• A brief note on UPSing: I have specifically looked for a UPS that is supported on Linux. Of these I found two of nearly same price and specs. I chose Omron and so far have not regretted my choice. Installation was as easy as running an RPM provided by Omron on a CDROM. UPS connects to the system by a 9-pin cable through a COM port. Whenever power goes off, a daemon (the one installed with RPM) runs whatever you say. Usually you run poweroff command in 30 seconds after power went down. During the next roughly 5.5 min your computer should shut itself down. I connected only the front-end machine and the Gigabit switch (that costs as much as two slave computers). I also let the external Ethernet cable pass through the UPS to avoid possible power surge.

IV. What did not work or misbehaved

USB mouse (labelled Dell by Logitech; a USB optical 3-button mouse) behaved strangely: it would work at first and then it would copy garbage, and finally stop being recognised. This may have been in part due to trying to install X-Windows but system reboots did not clear it. Magically it came back to life later. X-Windows I could not set up. Most likely the monitor is to blame, although the USB mouse may have something to do with that too. I do not think graphics adapter with its annoyingly large video memory (64M) gave any problem, though. Graphical installation of RedHat had no problem with graphics and mouse too. At some point xinetd on the master would not let any IP contact from the slaves. (Connexion refused). I am strongly convinced that restoring the /etc/hosts* files after I slightly changed them to what worked did not cure the problem. It seems that "setup" did something weird. I rerun and rebooted the machine many a time that day and somehow it went the way I wanted: rsh from slaves to master resumed working (the opposite never gave any problem; perhaps due to the fact I was messing with master's network files).

The default shell on all slaves is bash no matter what you do with chsh command. The same trouble was on another SUSE cluster here so it may be a general problem of NIS in Linux. On the master one can choose any shell. Possible solution: BEFORE CLONING, log in as each joe user and change the shell (not tried yet).

Note on mounting master's /usr on slaves as /usr/global. It worked quite fine. Most programs installed in master:/usr work on slaves right away. Some don't because they have some directories hardwired inside of the scripts. Possible solution is to do something like ln -s /usr/global/local/software /usr/local/software if it wants to be found in /usr/local/software. This has to be executed on each slave and since it requires root it takes time and is somewhat bothersome. A better solution: do it once ahead of time BEFORE CLONING (think WISELY what software you may need). Warning: in some cases running master's software on slaves would slow down the network. If you think/know this might be the case consider installing the software on slaves directly.

Funny thing happened with the additional 11 nodes. They had an on-board FastEthernet and an added-on PCI Gigabit card. Those were found by kudzu and labelled eth0 (Giga) and eth1(Fast). The interesting thing about multicards is that the last interface seems to be the default one. This means if you connect the nodes by Gigabit they cannot reach each other if you do rsh, because they think you are using FastEthernet. The solution is to switch around eth0 and eth1. This is easy, change in /etc/modules.conf
alias eth0 e100
alias eth1 e1000
into
alias eth0 e1000
alias eth1 e100
AND change /etc/sysconfig/network-scripts/ifcfg-eth?. I just realised that alternatively, one just changes GATEWAY in /etc/sysconfig/network (? not tried).

Something mysterious happenned when cloning 20 new machines that came with Seagate Barracuda ST340016A disks instead of previously used Western Digital disks. The latter ones were cloned without a trifle of trouble. Seagate disks could only be copied when nothing else was running on the machine, such as X Windows. Running dd from X Windows would lead to core dumps after a while and in other cases dd would just freeze. If one opened a couple of text-based windows with Alt-F? (without X Windows) then the process of cloning would become slower several times. I have no explanation for this, it looks like a bug in either Linux or firmware drivers or a bug in dd command. However, if left alone, Seagate disks were copied fine. To be precise there is one difference in copying WD and Seagate disks. WD disks had an extended partition, that had to be cloned by rebooting several times. Seagate disks were made to have no extended partition, with 4 normal ones: /, /work, /usr and swap. It is not clear if this has anything to do with dd failures. Seagate disks were extremely silent compared to noisy WD disks.

A trouble occurred with installing OS on the Seagate disks. I did not want to reinstall the whole system just because I cannot clone WD->Seagate (slightly different geometry). My plan was to copy / and /usr systems with dd because they had the same size and then install GRUB to MBR with:
root (hd2,0)
find /boot/grub/stage
setup (hd2)
(These are commands typed in after executing "grub"). This seemed to work but the disk would not boot (even GRUB would not boot). I could not use the recommended way of installing GRUB from a floppy since there is no floppy and an extrenal USB floppy drive would not be booted from. This having failed I did install the most minimal Linux from CDROM for the sole purpose of installing GRUB. After that I recopied /usr and / (I think I did cp -av /mnt/dsk1 /mnt/dsk2 for both after mounting what needed). Thus I made a master Seagate disk to be cloned. Interestingly, cp seemed faster than dd, although no exact check was done, perhaps because /usr uses only 1/3 of its space and dd copies everything.