Linux Bare Metal Recovery

As far as I can remember, I've no need to do Linux bare metal recovery.   Thanks to the flawless Linux OS running on stable hardware platform.  Nevertheless, there is an incident happened yesterday that a bare metal recovery is needed immediately.

Below are the (i) case and (ii) steps for recovery

Case

1. RHEL 5.5
2. IBM x3650 M3
3. 2 x Hard Disk Failures

Steps for recovery

1. Replace both Hard Disks
2. Re-create the RAID-1 on the Hard Disks
3. Kick start the same OS onto the repaired RAID-1 local hard disk
4. Mount the GRML image onto the iMM virtual media
5. Boot up the server using the image
6. Configure the network settings on the main interface
7. Bring up LVM2 system (/etc/init.d/lvm2 start)
8. Change the system root password (sudo passwd root)
9. Run rsync from another system to the repaired system to restore the backup copy
• backup-system#  rsync -avz --delete * <repaired-system>:/mnt
10. Reboot and do a health check

 Notes

1. If the ks.cfg is lost, then step 3 can be skipped.
2. If the ks.cfg is lost, extra step such as disk partitioning is required. 
3. For step 6, please watch out the order of the network interface in GRML image might be different from current server. Useful tip is to identify the correct interface using the MAC address.
4. For step 9, /mnt is mounted onto the root file system.

IP Address Change on ESX 4 host

1. Check current IP address
• esxcfg-vswif -l
2. Change to new IP address and netmask
• esxcfg-vswif -i <new IP address> -n <new netmask> vswif0
3. Change local hosts file with new IP
• vi /etc/hosts
4. Change to new default gateway
• vi /etc/sysconfig/network
5. Change DNS resolver settings (if applicable)
• vi /etc/resolv.conf
6. Change the vCenter agent configuration
• vi /etc/opt/vmware/vpxa/vpxa.cfg
7. Restart the network
• service network restart
8. Restart the Management Agent
• service mgmt-vmware restart
9. Restart the vCenter Agent
• service vmware-vpxa restart

Notes on vCenter upgrade from 4.0 to 4.1

This upgrade initiative is basically because it is found that Storage I/O Control is not available on ESX 4.0.  It is available on ESX 4.1 or later only.  However, vCenter 4.0 is not compatible with ESX 4.0 with the errors below.

Adding a ESX/ESXi 4.1 host to vCenter Server 4.0 fails with the error: interal error:vmodl.fault.hostCommunication

Notes about the vCenter upgrade from 4.0 to 4.1

1. The Guided Consolidation 4.0 extension has to be uninstalled before the vCenter upgrade.

2. The Update Manager 4.0 needs to be upgraded to 4.1 after the vCenter upgrade

3. The vCenter upgrade takes approximate 15 minutes for my installation with 2 ESX hosts.  The upgrade time largely depend on the size of the database

4. During the upgrade, the connection to vCenter will be disconnected but the current Virtual Machines and ESX hosts are still up and running

5. Make sure the account details to start up the vCenter Web Services are entered correctly or else the Web Services might fail to start up after the upgrade

Local Linux YUM repository for ISO image

The initiative of this setup is basically started when I find the PERL RPM package (needed by VMware Tools) is not installed on my Virtual Machine by default, which prevents me from continuing doing my VMware HA VM Monitoring Lab.

If you are sick of RPM installation inter-dependencies and are lazy (like me) to setup an apache instance for YUM, please continue to read. ;-)

• Mount the RHEL 6 ISO image

• # mkdir -p /mnt/iso/1
• # mount -o loop /dev/sr0 /mnt/iso/1
• # cd /mnt/iso/1/Packages
• # yum install createrepo-0.9.8-4.el6.noarch.rpm python-deltarpm-3.5-0.5.20090913git.el6.i686.rpm deltarpm-3.5-0.5.20090913git.el6.i686.rpm

• Create the repository

• # cd /mnt/iso; createrepo .
• # ......
• # ...... # Please take a cup of coffee
• # ......
• # yum clean all

• Populate the repo file
• # cd /etc/yum.repos.d
• # echo -e "[iso]\nbaseurl=file:///mnt/iso\nenabled=1" > iso.repo
• # yum install perl

• Import the public key for RPM

• Error shows when you run yum install perl
• Public key for perl-Pod-Simple-3.13-119.el6.i686.rpm is not installed
• Fix it as follows:
• cd /mnt/iso/1
• rpm --import RPM-GPG-KEY-redhat-release
• yum install perl