vCenter upgrade from 5.0 to 5.5 update 2

1. Verify vCenter 5.5 update 2 is supported on Windows 2008 SP2
2. Upgrade vCenter Database from SQL Server 2008 Standard - 64-bit to SQL Server 2008 Standard R2 SP2 - 64-bit
• Run SQL Server 2008 R2 Upgrade Advisor to check for the pre-requisites
• Upgrade to SQL Server 2008 Standard R2 - 64-bit
• Upgrade to SQL Server 2008 Standard R2 SP2 - 64-bit
3. Upgrade  vCenter Server from vCenter Server 5.0 GA to vCenter Server 5.5 update 2
• Run Host Agent Pre-Upgrade Checker for the pre-requisites
• Either can select Simple Install or Custom Install to install the components one at a time as follows:
1. vCenter Single Sign-On
• Prompt for the SSO administrator password
2. vSphere Web Client
• Prompt for the SSO administrator password entered in step 1
3. vCenter Inventory Service
• Prompt for the SSO administrator password entered in step 1
4. vCenter Server
• Prompt for the SSO administrator password entered in step 1
• Please note that the installation will prompt for 2 errors below
1. Warning 32014.A utility for phone home data collector couldn't be executed successfully. Please see its log file (with name PhoneHome) and vminst.log in system temporary folder for more details. Please note that you can also enable / disable this feature after installation through the application user interface.
2. The wizard was interrupted before VMware vCenter Server could be completely installed.
• The error is due to the 2 services below are incorrectly configured with ".\Administrator" as Log On As. They should be fixed as "Local System".  It is possibly a bug in vCenter 5.5 update 2 when upgrade.
1. VMware VirtualCenter Server
2. VMware VirtualCenter ManagementServices
• Once the issue is fixed, fix the vCenter installation by following the procedure - kb.vmware.com/kb/2017147 like this:
1. Click Start > Control Panel.
2. Click Programs and Features.
3. From the list of installed software, right-click VMware vCenter Server and click Change.
4. Last but not least, please remember to take VM snapshots at each checkpoints for quick rollback

Checksum type used in repomd.xml and for packages in the metadata between RHEL5 and RHEL6

• If you need to create a YUM repository on RHEL6 and need to provide the YUM repository service to RHEL5 clients, you need to create the repository using checksum type of sha because the of the differences below and RHEL5 clients cannot recognize sha256 checksum.

1. RHEL5 : The older default was "sha"
2. RHEL6 : The default is now "sha256".

•  Failure to do so will cause the error message below when you are accessing the YUM repository from a RHEL5 client with command like : "yum list"

1. Error Message : primary.xml.gz: [Errno -3] Error performing checksum

• Solution is as below:

1. createrepo -s sha

Reset root password on an ESXi 4.1 host

According to VMware KB http://kb.vmware.com/kb/1317898, root password cannot be reset without re-install.  Below is the a trick to reset the root password on an ESXi 4.1 without re-install.

1. Boot the ESXi host with a Linux boot-able media (e.g. DVD)
2. Mount the ESXi host partition containing the system data
• mkdir -p /tmp/mnt
• mount /dev/sda5 /tmp/mnt
3. Extract the shadow file of the ESXi installation and reset password
• mkdir -p /tmp/workdir
• cd /tmp/mnt
• cp state.tgz /tmp/workdir
• zcat state.tgz | tar xvf -
• zcat local.tgz | tar xvf -
• cd etc
• vi shadow "Reset the root password"
• cd ..
• tar zcvf local.tgz etc
• tar zcvf state.tgz local.tgz
• cp state.tgz /tmp/mnt
• sync; sync; sync
• umount /tmp/mnt 
4. Reboot the box

 

PXE boot setup for ESXi 5 installation

Hardware setup

1. VMware Workstation 8.0.4 with a virtual network bridged to external network.
• The virtual network needs to be bridged and cannot be Host-only or else the PXE boot does not work for the TFTP
2. Virtual Machine with 4 vCPU (2 dual-core processors), 4 GB memory and 60 GB virtual disk running RHEL 6.3 for PXE server
3. Virtual Machine with 2 vCPU (2 single-core processors), 2 GB memory and 40 GB virtual disk for ESXi 5 installation

Software setup

1. syslinux 3.86
• ESXi 5 PXE boot installation only works with syslinux 3.86. syslinux 4.0.6 is not compatible, for example, APPEND -c boot.cfg code cannot be recognized on the PXE configuration file
2. tftp-server 0.49
3. dhcp 4.1.1
4. httpd 2.2.15
5. VMware Workstation 8.0.4

Software installation and configuration of PXE server

1. tftp-server
• yum install tftp-server
• sed -e 's/\(\s\*\)disable\(\s\*\)= yes/\1disable\2= no/' -i /etc/xinetd/tftp
• service xinetd start
• cd /; ln -s /var/lib/tftpboot
2. dhcp
• yum install dhcp
• cd /etc/dhcp
• cp -rp dhcpd.conf dhcpd.conf.`date +%Y%m%d`
• Update dhcpd.conf as shown on dhcpd.conf in references section
• cd /etc/sysconfig
• cp -rp dhcpd dhcpd.`date +%Y%m%d`
• sed -e 's/DHCPDARGS=$/DHCPDARGS=eth1/' -i dhcpd
• service dhcpd start 
3. syslinux
• yum install perl-Crypt-PasswdMD5-1.3-6.el6.noarch -y
• yum install perl-Digest-SHA1-2.12-2.el6.x86_64 -y
• wget ftp://ftp.ntua.gr/pub/linux/scientificlinux/6.0/x86_64/os/Packages/syslinux-3.86-1.1.el6.x86_64.rpm
• rpm -ivh syslinux-3.86-1.1.el6.x86_64.rpm
• cp -p /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot
• mkdir -p /tftpboot/pxelinux.cfg
• Put the PXE configuration file as 01-mac_address_of_target_ESXi_host under the directory /tftpboot/pxelinux.cfg.  Sample is shown in references section.
• Note: The MAC address has to be in lower case hexadecimal characters
4. httpd
• httpd is installed by default on RHEL6 and so no need to install the package
• chkconfig httpd on; service httpd on
• cd /var/www/html ; mkdir -p /var/www/html/esx5i_ksFiles
• Update ks.cfg as shown on ks.cfg in references section
5. Firewall
•  Run system-config-firewall-tui to enable rule to allow tftp and http incoming traffic
6. ESXi 5 image repository
• On the PXE server VM, connect the CDROM to the ESXi 5 ISO image
• mount /dev/sr0 /mnt
• cd /mnt
• tar cvf - . | (cd /tftpboot; tar xvf -)
• Update /tftpboot/boot.cfg to indicate the location of the kickstart configuration file ks.cfg as follows.
• sed -e 's/\(kernelopt=runweasel\)/\1 ks=http:\/\/192.168.88.1\/esx5i_ksFiles\/ks.cfg/' -i boot.cfg

Next steps

1. Customize the kickstart configuration file ks.cfg for post installation setup.
2. Implementation of ESXi 5 auto deploy feature

References

• dhcpd.conf
• option domain-name "wolfliar.local";
• option domain-name-servers dc1.wolfliar.local;
• default-lease-time 600;
• max-lease-time 7200;
• ddns-update-style none;
• allow booting;
• allow bootp;
• authoritative;
• log-facility local7;
• class "pxeclients" {
•   match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
•   next-server 192.168.88.1;
•   filename "pxelinux.0";
• }
• subnet 192.168.88.0 netmask 255.255.255.0 {
•   range 192.168.88.100 192.168.88.149;
• }

• 01-mac_address_of_target_ESXi_host
• DEFAULT menu.c32 
• NOHALT 1
• PROMPT 0
• TIMEOUT 80
• LABEL install
•   KERNEL mboot.c32
•   APPEND -c boot.cfg
•   MENU LABEL ESXi-5.0.0-469512-standard ^Installer
• LABEL hddboot
•   LOCALBOOT 0x80
•   MENU LABEL ^Boot from local disk
•  
• MENU TITLE ESXi-5.0.0-469512-standard Boot Menu

• ks.cfg
• #
• # Sample scripted installation file
• #
• # Accept the VMware End User License Agreement
• vmaccepteula
•  
• # Set the root password for the DCUI and Tech Support Mode
• rootpw mypassword
•  
• # Install on the first local disk available on machine
• install --firstdisk --overwritevmfs
•  
• # Set the network to DHCP on the first network adapter
• network --bootproto=dhcp --device=vmnic0
•  
• # A sample post-install script
• %post --interpreter=python --ignorefailure=true
• import time
• stampFile = open('/finished.stamp', mode='w')
• stampFile.write( time.asctime() )
•  
• %firstboot --interpreter=busybox
• # Remove the uplink to the PXE boot environment from the 
• primary vSwitch
• esxcfg-vswitch -U vmnic4 vSwitch0
   

Linux Bare Metal Recovery

As far as I can remember, I've no need to do Linux bare metal recovery.   Thanks to the flawless Linux OS running on stable hardware platform.  Nevertheless, there is an incident happened yesterday that a bare metal recovery is needed immediately.

Below are the (i) case and (ii) steps for recovery

Case

1. RHEL 5.5
2. IBM x3650 M3
3. 2 x Hard Disk Failures

Steps for recovery

1. Replace both Hard Disks
2. Re-create the RAID-1 on the Hard Disks
3. Kick start the same OS onto the repaired RAID-1 local hard disk
4. Mount the GRML image onto the iMM virtual media
5. Boot up the server using the image
6. Configure the network settings on the main interface
7. Bring up LVM2 system (/etc/init.d/lvm2 start)
8. Change the system root password (sudo passwd root)
9. Run rsync from another system to the repaired system to restore the backup copy
• backup-system#  rsync -avz --delete * <repaired-system>:/mnt
10. Reboot and do a health check

 Notes

1. If the ks.cfg is lost, then step 3 can be skipped.
2. If the ks.cfg is lost, extra step such as disk partitioning is required. 
3. For step 6, please watch out the order of the network interface in GRML image might be different from current server. Useful tip is to identify the correct interface using the MAC address.
4. For step 9, /mnt is mounted onto the root file system.

IP Address Change on ESX 4 host

1. Check current IP address
• esxcfg-vswif -l
2. Change to new IP address and netmask
• esxcfg-vswif -i <new IP address> -n <new netmask> vswif0
3. Change local hosts file with new IP
• vi /etc/hosts
4. Change to new default gateway
• vi /etc/sysconfig/network
5. Change DNS resolver settings (if applicable)
• vi /etc/resolv.conf
6. Change the vCenter agent configuration
• vi /etc/opt/vmware/vpxa/vpxa.cfg
7. Restart the network
• service network restart
8. Restart the Management Agent
• service mgmt-vmware restart
9. Restart the vCenter Agent
• service vmware-vpxa restart

Notes on vCenter upgrade from 4.0 to 4.1

This upgrade initiative is basically because it is found that Storage I/O Control is not available on ESX 4.0.  It is available on ESX 4.1 or later only.  However, vCenter 4.0 is not compatible with ESX 4.0 with the errors below.

Adding a ESX/ESXi 4.1 host to vCenter Server 4.0 fails with the error: interal error:vmodl.fault.hostCommunication

Notes about the vCenter upgrade from 4.0 to 4.1

1. The Guided Consolidation 4.0 extension has to be uninstalled before the vCenter upgrade.

2. The Update Manager 4.0 needs to be upgraded to 4.1 after the vCenter upgrade

3. The vCenter upgrade takes approximate 15 minutes for my installation with 2 ESX hosts.  The upgrade time largely depend on the size of the database

4. During the upgrade, the connection to vCenter will be disconnected but the current Virtual Machines and ESX hosts are still up and running

5. Make sure the account details to start up the vCenter Web Services are entered correctly or else the Web Services might fail to start up after the upgrade