Reset root password on an ESXi 4.1 host

According to VMware KB http://kb.vmware.com/kb/1317898, root password cannot be reset without re-install.  Below is the a trick to reset the root password on an ESXi 4.1 without re-install.

1. Boot the ESXi host with a Linux boot-able media (e.g. DVD)
2. Mount the ESXi host partition containing the system data
• mkdir -p /tmp/mnt
• mount /dev/sda5 /tmp/mnt
3. Extract the shadow file of the ESXi installation and reset password
• mkdir -p /tmp/workdir
• cd /tmp/mnt
• cp state.tgz /tmp/workdir
• zcat state.tgz | tar xvf -
• zcat local.tgz | tar xvf -
• cd etc
• vi shadow "Reset the root password"
• cd ..
• tar zcvf local.tgz etc
• tar zcvf state.tgz local.tgz
• cp state.tgz /tmp/mnt
• sync; sync; sync
• umount /tmp/mnt 
4. Reboot the box

 

PXE boot setup for ESXi 5 installation

Hardware setup

1. VMware Workstation 8.0.4 with a virtual network bridged to external network.
• The virtual network needs to be bridged and cannot be Host-only or else the PXE boot does not work for the TFTP
2. Virtual Machine with 4 vCPU (2 dual-core processors), 4 GB memory and 60 GB virtual disk running RHEL 6.3 for PXE server
3. Virtual Machine with 2 vCPU (2 single-core processors), 2 GB memory and 40 GB virtual disk for ESXi 5 installation

Software setup

1. syslinux 3.86
• ESXi 5 PXE boot installation only works with syslinux 3.86. syslinux 4.0.6 is not compatible, for example, APPEND -c boot.cfg code cannot be recognized on the PXE configuration file
2. tftp-server 0.49
3. dhcp 4.1.1
4. httpd 2.2.15
5. VMware Workstation 8.0.4

Software installation and configuration of PXE server

1. tftp-server
• yum install tftp-server
• sed -e 's/\(\s\*\)disable\(\s\*\)= yes/\1disable\2= no/' -i /etc/xinetd/tftp
• service xinetd start
• cd /; ln -s /var/lib/tftpboot
2. dhcp
• yum install dhcp
• cd /etc/dhcp
• cp -rp dhcpd.conf dhcpd.conf.`date +%Y%m%d`
• Update dhcpd.conf as shown on dhcpd.conf in references section
• cd /etc/sysconfig
• cp -rp dhcpd dhcpd.`date +%Y%m%d`
• sed -e 's/DHCPDARGS=$/DHCPDARGS=eth1/' -i dhcpd
• service dhcpd start 
3. syslinux
• yum install perl-Crypt-PasswdMD5-1.3-6.el6.noarch -y
• yum install perl-Digest-SHA1-2.12-2.el6.x86_64 -y
• wget ftp://ftp.ntua.gr/pub/linux/scientificlinux/6.0/x86_64/os/Packages/syslinux-3.86-1.1.el6.x86_64.rpm
• rpm -ivh syslinux-3.86-1.1.el6.x86_64.rpm
• cp -p /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot
• mkdir -p /tftpboot/pxelinux.cfg
• Put the PXE configuration file as 01-mac_address_of_target_ESXi_host under the directory /tftpboot/pxelinux.cfg.  Sample is shown in references section.
• Note: The MAC address has to be in lower case hexadecimal characters
4. httpd
• httpd is installed by default on RHEL6 and so no need to install the package
• chkconfig httpd on; service httpd on
• cd /var/www/html ; mkdir -p /var/www/html/esx5i_ksFiles
• Update ks.cfg as shown on ks.cfg in references section
5. Firewall
•  Run system-config-firewall-tui to enable rule to allow tftp and http incoming traffic
6. ESXi 5 image repository
• On the PXE server VM, connect the CDROM to the ESXi 5 ISO image
• mount /dev/sr0 /mnt
• cd /mnt
• tar cvf - . | (cd /tftpboot; tar xvf -)
• Update /tftpboot/boot.cfg to indicate the location of the kickstart configuration file ks.cfg as follows.
• sed -e 's/\(kernelopt=runweasel\)/\1 ks=http:\/\/192.168.88.1\/esx5i_ksFiles\/ks.cfg/' -i boot.cfg

Next steps

1. Customize the kickstart configuration file ks.cfg for post installation setup.
2. Implementation of ESXi 5 auto deploy feature

References

• dhcpd.conf
• option domain-name "wolfliar.local";
• option domain-name-servers dc1.wolfliar.local;
• default-lease-time 600;
• max-lease-time 7200;
• ddns-update-style none;
• allow booting;
• allow bootp;
• authoritative;
• log-facility local7;
• class "pxeclients" {
•   match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
•   next-server 192.168.88.1;
•   filename "pxelinux.0";
• }
• subnet 192.168.88.0 netmask 255.255.255.0 {
•   range 192.168.88.100 192.168.88.149;
• }

• 01-mac_address_of_target_ESXi_host
• DEFAULT menu.c32 
• NOHALT 1
• PROMPT 0
• TIMEOUT 80
• LABEL install
•   KERNEL mboot.c32
•   APPEND -c boot.cfg
•   MENU LABEL ESXi-5.0.0-469512-standard ^Installer
• LABEL hddboot
•   LOCALBOOT 0x80
•   MENU LABEL ^Boot from local disk
•  
• MENU TITLE ESXi-5.0.0-469512-standard Boot Menu

• ks.cfg
• #
• # Sample scripted installation file
• #
• # Accept the VMware End User License Agreement
• vmaccepteula
•  
• # Set the root password for the DCUI and Tech Support Mode
• rootpw mypassword
•  
• # Install on the first local disk available on machine
• install --firstdisk --overwritevmfs
•  
• # Set the network to DHCP on the first network adapter
• network --bootproto=dhcp --device=vmnic0
•  
• # A sample post-install script
• %post --interpreter=python --ignorefailure=true
• import time
• stampFile = open('/finished.stamp', mode='w')
• stampFile.write( time.asctime() )
•  
• %firstboot --interpreter=busybox
• # Remove the uplink to the PXE boot environment from the 
• primary vSwitch
• esxcfg-vswitch -U vmnic4 vSwitch0
   

Linux Bare Metal Recovery

As far as I can remember, I've no need to do Linux bare metal recovery.   Thanks to the flawless Linux OS running on stable hardware platform.  Nevertheless, there is an incident happened yesterday that a bare metal recovery is needed immediately.

Below are the (i) case and (ii) steps for recovery

Case

1. RHEL 5.5
2. IBM x3650 M3
3. 2 x Hard Disk Failures

Steps for recovery

1. Replace both Hard Disks
2. Re-create the RAID-1 on the Hard Disks
3. Kick start the same OS onto the repaired RAID-1 local hard disk
4. Mount the GRML image onto the iMM virtual media
5. Boot up the server using the image
6. Configure the network settings on the main interface
7. Bring up LVM2 system (/etc/init.d/lvm2 start)
8. Change the system root password (sudo passwd root)
9. Run rsync from another system to the repaired system to restore the backup copy
• backup-system#  rsync -avz --delete * <repaired-system>:/mnt
10. Reboot and do a health check

 Notes

1. If the ks.cfg is lost, then step 3 can be skipped.
2. If the ks.cfg is lost, extra step such as disk partitioning is required. 
3. For step 6, please watch out the order of the network interface in GRML image might be different from current server. Useful tip is to identify the correct interface using the MAC address.
4. For step 9, /mnt is mounted onto the root file system.

IP Address Change on ESX 4 host

1. Check current IP address
• esxcfg-vswif -l
2. Change to new IP address and netmask
• esxcfg-vswif -i <new IP address> -n <new netmask> vswif0
3. Change local hosts file with new IP
• vi /etc/hosts
4. Change to new default gateway
• vi /etc/sysconfig/network
5. Change DNS resolver settings (if applicable)
• vi /etc/resolv.conf
6. Change the vCenter agent configuration
• vi /etc/opt/vmware/vpxa/vpxa.cfg
7. Restart the network
• service network restart
8. Restart the Management Agent
• service mgmt-vmware restart
9. Restart the vCenter Agent
• service vmware-vpxa restart

Notes on vCenter upgrade from 4.0 to 4.1

This upgrade initiative is basically because it is found that Storage I/O Control is not available on ESX 4.0.  It is available on ESX 4.1 or later only.  However, vCenter 4.0 is not compatible with ESX 4.0 with the errors below.

Adding a ESX/ESXi 4.1 host to vCenter Server 4.0 fails with the error: interal error:vmodl.fault.hostCommunication

Notes about the vCenter upgrade from 4.0 to 4.1

1. The Guided Consolidation 4.0 extension has to be uninstalled before the vCenter upgrade.

2. The Update Manager 4.0 needs to be upgraded to 4.1 after the vCenter upgrade

3. The vCenter upgrade takes approximate 15 minutes for my installation with 2 ESX hosts.  The upgrade time largely depend on the size of the database

4. During the upgrade, the connection to vCenter will be disconnected but the current Virtual Machines and ESX hosts are still up and running

5. Make sure the account details to start up the vCenter Web Services are entered correctly or else the Web Services might fail to start up after the upgrade

Local Linux YUM repository for ISO image

The initiative of this setup is basically started when I find the PERL RPM package (needed by VMware Tools) is not installed on my Virtual Machine by default, which prevents me from continuing doing my VMware HA VM Monitoring Lab.

If you are sick of RPM installation inter-dependencies and are lazy (like me) to setup an apache instance for YUM, please continue to read. ;-)

• Mount the RHEL 6 ISO image

• # mkdir -p /mnt/iso/1
• # mount -o loop /dev/sr0 /mnt/iso/1
• # cd /mnt/iso/1/Packages
• # yum install createrepo-0.9.8-4.el6.noarch.rpm python-deltarpm-3.5-0.5.20090913git.el6.i686.rpm deltarpm-3.5-0.5.20090913git.el6.i686.rpm

• Create the repository

• # cd /mnt/iso; createrepo .
• # ......
• # ...... # Please take a cup of coffee
• # ......
• # yum clean all

• Populate the repo file
• # cd /etc/yum.repos.d
• # echo -e "[iso]\nbaseurl=file:///mnt/iso\nenabled=1" > iso.repo
• # yum install perl

• Import the public key for RPM

• Error shows when you run yum install perl
• Public key for perl-Pod-Simple-3.13-119.el6.i686.rpm is not installed
• Fix it as follows:
• cd /mnt/iso/1
• rpm --import RPM-GPG-KEY-redhat-release
• yum install perl