Cisco UCS – Part 1 – Getting Start with Implementation of New Cisco UCS Domain

When we are starting with implementing new Cisco UCS Domain environment, there are few steps that need to be follow to complete installation and configuration part. Steps are mentioned in below table. We will discuss the same in detailed information in coming articles.

Step 1 Console Setup
Step 2 Configure Administration Policies
Step 3 Configure Equipment Policies
Step 4 Configure Unified Ports
Step 5 Configure Fabric Interconnect Server Ports
Step 6 Configure LAN Connectivity
Step 7 Configure SAN Connectivity
Step 8 Define Workloads

There is reference article from Cisco for your reference. Click here to redirect.

305200

Image Credit: Cisco

How to Upgrade from Windows vCenter 5.5 to VCSA 6.5 including SRM

In last couple of days, I heard this question many times for below scenario. Since 5.5 is about to expire this month, so IT admins are upgrading their environment to new version of vSphere. Though going directly to 6.7 is something that do not meet N-1 requirement for most of the environment. Hence most of the admin prefers to put environment on vSphere 6.5.

Very first question everyone think about is –

  • Does upgradation path from vCenter 5.5 to VCSA 6.5 is supported? Answer is YES.
  • Does upgradation path from VMware SRM 5.8.1 to 6.5 is supported? Answer is NO. You need to upgrade VMware SRM 5.8.1 to 6.0 or 6.1.2, and then upgrade SRM 6.0 to 6.5.

Here we have details of existing environment and requirement which need to meet.

Existing Environment: –

  • Windows based vCenter Server – Version 5.5
  • Site Recovery Manager – Version 5.8
  • Replication Type in SRM – vSphere Replication

Requirement: –

  • Appliance based vCenter Server (VCSA) – Version 6.5
  • Site Recovery Manager – Version 6.5
  • Replication Type in SRM – vSphere Replication

Sequence to upgrade Windows based vCenter 5.5 to VCSA 6.5: –

SRM

If you are planning to upgrade your vSphere environment, follow this order to upgrade vCenter server, SRM, and vSphere Replication.

Overview of Upgrade order: –

  • Since vCenter server 5.5 doesnt have PSC server, you need to make sure that you install PSC when prior to upgrade vCenter server.
  • Upgrade PSC and vCenter Server in Protected Site.
  • Upgrade vSphere Replication appliance in Protected Site.
  • Upgrade Site Recovery Manager in Protected Site.
  • Perform the same steps in Recovery Site.
  • If you are using array based replication in SRM, you need to upgrade SRA in both site.
  • Once up gradation sequence has been done, verify vCenter server and SRM sites status.
  • Upgrade ESXi host in both protected and recovery sites.

Step by Step Guide to Upgrade vSphere Environment: –

Prerequisites: –

  • Download vCenter Server Appliance ISO image, VMware SRM 6.5 setup, and vSphere Replication appliance from VMware download portal.
  • Ensure that you have a windows machine from where you will initiate the installation.
  • Ensure to get SSO credentials and VCDB and SRM DB database handy.
  • Note down the details of ESXi Host where you will deploy VCSA appliance.
  • If you want to use the same name of vCenter Server which you are using currently, you need to rename windows vCenter VM with alternate name.
  • If you are using VMware SRM 5.8.1, then you need to upgrade it to VMware SRM 6.0 and then you can upgrade it to Vmware SRM 6.5.

vCenter Server Upgrade: –

There are two stage process to upgrade vCenter Server.

  1. Deployment of VCSA
  2. Migration of Windows vCenter Data to newly deployed VCSA.

The-VCSA-6.5-Stages

Stage 1:

  • Now mount VCSA image ISO in any windows machine. Explore the ISO folder and navigate to VMware-Migration-Assistant.exe. Right click and click on Run as Administrator.
  • Follow the steps and provide SSO credentials.
  • Once you will get black screen with a message – Waiting for migration to Start, Switch to ISO folder again.
  • Go to vcsa-ui-installer/win32 folder and click on Installer and Run as administrator.
  • On windows screen, you will get four options.
    • Install
    • Upgrade
    • Migrate
    • Restore
  • Click on Migrate and follow the steps to complete the deployment of VCSA.

Stage 2:

  • Once deployment gets complete, you need to switch to stage 2. Here you need to migrate existing windows vCenter server data to VCSA.
  • During the data migration the Windows vCenter will be shutdown and the VCSA will be configured with its IP address.
  • Follow the steps and complete stage 2. Now you can access vCenter server using Web client.

 

Upgrade vSphere Replication Appliance: –

Download vSphere replication appliance and upgrade appliance by following below VMware document.

https://docs.vmware.com/en/vSphere-Replication/6.5/com.vmware.vsphere.replication-admin.doc/GUID-30083484-FB13-485E-AEC9-0695EADB7B3D.html

Upgrade VMware Site Recovery Manager: –

Follow below article to upgrade VMware Site Recovery Manager.

https://docs.vmware.com/en/Site-Recovery-Manager/6.5/com.vmware.srm.install_config.doc/GUID-4642FBA0-6D25-4B68-912A-1BDD8D02EF00.html

 

 

 

 

 

 

How VMware HA Works | Deep Dive

Overview of HA (High Availability): –

  • When you creates HA cluster very first time, then Virtual Machines are configured with cluster default settings.
    • VM Restart Priority
    • Host Isolation Response
    • VM Monitoring
  • There is master host election when the cluster is first created. All other hosts are slaves.
  • Master host is responsible for monitoring the host connectivity with slave host.
  • Master host also deals with different possible issues that can happen.
    • Host get network isolated.
    • Host fails (Hardware or other problem).
    • Host loses connection to the master host.
  • For Virtual Machine monitoring, there are three options.
    • Leave running (Default)
    • Shutdown (Required VM Tools)
    • Power Off

Component of HA: –

  • FDM
  • Hostd
  • vcenter

FDM (Fault Domain Manager): –

  • Communicating host resource information, VM state, and HA properties to the other hosts.
  • It also handles heartbeat mechanism.HA deep dive
  • It provides VM placement and VM restart.
  • HA has no dependency on DNS. It works on IP Address. This is improvement in FDM.
  • FDM directly talk to hostd and vCenter
  • FDM is not dependant on VPXA.
  • You can check FDM logs – fdm.log in /var/log/

HOSTD Agent: –

  • It is agent which is installed on ESXi host.
  • Responsible for many task like power on Virtual Machine
  • If HOSTD is unavailable or not running, host will not participate in any FDM related process.
  • FDM relies on HOSTD for information about the VM that are registered to the host, and manager VM through HOSTD API.
  • FDM is dependant on HOSTD. If HOSTD is not operational, FDM halts all functions and wait for HOSTD to become operations.

Use of vCenter in HA: –

  • Deploying and configuring HA Agent.
  • Communication of cluster confiugration change
  • Protection of VM
  • Pushing out the FDM to the ESXi hosts.
  • Communicate configuratoin changes in the cluster to the host.
  • HA leverage vCenter to retrieve information about the status of VM.

Fundamental Concepts of HA: –

  • Master/Slave Agent

  • Heart-beating

  • Isolated vs Network Partitioned

  • VM Protection

  • Component Protection

Understand Master Host: –

  • HA Architecture includes the concept of Master and Slave HA agent.
  • There is only one master slave in a HA cluster, except during Network Partition scenario.master host
  • Master is responsible for monitoring the health of VM
  • Restart any VM which fails.
  • Slave pass information to master.
  • HA agent also implements the VM/App monitoring feature which allows it to restart virtual machine in case of a OS or restart service in case of application failure.
  • Master Agent keep track of the VM for which it is responsible for, and take action when appropriate.
  • Master will claim responsibility for a VM by taking ownership of the datastore on which VM configuration file is stored.
  • Master is responsible for exchanging state information with vCenter Server.
  • Send/Receive information to vCenter when required.
  • Master host initiate the restart of VM when host failed.

What if Master Fails?

master fails

HA election occurs when you enable HA on VMware Cluster and master host:-

  • Fails
  • Become Network partition or isolated.
  • Disconnect from vCenter Server.
  • Put in maintenance or standby.

HA election takes 15 seconds to elect slave as a master. It works over UDP protocol.

Make the election process on the basis of highest number of datastore.

If two or more host has some number of datastore, the highest/largest MOID will get preference. It’s done on basis of lexically MOID. Let’s take a value of MOID of two Hosts 99 and 100. Here 9(99) is greater then 1(100)(9 >1). In this example, 99 is largest MOID.

When master is elected, it will try to acquire the ownership of datastore which it can directly access by proxying request to one of the slave connected to it using the management network.

For regular storage architecture, it does this by locking a file called “Protected List”.

Naming format and location for this Protected List file is as below.

./vSphere HA/ <Cluster Specific Directory>/ProtectedList

Structure of cluster specific directory.

<UUID of VC> -<Number of the MOID of Cluster>-<Random 8 character string>-<Name of the host running VC>

Understand Slave Host: –

  • It monitors state of Virtual machine and inform Master host.
  • Monitor health of master by monitoring heatbeat.
  • Slave host sends heartbeat to master so that master can detect outage.

Local Files for HA: –

When HA is configured on a host, the host will store specific information about it’s cluster locally.

  • Cluster Config
    • It’s not human readable.
    • It contains configuration details of cluster.
  • vmmetadata
    • This file is also not human readable.
    • It contains actual compatibility information matrix for every HA protected VM and list all with which it is compatible.
    • Metadata includes the custom properties, descriptions, tags, owner, cost center, etc regarding a Virtual Machine.
  • fdm.cfg
    • Configuration setting and logging and syslog details are store in this file.
  • hostlist
    • A list of host participating in the cluster, including hostname, IP address, MAC address and heartbeat datastore.

Understand Heartbeating: –

Mechanism used by HA which check if host is alive.

There are two types of Heartbeat.

  1. Network Heartbeat
  2. Datastore Heartbeat

Network Heartbeat: –

  • It use by HA to determine if a ESXi host is alive.
  • Slave send network heartbeat to master and master to slave.
  • It send heartbeat by default every second.

Datastore Heartbeat: –

  • It add on extra level of resiliency and prevent unnecessary restart attempts.
  • Datastore heartbeat enables a master to determine the state of a host that is not reachable via management network.
  • By default there are two datastores get selected. But it can be possible to add more datastores. You can do this by following string in Advance options. Valid values can range from 2-5 and the default is 2.
das.heartbeatdsperhost
  • Selection process gives preference VMFS datastores over NFS.

Network Isolated vs Partitioned Network Partitioned: –

Isolated vs partitioned

Isolated

  • When it doesn’t observe any HA management traffic on management network and it cannot ping the configured isolation network address.
  • Host is isolated only when host inform the master via the datastore that is isolated.

Partitioned

  • When host is operational but cannot reach over management network.
  • There will be multiple masters in case of network partitioned.

 

How vMotion Works ?

What is vMotion?

  • vMotion  enables live migration of a running virtual machine between ESXi Hosts
  • It is transparent to the Virtual Machine’s OS and applications.
  • Invaluable tool to admin to achieve the followings;
    • Avoid Server Downtime
    • Allow Troubleshooting
    • Provide Flexibility
  • Key enabler of DRS, DPM, and FT

vMotion works

What needs to be migrated?

  • Processor and devices state  – CPU, Network, SVGA
  • Disk – Use shared storage between source and destination
  • Memory – Pre copy memory while Virtual Machine is running

How vMotion Works?

  • Quiesce Virtual Machine on source machine.
  • Transfer memory and device state(checkpoint) from source to destination.
  • Resume Virtual Machine on Destination
  • Copy remainder of memory from source to destination
  • Free Virtual Machine resources on source machine.

vMotion Step by Step

Other Interesting Facts, Problems, and Troubleshooting: –

  • Virtual  Machine remains suspended during memory transfer
  • Copying Virtual Machine with large memory size may problematic.
  • 64 GB Virtual Machine requires around 57 seconds on 10 GbE NIC.
  • VMotion will check the remote system to make sure there is enough RAM and CPU before it begins the process.

Troubleshooting: –

  • Migration ID is same on source and destination.
    • Go to VMkernel log (/var/log/vmkernel.log)
    • Grep the migration ID for all vMotion related timing and statistics.

That’s it from here. Stay connected.