How to Upgrade Your Entire HPE SimpliVity Environment including VMware vSphere, Firmware and Third-Party Software

This article is about upgrading your HPE SimpliVity environment, including all of the components that come with this upgrade: VMware vSphere, firmware and your third-party software.

I will describe the global steps, time estimation, do’s and don’ts and tips from my side as I have a lot of experience performing these kind of upgrades.

The information in this article is based on HPE SimpliVity 4.0.0.

How to get started?

First of all, it is important to start off this project in the right way. And trust me, there’s many wrong ways out there. Start off my asking these questions to yourself, or your team:

• Do we have an active Support Contract and support phone numbers for our server components?
• Do we have an up-to-date overview of our infrastructure, including server names, models, version numbers and login credentials?
• Do we have the details of our out-of-band solution (IPMI, iLO, iDRAC)?
• How is the current health of our environment? Are there any outstanding warning or critical alerts?

Once you have got a clear and positive answer to these questions, you can continue with the next steps!

Step One: Creating an Overview

This is an extent to the questions I just shared with you. An overview of the current version and new, or target version of each component.

I usually create an Excel spreadsheet for this, and include the following details on each row:

• Server or system name
• Brand and Model
• Current status (so you can put some comments if it’s not 100% healthy)
• Current ESXi version and build
• Current SimpliVity OVC version and build
• Current BIOS/iLO/iDRAC version
• Target ESXi version and build (leave blank for now)
• Target SimpliVity OVC version and build (leave blank for now)
• Target BIOS/iLO/iDRAC version (leave blank for now)

Do this for every SimpliVity OmniStack or OmniCube machine.

And finally include a line with details around your vCenter Server and SimpliVity Arbiter (current and new version, build etc).

Step Two: Selecting Versions

The SimpliVity version is the leading version. That means, that whatever version you pick, will decide what VMware vCenter Server, VMware ESXi and firmware version you will be able to use.

The target SimpliVity version you choose, is of course yours to choose. I usually try to upgrade to the latest version, if it’s at least 1 month old. That way, the biggest issues with the release should already be known and if the vendor decides to pull it back, you are not stuck with a broken release.

Caution: One thing to note here is that, once you know the model and type of your servers and the SimpliVity version you would like to go to – is to check the HPE SimpliVity Interoperability Guide for that specific version and see if there are any constraints that prevent you from moving to this version.

One thing to note here is that your hardware can be a constraint too. For example the newest SimpliVity version (4.0.0 at moment of writing) might support vSphere 6.7 U3, but your server hardware might only support up to vSphere 6.5 U2. Be sure to check out the VMware HCL to find out what version your model supports.

A final note here, is that upgrading vCenter Server and ESXi in this process also requires your third-party software to be compatible with these new versions. Think of your backup software, monitoring software, scripts and plugins for example. They might require an upgrade before you start touching vCenter Server and ESXi.

With all this, you should be able to select the right version of SimpliVity, vCenter Server, ESXi and firmware. Once you selected these, perform a final check with the HPE SimpliVity Interoperability Guide for the SimpliVity version you are trying to install to make absolutely sure that these versions are compatible with each other.

Step Three: Downloading Software

Now you know what you need to download! Head over to the designated websites to download the following items and place them on your management server in easy to distinguishable folders:

• HPE SimpliVity Software Bundle of your choosing (HPE Website under Software and Licensing – active support required)
• Firmware upgrade ISO files
• vCenter Server ISO (or patch bundle when not upgrading to a major release)
• ESXi Offline Bundle (ZIP) – vendors usually provide a customized ISO that includes vendor-specific drivers

This structure can of course look different if you also include the upgrade of standalone ESXi hosts (non-SimpliVity) in your environment, but that is out of scope for this article.

By placing the software ready to use on your management server, it will be easier to perform the upgrade and exactly know where your software is. And prevent needing to download software on the fly (especially the SimpliVity package as it’s about 12 GB in size) as you will want to focus on upgrading during your maintenance window and not waiting for a download to finish.

Step Four: Upgrading Firmware

This is where the fun starts!

We will start off with upgrading the firmware of the SimpliVity OmniCubes (Legacy hardware) or SimpliVity OmniStacks (HPE hardware).

In this article, I will be focussing on the upgrade of HPE ProLiant servers, but the approach is similar.

One thing to note here, is that HPE supplies you with two firmware ISO files for Gen9 servers – one for the LSI controller and one for the other firmware upgrades. Upgrade the LSI controller first.

1. Open a connection to the iLO of the server you want to upgrade
2. Open the .NET Integrated Remote Console
3. When you see the console, verify you’re on the right system
4. Mount the firmware ISO you want to use, using Virtual Drives and Image File (CD-ROM/DVD)
5. In the iLO main screen, go to Virtual Media and Boot Order. Configure the One-Time-Boot option to match CD/DVD Drive (click apply)
6. Go into vSphere Web Client and put ESXi in Maintenance Mode (move your VMs manually off the host when you don’t have Fully Automated DRS enabled)
7. When all VMs are off, shut down the OVC on this host
8. The host should now be in Maintenance Mode – so perform the reboot and continue in the iLO remote console
9. Follow the on-screen instructions for the firmware upgrade and once it completes, let it boot back into ESXi
10. Finished! And repeat this for all of the SimpliVity nodes in your cluster

A firmware upgrade of a HPE DL380 Gen9 takes about one hour, including the LSI firmware upgrade, SPP firmware upgrade and reboots. I’ve seen other brands, for example Lenovo, take about 90 – 120 minutes from shutdown until back online in vCenter Server.

Step Five: Upgrading SimpliVity

Next up: SimpliVity components. I’ll write these in a numbered format as before so you can see exactly what needs to be done in what order.

Caution: Before cracking on, be 100% sure that all related components are compatible with the version of SimpliVity you are deploying. For example, vCenter Server, ESXi and firmware version – these should be mentioned in the HPE SimpliVity Interoperability Guide and marked as compatible before you continue. It could be you need to perform some upgrades before you can upgrade your SimpliVity software.

The following steps use software from the HPE SimpliVity bundle you downloaded earlier.

1. Upgrade the SimpliVity Arbiter software to match the version you are rolling out
2. Use the SimpliVity Upgrade Manager to start upgrading your cluster and bring your OmniCube/Stack Virtual Controllers to the desired version
3. Verify all components are healthy and VMs are Storage HA by using PuTTy to connect to one of your OVCs and running svt-federation-show and svt-vm-show
4. Upgrade your vSphere Web Client plugin
5. If you have any standalone hosts that access the SimpliVity datastores, upgrade their SimpliVity VAAI-NAS plugin
6. Done!

Upgrading time depends on the size of your environment, but in my experience it takes roughly 30 minutes per-OVC.

Step Six: Upgrading vCenter Server

Upgrading vCenter Server has never been more easy! In this section I am assuming you are already using the vCenter Server Appliance – and not the Windows edition.

You downloaded the ISO file before, so let’s get to the steps:

1. Prepare a temporary IP address for vCenter Server that will be used for upgrading purposes
2. Mount the vCenter Server ISO to your management server / system
3. Open the UI installer
4. Use the wizard to fill out all the details of your current vCenter Server and new vCenter Server (it will create a new appliance and import the current configuration)
5. This process might take some time, especially if you choose to include all events, tasks and performance data. I personally never include them, but I guess in some situations you do want to keep them
6. Your new vCenter Server should be up and running all fine! The old appliance should be shut down and can be safely deleted in a while when your new appliance works okay. I usually keep it around for about 30 days
7. Also check if your HPE SimpliVity and other third-party plugins work okay within the vSphere Web Client (there’s now a HTML5 plugin for HPE SimpliVity)
8. Verify if your third-party software such as backup, monitoring and scripts are working as expected
9. Done!

Upgrading vCenter Server can be quick or take hours. If you don’t choose to import all of the historical events, performance data and tasks, it should be up and running within an hour. Otherwise, the wizard will tell you an estimation of upgrade time (I’ve seen times where it showed over 6 hours).

Step Seven: Upgrading ESXi and VMware Tools

Final step is to upgrade ESXi and push out the latest version of VMware Tools to your VMs!

Update: Want to be SMART? Use the HPE SimpliVity Upgrade Manager to perform ESXi upgrades for you! Of course, these only work on SimpliVity nodes, but it saves you quite some copy-pasting in shell. Just put the HPE customized ESXi bundle for SimpliVity inside the Upgrade Manager folder, and run it! During the wizard, you just skip the SimpliVity upgrade option and select the new ESXi version. Follow the procedure below for manual upgrades, which you will need for standalone ESXi hosts anyway when you don’t like to use VMware Update Manager.

To prepare for this step, I usually upload the ESXi offline bundle you downloaded before to a datastore that is accessible from all to-be-upgraded hosts.

Choose whether you want to use VUM (vSphere Update Manager) or the CLI / SSH – I always use SSH as it’s much quicker and less error prone in my experience. So the following list is for when you are using SSH 🙂

1. Put your host in maintenance mode
2. Log into ESXi with PuTTy or alternative SSH client
3. Navigate to the folder that contains the ESXi offline bundle
4. Find out the profile name contained in the bundle by running esxcli software sources profile list -d “/vmfs/volumes/path-to-bundle.zip”
5. Start installing the bundle using the following command: esxcli software profile install -p PROFILENAME -d “/vmfs/volumes/path-to-bundle.zip“
6. This process usually takes a minute or 2 and then comes back with completion and mentioning a reboot is required
7. Reboot the server
8. Verify if the new version installed okay and that the ESXi host appears back in vCenter Server (if it doesn’t appear after 5 minutes after it is fully booted, use a manual “connect” from vCenter Server)
9. Exit maintenance mode and start up your OVC VM
10. Log into one of your other OVC’s using SSH and run svt-federation-show to verify the status of your Federation and see if all OVC’s are started up okay
11. When everything seems healthy, start using svt-vm-show to see if your VMs are getting back in-sync
12. When all your VMs are in-sync, move on to the next host

Note: Sometimes, installing the offline bundle comes back with an error saying there’s a dependency missing or conflict with an existing VIB. Find out if you are using this particular VIB, and if not, you can safely delete it using the CLI using esxcli software vib remove –vibname=NAME.

ESXi Upgrades, using the CLI, are quick. Installation takes about 5 minutes and then a reboot follows, so around 20 minutes in most cases! VUM takes a bit more time because it performs more checks and I believe it’s just less efficient in the background.

Issues After Firmware Upgrade

During my last firmware upgrade on a HPE DL380 Gen9 (upgrading to SVTSP-2019_0315.04 / 790-000107-lsi-j) I ran into a problem where the OVC didn’t come online fully. It showed a failed start for SimpliVity HAL Daemon and SimpliVity OmniCube services.

When running the command svtfs in SSH on the OVC, it showed some garbled text and that it was not running. Running tail -f $SVTLOG also looked very unhealthy (Thrift error, no such file or directory, Unable to connect to HAL server).

Apparently, the SimpliVity node created a “nostart” file on the OVC in /var/svtfs/svt-hal/0 with the following contents: Storage Agent found a critical failure!

I have seen this issue before with another client, and simply renaming the file to for example nostart.old (rm nostart nostart.old) should fix the issue. Restart your OVC to see if it’s working. The nostart file was created after the firmware upgrade – and blocks a normal boot of the OVC. As I was sure the environment was healthy before I started, and the firmware upgrade triggered this, I felt confident performing this procedure. If you’re not, please reach out to HPE SimpliVity support to have them check your issue.

Final Remarks

The order I used above, is one that has been working for all of my upgrade projects. However – it could be that your environment requires a different approach. You might need to start off with a vCenter Server upgrade, or maybe you’re running behind on firmware and need some intermediate upgrades first. See what is a supported upgrade path for your particular environment using the HPE SimpliVity Interoperability Guide to be 100% sure you don’t end up in an unsupported, or worse: broken environment.

This concludes all of the steps required to bring your environment into a fully up-to-date state. I hope this was helpful to you, and if you have any questions or feedback, leave them under this post!

And if you need any support when you are facing upgrades like mentioned here, please reach out to me using my IT Support page.

Advertisement