Upgrading HPE SimpliVity using the CLI when Upgrade Manager is giving you headaches

So, you’re ready to upgrade your environment to a higher SimpliVity version right? Good stuff! It’s great to keep your environment up-to-date to receive new features and security updates.

For SimpliVity, you would typically use HPE SimpliVity Upgrade Manager. However, this software sometimes seems to have a mind of its own and will not launch, or actually fire off your upgrade.

This post is dedicated to using the CLI as a way to upgrade your environment, and is from this day on also my personal preferred method.

Overview

This post is baed on a customer upgrade from SimpliVity 3.7.6 U1 to 3.7.10. They are running OmniCube CN-3000 SimpliVity nodes (Dell) and have already upgraded the required components for this version of SimpliVity (SimpliVity Plug-In and Arbiter, VMware vCenter/ESXi, firmware and third-party software).

HPE SimpliVity Upgrade Manager

So before we get to the good bits, I’d like to share some information about the HPE SimpliVity Upgrade Manager.

It’s a good looking tool, and has served me well in the past. It’s orchestrated, so you just fire away on a cluster or multiple clusters and it does its job.

In my experience, when it starts upgrading, and the first node upgraded successfully, it will perform this on the other nodes without issues.

However- before you are cheering wildly behind your desk- there are some dependencies and buggy situations here.

Java

First of all, you need Java installed on your system. For some clients, this has proven to be an issue. No Java, means no Upgrade Manager. You could run it from your local machine, or spin up a new VM, but you need to have Java installed locally.

Bandwidth

Secondly, when using Upgrade Manager, you will be transferring software from the machine its run from, to the OVCs. The upgrade package is around 2 GB and will be transferred to each and every OVC before starting the upgrade on that particular node. Keep this in mind if your bandwidth is limited!

Bugs

Mainly, this has been around Java for me. Problems with starting, but also problems firing off the upgrade. For example the most recent one that didn’t let me go on with the upgrade came back with “Failed: java.io.IOException: Error writing request body to server“.

See screenshot below.

Introduction to the CLI Upgrade

My God I love the CLI upgrade! It’s so quick and easy to follow! With enthusiasm I am writing this, and am having a hard time not to splash this section full with text. So, I’m going to break it down in some pieces below.

But before you head on, I want to let you know that using the CLI does require you to feel comfortable with text-only screens and Linux commands. I mean, you have to start somewhere, but if you have no experience, perhaps its good to have a colleague next to you who already feels comfortable with the CLI.

In this CLI upgrade, we will be connecting to the OVC’s (OmniCube Virtual Controller) using SSH. PuTTY has my preference, but you can use any client.

Now read on! 🙂

Preparation for CLI Upgrade

Before we can start firing away commands, you should prepare some things.

First of all, use a machine (whether its remote or local) with a good resolution. If you’re using PuTTY, it’s very valuable to be able to dump as much text information on your screen.

Next, when using PuTTY, it’s advisable to log all screen output to a text file. That means that everything you see on your screen, is logged into the file for troubleshooting purposes, or just as a piece of documentation when you need it. If there’s anything going wrong you have proof of what you did, and you can also supply it to HPE support if they need to assist.

Also, keep the administrator@vsphere.local password near. You will be using it many times. Perhaps you can copy paste it easily from your password manager, or work with a local notepad file (don’t save it to prevent a security risk!)

Downloading and Uploading The Bits

Next would be downloading the huge HPE SimpliVity software package (around 11GB) from the HPE website. You need a valid contract to download this software. Extract the file afterwards. One you’ve got all the files, upload the following files to one of your SimpliVity datastores:

SimpliVity-OmniCube-Software-revA-3.7.10.200.tar
SimpliVity-OmniCube-Software-revA-3.7.10.200.sha1

I would suggest putting them in a subfolder on your datastore, for example .svtupgrade

It will probably look something like this:

Now the next step is to note the actual path to the .svtupgrade folder.

Navigate to the datastore in vSphere Web Client and go to Configure > Device Backing. Here, you will see the path to the datastore as displayed below.

For example, the path here could be:

/mnt/svtfs/0/ba1951b6-52c3-xxxxxxx

That means that the full path to the upgrade files, would be:

/mnt/svtfs/0/ba1951b6-52c3-xxxxxxx/.svtupgrade/SimpliVity-OmniCube-Software-revA-3.7.10.200.tar

Save this path somewhere for later use.

Firing Commands, finally!

Okay here we go! If you haven’t done already, open up a PuTTY session to the OVC you want to upgrade. Log in using administrator@vsphere.local and the corresponding password.

Then, start off with checking your environment health. Use the following command to view the state of your OVCs, version and IP addresses:

svt-federation-show

Everything green? Good! Now use the following command to view all of your VMs across your federation and filter out the unhealthy ones (who are not in HA sync or have a problem with availability):

svt-vm-show --violations

All as expected? All in-sync? Good! That means we are ready to upgrade.

Starting the upgrade

As you are logged in to the first OVC you want to upgrade, use the following command to fire off the upgrade using the unique path you saved earlier:

svt-software-upgrade --omnicube --pkgpath /mnt/svtfs/0/GUID_HERE/.svtupgrade/SimpliVity-OmniCube-Software-revA-3.7.10.200.tar

After you fired this off, the following information appears and requires your manual input.

Welcome to SimpliVity OmniCube 3.7.6.160
administrator@vsphere@omnicube-ip0-103:~$ svt-software-upgrade --omnicube --pkgpath /mnt/svtfs/0/ba1951b6-52c3-4cc6-a453-ab5fecc6798d/.svtupgrade/SimpliVity-OmniCube-Software-revA-3.7.10.200.tar
You are about to upgrade only this OmniStack host. (Repeat the operation on all remaining OmniStack hosts in this DataCenter).
Proceed? (y/n): y
SimpliVity recommends upgrading a Federation during scheduled system maintenance periods or when you expect minimal Federation use.
Upgrading during periods of high IOPS can result in long upgrade times.
Proceed? (y/n): y
Upgrade task with id 564d6408-4848-5651-ea59-c00f8beab5a7:564d6408-4848-5651-ea59-c00f8beab5a7:844a71df-966e-48d1-acfd-287afdc89399 has been started.
administrator@vsphere@omnicube-ip0-103:~$

Alright! That’s it! It’s running! 🙂

Monitoring

Now, to follow the progress, use the following command to “tail” the upgrade log – giving you a live view of what’s going on.

tail -f /var/log/svt-upgrade.log

This is some example output right after launching the upgrade:

administrator@vsphere@omnicube-ip0-103:~$ tail -f /var/log/svt-upgrade.log
2020/04/16 23:36:39 INFO> copyvalidate.pl:283 main:: - Opened FIFO /tmp/copyvalidate-13954//tar.fifo
2020/04/16 23:36:39 INFO> copyvalidate.pl:319 main:: - Reading /mnt/svtfs/0/ba1951b6-52c3-4cc6-a453-ab5fecc6798d/.svtupgrade/SimpliVity-OmniCube-Software-revA-3.7.10.200.tar from disk
2020/04/16 23:36:39 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 0% processed (0/1762560000) after 1s
2020/04/16 23:36:41 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 2% processed (35258368/1762560000) after 3s
2020/04/16 23:36:43 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 4% processed (70516736/1762560000) after 5s
2020/04/16 23:36:46 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 7% processed (123404288/1762560000) after 8s
2020/04/16 23:36:48 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 9% processed (158662656/1762560000) after 10s
2020/04/16 23:36:50 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 11% processed (193888256/1762560000) after 12s

As you can see, detailed information appears live on your screen. Any errors, information or warnings will appear. And remember? We started logging this whole session to text? You can read all this back, as a reference or for troubleshooting if you need 🙂

After the upgrade has finished, it will reboot the OVC. So your PuTTY will disconnect the session.

If nothing seems to happen, it could be that it’s doing some data work in the background. If you’re curious like me, you can also use the following command to tail the $SVTLOG – which is the active log of the OVC:

tail -f $SVTLOG

In my case the following lines appeared and nothing seemed to happen:

INFO svtfs.app Stopping HAL...
INFO svtfs.app HAL has been stopped.

Don’t worry – check the CPU performance of your OVC in the vSphere Web Client and see that there’s actually things happening in the background (about 23% constant CPU load in my case). Don’t reboot your OVC as this might break it and corrupt the database! At least wait one hour – if you’re starting to get worried, contact HPE Support.

You can see the current status of the upgrade using the following command:

svt-software-status-show

It will either tell you it’s still in progress, failed or completed. And whether to contact HPE support or not.

Rollback

Whenever something goes wrong in this process, after the first reboot, OVC will do a check to see if the upgrade went well. If not – it will reboot again automatically and perform a rollback to your previous version.

Upgrade and Commit

Repeat this whole process for every OVC, until all of them are up-to-date, running the same version and are healthy.

Then it’s time for a commit, this sets your upgrade into stone and a rollback will no longer be possible.

Use this command to commit the upgrade:

svt-software-commit

Finish!

You did it! Maybe your first out of many CLI upgrades! 🙂

I hope you enjoyed this guide, and that everything went smoothly.

If you would like any support with your upgrade, be sure to reach out to me as I offer professional remote support services. For more information see this page.

Advertisement