So, you’re ready to upgrade your environment to a higher SimpliVity version right? Good stuff! It’s great to keep your environment up-to-date to receive new features and security updates.
For SimpliVity, you would typically use HPE SimpliVity Upgrade Manager. However, this software sometimes seems to have a mind of its own and will not launch, or actually fire off your upgrade.
This post is dedicated to using the CLI as a way to upgrade your environment, and is from this day on also my personal preferred method.
This post is baed on a customer upgrade from SimpliVity 3.7.6 U1 to 3.7.10. They are running OmniCube CN-3000 SimpliVity nodes (Dell) and have already upgraded the required components for this version of SimpliVity (SimpliVity Plug-In and Arbiter, VMware vCenter/ESXi, firmware and third-party software).
HPE SimpliVity Upgrade Manager
So before we get to the good bits, I’d like to share some information about the HPE SimpliVity Upgrade Manager.
It’s a good looking tool, and has served me well in the past. It’s orchestrated, so you just fire away on a cluster or multiple clusters and it does its job.
In my experience, when it starts upgrading, and the first node upgraded successfully, it will perform this on the other nodes without issues.
However- before you are cheering wildly behind your desk- there are some dependencies and buggy situations here.
First of all, you need Java installed on your system. For some clients, this has proven to be an issue. No Java, means no Upgrade Manager. You could run it from your local machine, or spin up a new VM, but you need to have Java installed locally.
Secondly, when using Upgrade Manager, you will be transferring software from the machine its run from, to the OVCs. The upgrade package is around 2 GB and will be transferred to each and every OVC before starting the upgrade on that particular node. Keep this in mind if your bandwidth is limited!
Mainly, this has been around Java for me. Problems with starting, but also problems firing off the upgrade. For example the most recent one that didn’t let me go on with the upgrade came back with “Failed: java.io.IOException: Error writing request body to server“.
See screenshot below.
Introduction to the CLI Upgrade
My God I love the CLI upgrade! It’s so quick and easy to follow! With enthusiasm I am writing this, and am having a hard time not to splash this section full with text. So, I’m going to break it down in some pieces below.
But before you head on, I want to let you know that using the CLI does require you to feel comfortable with text-only screens and Linux commands. I mean, you have to start somewhere, but if you have no experience, perhaps its good to have a colleague next to you who already feels comfortable with the CLI.
In this CLI upgrade, we will be connecting to the OVC’s (OmniCube Virtual Controller) using SSH. PuTTY has my preference, but you can use any client.
Now read on! 🙂
Preparation for CLI Upgrade
Before we can start firing away commands, you should prepare some things.
First of all, use a machine (whether its remote or local) with a good resolution. If you’re using PuTTY, it’s very valuable to be able to dump as much text information on your screen.
Next, when using PuTTY, it’s advisable to log all screen output to a text file. That means that everything you see on your screen, is logged into the file for troubleshooting purposes, or just as a piece of documentation when you need it. If there’s anything going wrong you have proof of what you did, and you can also supply it to HPE support if they need to assist.
Also, keep the email@example.com password near. You will be using it many times. Perhaps you can copy paste it easily from your password manager, or work with a local notepad file (don’t save it to prevent a security risk!)
Downloading and Uploading The Bits
Next would be downloading the huge HPE SimpliVity software package (around 11GB) from the HPE website. You need a valid contract to download this software. Extract the file afterwards. One you’ve got all the files, upload the following files to one of your SimpliVity datastores:
I would suggest putting them in a subfolder on your datastore, for example .svtupgrade
It will probably look something like this:
Now the next step is to note the actual path to the .svtupgrade folder.
Navigate to the datastore in vSphere Web Client and go to Configure > Device Backing. Here, you will see the path to the datastore as displayed below.
For example, the path here could be:
That means that the full path to the upgrade files, would be:
Save this path somewhere for later use.
Firing Commands, finally!
Okay here we go! If you haven’t done already, open up a PuTTY session to the OVC you want to upgrade. Log in using firstname.lastname@example.org and the corresponding password.
Then, start off with checking your environment health. Use the following command to view the state of your OVCs, version and IP addresses:
Everything green? Good! Now use the following command to view all of your VMs across your federation and filter out the unhealthy ones (who are not in HA sync or have a problem with availability):
All as expected? All in-sync? Good! That means we are ready to upgrade.
Starting the upgrade
As you are logged in to the first OVC you want to upgrade, use the following command to fire off the upgrade using the unique path you saved earlier:
svt-software-upgrade --omnicube --pkgpath /mnt/svtfs/0/GUID_HERE/.svtupgrade/SimpliVity-OmniCube-Software-revA-220.127.116.11.tar
After you fired this off, the following information appears and requires your manual input.
Welcome to SimpliVity OmniCube 18.104.22.168 administrator@vsphere@omnicube-ip0-103:~$ svt-software-upgrade --omnicube --pkgpath /mnt/svtfs/0/ba1951b6-52c3-4cc6-a453-ab5fecc6798d/.svtupgrade/SimpliVity-OmniCube-Software-revA-22.214.171.124.tar You are about to upgrade only this OmniStack host. (Repeat the operation on all remaining OmniStack hosts in this DataCenter). Proceed? (y/n): y SimpliVity recommends upgrading a Federation during scheduled system maintenance periods or when you expect minimal Federation use. Upgrading during periods of high IOPS can result in long upgrade times. Proceed? (y/n): y Upgrade task with id 564d6408-4848-5651-ea59-c00f8beab5a7:564d6408-4848-5651-ea59-c00f8beab5a7:844a71df-966e-48d1-acfd-287afdc89399 has been started. administrator@vsphere@omnicube-ip0-103:~$
Alright! That’s it! It’s running! 🙂
Now, to follow the progress, use the following command to “tail” the upgrade log – giving you a live view of what’s going on.
tail -f /var/log/svt-upgrade.log
This is some example output right after launching the upgrade:
administrator@vsphere@omnicube-ip0-103:~$ tail -f /var/log/svt-upgrade.log
2020/04/16 23:36:39 INFO> copyvalidate.pl:283 main:: - Opened FIFO /tmp/copyvalidate-13954//tar.fifo
2020/04/16 23:36:39 INFO> copyvalidate.pl:319 main:: - Reading /mnt/svtfs/0/ba1951b6-52c3-4cc6-a453-ab5fecc6798d/.svtupgrade/SimpliVity-OmniCube-Software-revA-126.96.36.199.tar from disk
2020/04/16 23:36:39 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 0% processed (0/1762560000) after 1s
2020/04/16 23:36:41 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 2% processed (35258368/1762560000) after 3s
2020/04/16 23:36:43 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 4% processed (70516736/1762560000) after 5s
2020/04/16 23:36:46 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 7% processed (123404288/1762560000) after 8s
2020/04/16 23:36:48 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 9% processed (158662656/1762560000) after 10s
2020/04/16 23:36:50 INFO> Statusfile.pm:208 Util::Statusfile::reportPercentage - 11% processed (193888256/1762560000) after 12s
As you can see, detailed information appears live on your screen. Any errors, information or warnings will appear. And remember? We started logging this whole session to text? You can read all this back, as a reference or for troubleshooting if you need 🙂
After the upgrade has finished, it will reboot the OVC. So your PuTTY will disconnect the session.
If nothing seems to happen, it could be that it’s doing some data work in the background. If you’re curious like me, you can also use the following command to tail the $SVTLOG – which is the active log of the OVC:
tail -f $SVTLOG
In my case the following lines appeared and nothing seemed to happen:
INFO svtfs.app Stopping HAL...
INFO svtfs.app HAL has been stopped.
Don’t worry – check the CPU performance of your OVC in the vSphere Web Client and see that there’s actually things happening in the background (about 23% constant CPU load in my case). Don’t reboot your OVC as this might break it and corrupt the database! At least wait one hour – if you’re starting to get worried, contact HPE Support.
You can see the current status of the upgrade using the following command:
It will either tell you it’s still in progress, failed or completed. And whether to contact HPE support or not.
Whenever something goes wrong in this process, after the first reboot, OVC will do a check to see if the upgrade went well. If not – it will reboot again automatically and perform a rollback to your previous version.
Upgrade and Commit
Repeat this whole process for every OVC, until all of them are up-to-date, running the same version and are healthy.
Then it’s time for a commit, this sets your upgrade into stone and a rollback will no longer be possible.
Use this command to commit the upgrade:
You did it! Maybe your first out of many CLI upgrades! 🙂
I hope you enjoyed this guide, and that everything went smoothly.
If you would like any support with your upgrade, be sure to reach out to me as I offer professional remote support services. For more information see this page.
8 thoughts on “Upgrading HPE SimpliVity using the CLI when Upgrade Manager is giving you headaches”
Hi Thanks for your article.
The error in the UM is a Spaceprobe problem. This can be solve with an script from HPe. But this problem is fixed in 4.0.0 or 4.0.1 but i guess it 4.0.1.
But the UM is such a buggy SW incredible. Im struggling with the FW. I never saw that the UM updated the FW correctly. In my case he always skip the FW upgrade.
Thanks for your additional information Alexander! I never really applied firmware upgrades through the UM – I always use the ISO files and upgrade the components on each host one by one. Time consuming but I guess 100% success guarantee 😊
The Spaceprobe error could be due to leftover files left by the upgrade manager in the temporary .svtupgrade diretory when it fails which can be in any of your datastores. Just delete those files and you should be good to go.
Thanks for your feedback Luis!
Hi Rene, thanks very much for your post. I am attempting to upgrade the OmniStack software on two SimpliVity clusters in a Federation from v3.7.8 to v4.0.1U1 using Upgrade Manager. I wanted to upgrade one cluster at a time so I selected the first cluster and it began upgrading the first OVC and got to around 95% complete when the process failed. I noticed that the OVC rebooted twice, so the second time must have been the rollback as you wrote above. If I attempt the CLI upgrade method, do I need to shut down or vMotion any VMs from the SimpliVity hosts before commencing? I also have compute-only hosts in each of my clusters, do I need to shut down/vMotion VMs off these hosts before commencing?
Hi George! Thanks for reaching out.
So, yeah, the deployment manager is easy but if it doesn’t work, the CLI might be the better way to go.
Two OVC reboots does indeed mean a rollback. So something is stuck in the background preventing it from finalizing the upgrade.
When you use the CLI upgrade, it will actually perform the exact same steps in the background as the GUI. Upgrading the OVC does not require a host reboot so moving VMs is unnecessary.
Your computer nodes, well – the documentation does tell you to move these or shut them down. In my experience tho, this has never been any problem (as the OVC IP will move to another host). If you’re willing to try without moving, just make sure you’re doing it outside of office hours.
Oh and final thing here: do you know why the GUI upgrade failed? Check the logs as the issue might be something that can be resolved on your end!
Let me know how it works out for you =)
Thanks very much for getting back to me so promptly and for the advice and explanation 🙂
I logged a support case with HPE and when they ran an svt-federation-show, although it showed “all greens”, it also showed the OVC that failed the upgrade had a “Family” showing as “Unknown” whereas all the others had a “Family” showing “vSphere”. He got me to re-generate the certificates on all 5 of my OVCs, after which the Upgrade Manager GUI worked flawlessly 🙂
The HPE tech also assured me that VMs running on the compute-only hosts did not need to be shut down or vMotioned off them because – as you stated – their vSwitches are configured with a vmkernel “SVT_StorPG” PortGroup connecting to the storage. He said that had these hosts been configured to connect to the storage via the management IPs on the SimpliVity hosts, I then would have to move/shutdown their VMs.
So now that I’ve successfully upgraded all the OVCs to v4.0.1U1, I’m now seeing a warning message on all the SimpliVity hosts saying “OmniStack upgrade requires a host reboot to apply ESXi settings”. I did not upgrade the ESXi software on any of the hosts as they’re already on a version/build that’s compatible with 4.0.1U1, so I don’t know why this message appears?
So welcome! And thanks for feeding back your update! Regeneration of the OVC certificates is something I haven’t seen yet! But it does make sense if they were expired for example.
Weird about the message you are seeing. Perhaps it changed some of the advanced configuration parameters (NFS, timeouts, network/storage properties) to work better with SimpliVity. And when you tweak those settings, it usually requires a reboot!
Let me know if you’re running into anything else! Have a beautiful day! 🙂