Thinking out loud: The cloud (IaaS) delusion


Just so we’re all being honest here, I’m not going to sit here and lie about how I’m not biased and I’m looking at both sides 100% objectively.  I mean I’m going to try to, but I have a slant towards on prem, and a lot of that is based on my experience and research with IaaS solutions as they exist now.  My view of course is subject to change as technology advances (as anyones should), and I think with enough time, IaaS will get to a point where its a no brainer, but I don’t think that time is yet for the masses.  Additionally, I think its worth noting that in general, like any technology, I’m a fan of what makes my life easier, what’s better for my employer, and what’s financially sound.  In many cases cloud fits those requirement, and I currently run and have run cloud solutions (long before being trendy).  I’m not anti cloud, I’m anti throwing money away, which is what IaaS is mostly doing.

Where is this stemming from?  After working with Azure for the past month, and reading why I’m a cranky old SysAdmin for not wanting to move my datacenter to the cloud, I wanted to speak up on why in contrary, I think you’re a fool if you do.  Don’t get me wrong, I think there are perfectly valid reasons to use IaaS, there are things that don’t make sense to do in house, but running a primary (and at times a DR) datacenter in the cloud, is just waisting money and limiting your companies capabilities.  Let’s dig into why…

Basic IaaS History:

Let’s start with a little history as I know it on how IaaS was initially used, and IMO, this is still the best fit for IaaS.

I need more power… Ok, I’m done, you can have it back.

There are companies out there (not mine) that do all kinds of crazy calculations, data crunching and other compute intensive operations.  They needed huge amounts of compute capacity for relatively short periods of time (or at least that was the ideal setup).  Meaning, they were striving to get the work done as fast as possible, and for arguments sake, let’s just say their process scaled linearly as they added compute nodes.  There was only so much time, so much power, so much cooling, and so much budget to be able to house all these physical servers for solving what is in essence one big complex math equation.  What they were left with was a balancing act of buying as much compute as they could manage, without being excessively wasteful.  After all, if they purchased so much compute that they could solve the problem in a minimal amount of time, unless they were keeping those server busy, once the problem was solved, it was a waste of capital.  About 10 years ago (taking a rough guess here), AWS releases this awesome product capable of renting compute by the hour, and offering whats basically unlimited amounts of cpu / gpu power.  Now all of a sudden a company that would have had to operate a massive datacenter has a new option of renting mass amounts of compute by the hour.  This company could fire up as many compute nodes as they could afford, and not only could they solve their problem quicker, but they only had to pay for the time they used.

I want to scale my web platform on demand…. and then shrink it, and then scale it, and then shrink it.

It evolved further, if its affordable for mass scale up and scale down for folks that fold genomes, or trend the stock market, why not for running things like next generation web scale architectures.  Sort of a similar principle, except that you run everything in the cloud.  To make it affordable, and scalable, they designed their web infrastructure so that it could scale out, and scale on demand.  Again, we’re not talking about a few massive database servers, and a few massive web servers, we’re talking about tons of smaller web infrastructure components, all broken out into smaller independently scalable components.  Again the cloud model worked brilliantly here, because it was built on a premise that you designed small nodes, and scaled them out on demand as load increased, and destroyed nodes as demand dwindled.  You could never have this level of dynamic capacity affordably on prem.

I want a datacenter for my remote office, but I don’t need a full server, let alone multiples for redundancy.

At this stage IaaS is working great for the DNA crunchers and your favorite web scale company, and all the while, its getting more and more development time, more functionally, and finally gaining the attention of more folks for different use cases.  I’m talking about folks that are sick of waiting on their SysAdmins to deploy test servers, or folks that needed a handful of servers in a remote location, folks that only needed a handful of small servers in general, and didn’t need a big expensive SAN or server. Again, it worked mostly well for these folks.  They saved money by not needing to manage 20 small datacenters, or they were able to test that code on demand and on the platform they wanted, and things were good.

The delusion begins…

Fast forward to now, and everyone thinks that if the cloud worked for the genome folders, the web scale companies and finally for small datacenter replacements, then it must also be great for my relatively speaking static, large legacy enterprise environment.  At least that’s what every cloud peddling vendor and blogger would have you believe, and thus the cloud delusion was born.

Why do I call it the cloud delusion?  Simple, your enterprise architecture is likely NOT getting the same degrees of wins that these types of companies were/are getting out of IaaS.

Let’s break it down the wins that cloud offered and offers you.  In essence, if this is functionality that you need, then the cloud MAY make sense for you.

  1. Scale on demand:  Do you find your self frequently needing to scale servers by the hundreds every, day, week or even month?   Shucks, I’ll even give you same leeway and ask if you’re adding multiple hundred servers every year?   In turn are you finding that you are also destroying said servers in this quantity?  We’re trying to find out if you really need the dynamic scale on demand advantage that the cloud brings over your on prem solution.
  2. Programatic Infrastructure:  Now I want to be very clear with this from the start, while on prem may not be as advanced as IaaS, infrastructure is mostly programatic on prem, so weigh this pro carefully.  Do you find that you hate using a GUI to manage your infrastructure, or need something that you can that can be highly repeatable, and fully configurable via a few JSON files and a few scripts?  I mean really think about that.  How many of you right now are just drowning because you haven’t automated your infrastructure, and are currently head first in automating every single task you do?  If so, the cloud may be a good fit then because practically everything can be done via a script and some config files.  If however, you’re still running through a GUI, or using a handful of simple scripts, and really have no intention of doing everything through a JSON file / script, its likely that IaaS isn’t offering you a big win here.  Even if you are, you have to question if your on prem solution offers similar capabilities, and if so, whats the win that a cloud provider offers that your on prem does not.
  3. Supplement infrastructure personnel:    Do you find your infrastructure folks are holding you back?  If only they didn’t have to waste time on all that low level stuff like managing hypervisors, SANs, switches, firewalls, and other solutions, they’d have so much free time to do other things.  I’m talking about things like patching firmware, racking / unracking equipment, installing hypervisors, provisioning switch ports.  We’re talking about all of this consuming a considerable portion of your infrastructure teams time.  If they’re not spending that much time on this stuff (and chances are very high that they’re not), then  this is not going to be a big win for you.  Again, companies that would have teams busy with this stuff all the time, probably have problem number 1 that I identified.  I’d also like to add that even if this is an issue you have, there is still a limited amount of gain you’ll get out of this.  You’re still going to need to provision storage, networking and compute, but now instead of in the HW, it will simply be transferred to a CLI / GUI.  Mostly the same problem, just a different interface.  Again, unless you plan to solve this problem ALONG with problem 2, its not going to be a huge win.
  4. VM’s on demand for all:  Do you plan on giving all your folks (developers, DBA,  QA, etc.) access to your portal to deploy VM’s?  IaaS has an awesome on demand capability that’s easy to delegate to folks.  if you’re needing something like this, without having to worry about them killing your production workload, then IaaS might be great for you.  Don’t get me wrong, we can do this on prem too, but there’s a bit more work and planning involved.  Then again, letting anyone deploy as much as they want, can be an equally expensive proposition.  Also, let’s not forget problem number 2, chances are pretty high, your folks need some pre-setup tasks performed, and unless you’ve got that problem figured out, VM’s on demand probably isn’t going to work well anywhere, let alone the cloud.
  5. At least 95% of your infrastructure is going to the cloud:  While the number may seem arbitrary (and to some degree it is a guess), you need a critical mass of some sort for it to make financial sense to send you infrastructure to the cloud (if you’re not fixing a point problem).  What good is it to send 70% of your infrastructure to the cloud, if you have to keep 30% on prem.  You’re still dealing with all the on prem issues, but now your economies of scale are reduced.  If you can’t move the lions share of your infrastructure to the cloud, then what’s the point in moving random parts of it?  I’m not saying don’t move certain workloads to the cloud.  For example, if you have a mission critical web site, but everything else its ok to have an outage for, then move that component to the cloud.  However, if most of your infrastructure needs five 9’s, and you can only move 70% of it, then you’re still stuck supporting five 9’s on prem, so again, what’s the point?

Disclaimer:  Extreme amounts of snark are coming, be prepared.

Ok, ok maybe you don’t need any of these features, but you’ve got money to burn, you want these features just because you might use them at some point, everyone else is “going cloud” so why not you, or who knows whatever reason you might be coming up with for why the cloud is the best decision.  What’s the big deal, I mean you’re probably thinking you lose nothing, but gain all kinds of great things.  Well that my friend is where you’d be wrong.  Now my talking points are going to be coming from my short experience with Azure, so I can’t say these apply to all clouds.

  1. No matter what, you still need on prem infrastructure.  Maybe its not a hoard of servers, but you’ll need stuff.
    1. Networking isn’t going anywhere (should have been a network engineer).    Maybe you won’t have as many datacenter switches to contend with (and you shouldn’t have a lot if your infrastructure is modern and not greater than a few thousand VM’s), but you’ll still need access switches for you staff.  You’re going to need VPN’s and routers.  Oh, and NOW you’re going to need a MUCH bigger router and firewall (err… more expensive).  All that data you were accessing locally now has to go across the WAN, if you’re encrypting that data, that’s going to take more horsepower, and that means bigger badder WAN networking.
    2. You’re probably still going to have some form of servers on site.  In a windows shop that will be at least a few domain controllers, you’ll also have file server caching appliances, and possibly other WAN acceleration devices depending on what apps you’re running in the cloud.
    3. Well, you’ve got this super critical networking and file caching HW in place, you need to make sure it stays on.  That potentially is going to lead back to UPS’s at a minimum and maybe even a generator.  Then again, being fair, if the power is out, perhaps its out for your desktops too, so no one is working anyway.  That’s a call you need to make.
    4. Is your phone system moving to the cloud too?  No… guess you’re going to need to maintain servers and other proprietary equipment for that too.
    5. How about application “x”?  Can you move it to the cloud, will it even run in the cloud?  Its based on Windows 2003, and Azure doesn’t support Windows 2003.  What are application “X”‘s dependencies and how will they effect the application if they’re in the cloud?  That might mean more servers staying on prem.
  2. They told you it would be cheaper right, I mean the cloud saves you on so much infrastructure, so much personnel power, and it provides this unlimited flexibility and scalability that you don’t actually need.
    1. Every VM you build now actually has a hard cost.  Sorry, but there’s no such thing as “over provisioning” in the cloud.  Your cloud provider gets to milk that benefit out of you  and make a nice profit.  Yeah I can run a hundred small VM’s on a single host, those same VM’s I’d pay per in a cloud solution.  But hey, its cheaper in the cloud, or so the cloud providers have told me.
    2. Well at least the storage is cheaper, except that to get decent performance in the cloud, you need to run on premium storage and premium storage isn’t cheap (and not really all that premium either).  You don’t get to enjoy the nice low latency, high iop, high throughput, adaptive caching (or all flash) that your on prem SAN provided.  And if you want to try and match what you can get on prem, you’ll need to over-provision your storage, and do crazy in guest disk stripping techniques.
    3. What about your networking?  I mean what is one of the most expensive reoccurring  networking costs to a business?  The WAN links… well they just got A LOT more expensive.  So on top of now spending more capex on a router and firewall, you also need to pump more money into the WAN link so your users have a good experience.  Then again, they’ll never have the same sub-millisecond latency that they had when the app was local to them.
      1. No problem you say, I’ll just move my desktop to the cloud, and then you remember that the latency still exists, its just been moved from client and application, to the user interfacing with the client.  Not really sure which is worse.
        1. Even if you’re not deterred by this, now you’re incurring the costs of running your desktops in the cloud.  You know, the folks that you force 5 years or older desktops on.
    4. How many IP’s or how many NIC’s does your VM have?  I hope its one and one.  You see there are limitations (in Azure) of one IP per NIC, and in order to run multiple NIC’s per server, you need a larger VM.  Ouch…
    5. I hope you weren’t thinking you’d run exactly 8 vCPU’s and 8GB of vRAM because that’s all your server needs.  Sorry, that’s not the way the cloud works.  You can have any size VM you want, as long as its the sizes that your cloud provider offers.  So you may end up paying for a VM that has 8 vCPU and 64GB of RAM because that’s the closest fit.  But wait, there’s more…  what if you don’t need a ton of CPU or RAM, but you have a ton of data, say a file server.  Sorry, again, the cloud provider only enables a certain number of disks per vCPU, so you now need to bump up your VM size to support the disk size you need.
    6. At least with cloud, everything will be easy, I mean yeah it might cost more, but oh… the simplicity of it all.  Yep, because having a year 2005 limitation of 1TB disks just makes everything easy.  Hope you’re really good with dynamic disks, windows storage spaces, or LVM (Linux) because you’re going to need it  Also, I hope you have everything pre-thought out if you plan to stripe disks in guest.  MS has the most unforgiving disk stripping capabilities if you don’t.
    7. Snapshots, they at least have snapshots… right?  Well sort of, except its totally convoluted, not something you’d probably never want to implement for fear of wrecking your VM (which is what you were trying to avoid with the snap right?).
    8. Ok, ok, well how about dynamically resizing your VM’s?  They can at least do that right?  Yes, sort of, so long as your sizing up in a specific VM class.  Otherwise TMK, you have to rebuild once you outgrow a given VM class.  For example, the “D series” can be scaled until you reach the maximum for the “D”.  You can’t easily convert it to a “G” series in a few clicks to continue growing it.
    9. Changes are quick and non-disruptive right?  LOL, sure with any other hypervisor they might be, but this is the cloud (Azure) and from what I can see, its iffy if your VM’s don’t need to be shutdown, or even worse, if you do something that is supported hot, you may see longer than normal stuns.
    10. Ever need to troubleshoot something in the console?  Me too, a shame because Azure doesn’t let you access the console.
    11. Well at least they have a GUI for everything right?  Nope, I found I need to go drop into PS more often than not.  Want to resize that premium storage disk, that’s gonna take a powershell cmdlet.  That’s good though right, I mean you like wasting time finding the disk guid, digging into a CLI, just to resize one disk, which BTW is a powered off operation, WIN!
    12. You like being in control of maintenance windows right?  Of course you do, but with cloud you don’t get a say.

I could keep going on, but honestly I think you get the point.  There are caveats in spades when switching to the cloud as a primary (or even DR) datacenter.  Its not a simple case of paying more for features you don’t have, you lose flexibility / performance, and you pay more for it too.

Alright, but what about all those bad things they say about on prem, or things like TCO they’re trying to woo you to the cloud for.  Well lets dig into it a bit.

  1. Despite what “they” tell you, they’re likely out of touch.  Most of the cloud folks you’re dealing with, have been chewing their own dog food so long, they don’t have a clue about what exists in the on prem world, let alone dealing with your infrastructure and all its nuances.  They might convince you they’re infrastructure experts, but only THEIR infrastructure, not yours and certainly not on prem in general.  Believe me, most of them have been in their bubble for half a decade at least, and we all know how fast things change in technology, they’re new school in cloud, but a dinosaur in on prem.  Don’t misunderstand me, I’m not saying they’re not smart, I’m saying I doubt they have the on prem knowledge you do, and if you’re smart, you’ll educate yourself in cloud so you’re prepared to evaluate if IaaS really is a good fit for you and your employer.
  2. Going cloud is NOT like virtualization.  With virtualization you didn’t change the app, you didn’t’ lose control and more importantly it actually saved you money and DID provide more flexibility, scalability and simplicity.  Cloud does not guarantee any of those for a traditional infrastructure.  Or rather it may offer different benefits, that are not as equally needed.
  3. They’ll tell you the TCO for cloud is better and they MAY be right if you’re doing foolish things like.
    1. Leasing servers and swapping them every three years.  A total waste of money.  There’s very few good reason you aren’t financing a server (capex) and re-purposing that server through a proper lifecycle.  Five years is the minimum maximum life cycle for a modern server.  You have DR, and other things you can use older HW for.
    2. You’re not maxing out the cores in your server to maximize licensing costs, reduce network connectivity costs, and also reduce power, cooling and rack space.  An average dual socket 18 core server can run 150 average VM’s without breaking a sweat.
    3. Your threshold for a maxed out cluster is too low.  There’s nothing wrong with a 10:1 or even a 15:1 vCPU to pCPU ratio so long as your performance is ok.  Your milage may vary, but be honest with yourself before buying more servers based on arbitrary numbers like these.
    4. You take advice from a greedy VAR.  Do yourself a favor and just hire a smart person that knows infrastructure.  They’ll be cheaper than all the money you waste on a VAR, or cloud.   You should be pushing for someone that  is border line architect, if not an architect.
      1. FYI, I’m not saying all VARs are greedy, but more are than not.  I can’t tell you how many interviews I’ve had where I go “yeah, you got up sold”.
    5. Stop with this BS of only buying “EMC” or “Cisco” or “Juniper” or whatever your arbitrary preferred vendor is. Choose the solution based on price, reliability, performance, scalability and simplicity, not by its name.  I picked Nimble when NetApp would have been an easy, but expensive choice.  Again, see point 4 about getting the right person on staff.
    6. Total datacenter costs (Power, UPS, generator and cooling) are worth considering, but are often not as expensive as the providers would have you think.  If this is the only cost saving’s point they have you sold on, you should consider colocation first which takes care of some of that, but also incurs some of the same costs/caveats that come with cloud (but not nearly as many).  Again, I personally think this is FUD, and in a lot of cases, IT departments, let alone businesses don’t even see the bill for all of this.  Even things like DC space, if you’re using newer equipment, the rack density you get through virtualization is astounding.
    7. You’re not shopping your solution, ever.  I know folks that just love to go out to lunch (takes one to know one), and their VAR’s and vendors are happy to oblige.  If your management team isn’t pushing back on price, and lets you run around throwing PO’s like monopoly money, there’s a good chance you’re paying more for something than you need to.
    8. You suck at your job, or you’ve hired the wrong person.  Sounds a little harsh,but again, going back to point 4.  if you have the right people on staff,you’ll get the right solutions, they’ll be implemented better, and they’ll get implemented quicker.  Cloud by the way, only fixes certain aspects of this problem.
  4. They’ll tell you can’t do it better than them, they scale better, and it would cost you millions to get to their level.  They’re right, they can build a 100,000 VM host datacenter better than you or I, and they can run it better.  But you don’t need that scale, and more importantly, they’re not passing those economies of scale on to you.  That’s their profit margin.  Remember, they’re not doing this to save you money, they’re doing this to make money.   In your case, if your DC is small enough (but not too small) you can probably do it MUCH cheaper than what you’d pay for in a cloud, and it will likely run much better.
  5. They’ll tell you’ll be getting rid of a SysAdmin or two thanks to cloud.  Total BS… An average sysadmin (contrary to marketing slides) does not spend a ton of time with the mundane task or racking HW, patching hypervisors (unless its Microsoft :-)), etc.  They spend most of their time managing the OS layer, doing deployments, etc, which BTW all still need to be done in the cloud.

For now, that’s all I’ve got.  I wrote this because I was so tired of hearing folks spew pro cloud dogma from their mouthes without even having a simplistic understanding of what it takes to run infrastructure in the cloud or on prem.  Maybe I am the cranky main frame guy, and maybe I’m the one who is delusional and wrong.  I’m not saying the cloud doesn’t have its place, and I’m not even saying that IaaS won’t be the home of my DC in ten years.  What I am saying is right now, at this point in time, I see moving to the cloud as big expensive mistake if your goal is to simply replace your on prem DC.  If you’re truly being strategic with what you’re using IaaS for, and there are pain points that are difficult to solve on prem, then by all means go for it.  Just don’t tell me that IaaS is ready for general masses, because IMO, it has a long ways to go yet.

Powershell Scripting: Get-ECSVMwareVirtualDiskToWindowsLogicalDiskMapping

Building off my last function Get-ECSPhysicalDiskToLogicalDiskMapping: which took a windows physical disk and mapped it to a windows logical disk, this function will take a VMware virtual disk and map it to a windows logical disk.

This function has the following dependencies and assumptions:

  • It depends on my windows physical to logical function and all its dependencies.
  • It assumes your VM name matches your windows server name
  • It assumes you’ve pre-loaded the VMware powershell snap-in.

The basic way the function works, is it starts by getting the windows physical to logical mapping and storing that in an array.  This array houses two key peaces of information.

  • The physical disk serial number
  • The logical disk name (C:, D:, etc.)

Then we get a list of all of the VM’s disks, which of course has the same exact serial number, just formatted a little differently (which I convert in the function).

Finally, we’re going to map the windows physical disk serial number to the VMware virtual disk serial number, and add the VMware virtual disk name and the Windows Logical disk name (we don’t care about the windows physical disk, it was just used for the mapping) into the final array, and echo them out for your use.

See below for an example:

Get-ECSVMwareVirtualDiskToWindowsLogicalDiskMapping -ComputerName “YourComputerName”

VMwareVirtualDisk WindowsLogicalDisk
—————– ——————
Hard disk 1 C:
Hard disk 2 K:
Hard disk 3 F:
Hard disk 4 N:
Hard disk 5 J:
Hard disk 6 V:
Hard disk 7 W:
Hard disk 8 G:

… and there you have it, a very quick way to figure out which vmware virtual disk houses your windows drive.  You’ll find the most recent version of the function here.

Powershell Scripting: Get-ECSPhysicalDiskToLogicalDiskMapping

I figured it was about time to knock out something a little technical for a change, and I figured I’d start with this little function, which is part of a larger script that I’ll talk more about later.

There may be a time where you need to find the relationship between a physical disk drive and your logical drive.  In my case, I had a colleague ask me if there was an easy way to match a VMware disk to a Window disk so he could extend the proper drive.  After digging into it a bit, I determined it was possible, but it was going to take a little work.  One of the prerequisites is to first find which drive letter belongs to which physical disk (from windows view).

For this post, I’m going to go over the prerequisite function I built to figure out this portion.  In a later post(s) we’ll put the whole thing together.  Rather than burying this into a larger overall script (which is the way it started), I broke it out and modularized it so that you may be able to use it for other purposes.

First, to get the latest version of the function, head on over to my GitHub project located here.  I’m going to be using GitHub for all my scripts, so if you want to stay up to date with anything I’m writing, that would be a good site to bookmark.  I’m very new to Git, so bear with me as I learn.

To start, as you can tell by looking at the function, its pretty simple by in large.  It’s 100% using WMI.  A lot of this functionality existed in Windows 2012+, but I wanted something that would work with 2003+.   Now that you know its based on WMI, there’s two important notes to bear in mind:

  1. WMI does not require admin rights for local computers, but it does for remote.  Keep this in mind if you’re planning to use this function for remote calls.
  2. WMI also requires that you have the correct FW ports open for remote access.  Again, I’m not going to dig into that.  I’d suspect if you’re doing monitoring, or any kind of bulk administration, you probably already have those ports open.

Microsoft basically has the mapping in place within WMI, the only problem is you need to connect a few different layers to get from Physical to Logical mapping.  In reading my function, you’ll see that I’m doing a number of nested foreach loops, and that’s the way I’m connecting things together.  Basically it goes like this….

  1. First we need to find the physical disk to partition mappings doing the following:  WIN32_DiskDrive property DeviceID  is connected to Win32_DiskDriveToDiskPartition property of Antecedent .  ***NOTE: the DeviceID needed to be formatted so that it matched the Actecedent by adding extra backslashes “\”.
  2. Now we need to map the partition(s) on the physical disk to the logical disk(s) that exist doing the following: Win32_DiskDriveToDiskPartition property Dependent is connected to Win32_LogicalDiskToPartition  property of Actecedent.
  3. Now that we know what logical drives exist on the physical disk, we can use the following to grab all the info we want about the logical drive doing the following: Win32_LogicalDiskToPartition property of Dependent maps to Win32_LogicalDisk property of “_path”.

That’s the basic method for connecting Physical Disk to Logical Disk.  You can see that I use an array to store results, and I’ve picked a number of properties from the the physical disk and the logical disk that I needed.  You could easily add other properties in that you want to serve your needs.  And if you do… please contribute to the function.

As for how to use the function, its simple.

For local computer, simply run the function with no parameters.

LogicalDiskSize : 1000198832128
PhysicalDiskController : 0
ComputerName : PC-2158
PhysicalDiskControllerPort : 1
LogicalDiskFreeSpace : 946930122752
PhysicalDiskNumber : 0
PhysicalDiskSize : 1000194048000
LogicalDiskLetter : E:
PhysicalDiskModel : Intel Raid 1 Volume
PhysicalDiskDiskSerialNumber : ARRAY

LogicalDiskSize : 255533772800
PhysicalDiskController : 0
ComputerName : PC-2158
PhysicalDiskControllerPort : 0
LogicalDiskFreeSpace : 185611300864
PhysicalDiskNumber : 1
PhysicalDiskSize : 256052966400
LogicalDiskLetter : C:
PhysicalDiskModel : Samsung SSD 840 PRO Series
PhysicalDiskDiskSerialNumber : S12RNEACC99205W

For a remote system simply specify the “-ComputerName” paramter

Get-ECSPhysicalDiskToLogicalDiskMapping -ComputerName “pc-2158”
LogicalDiskSize : 1000198832128
PhysicalDiskController : 0
ComputerName : PC-2158
PhysicalDiskControllerPort : 1
LogicalDiskFreeSpace : 946930122752
PhysicalDiskNumber : 0
PhysicalDiskSize : 1000194048000
LogicalDiskLetter : E:
PhysicalDiskModel : Intel Raid 1 Volume
PhysicalDiskDiskSerialNumber : ARRAY

LogicalDiskSize : 255533772800
PhysicalDiskController : 0
ComputerName : PC-2158
PhysicalDiskControllerPort : 0
LogicalDiskFreeSpace : 185611976704
PhysicalDiskNumber : 1
PhysicalDiskSize : 256052966400
LogicalDiskLetter : C:
PhysicalDiskModel : Samsung SSD 840 PRO Series
PhysicalDiskDiskSerialNumber : S12RNEACC99205W

Hope that helps you down the road.  Again, this is going to be part of a slightly larger script that will ultimately map a Windows Logical Disk to a VMware Virtual Disk to make finding which disk to expand easier.

Review: 2.5 years with Nimble Storage

Disclaimer: I’m not getting paid for this review, nor have I been asked to do this by anyone.  These views are my own,  and not my employers, and they’re opinions not facts .


To begin with, as you can tell, I’ve been running Nimble Storage for a few years at this point, and I felt like it was time to provide a review of both the good and bad.  When I was looking at storage a few years ago, it was hard to find reviews of vendors, they were very short, non-informative, clearly paid for, or posts by obvious fan boys.

Ultimately Nimble won us over against the various  storage lines listed below.  Its not a super huge list as there was only so much time and budget that I had to work with .  There were other vendors I was interested in but the cost would have been prohibitive, or the solution would have been too complex.  At the time, Tintri and Tegile never showed up in my search results, but ultimately Tintri wouldn’t have worked (and still doesn’t) and Tegile is just not something I’m  super impressed with.

  • NetApp
  • X-IO
  • Equallogic
  • Compellent
  • Nutanix

After a lot of discussions and research, it basically boiled down to NetApp vs. Nimble Storage, with Nimble obviously winning us over.  While I made the recommendation with a high degree of trepidation and even after a month with the storage, wondered if I totally made an expensive mistake, I’m happy to say, it was and is still is a great storage decision.  I’m not going into why I chose Nimble over NetApp, perhaps some other time, for now this post is about Nimble, so let’s dig into it.

When I’m thinking about storage, the following are the high level area’s that I’m concerned about.  This is going to be the basic outline of the review.

  • Performance / Capacity ratios
  • Ease of use
  • Reliability
  • Customer support
  • Scaling
  • Value
  • Design
  • Continued innovation

Finally, for your reference, we’re running 5 of their 460’s, which is between their cs300 and cs500 platforms and these are hybrid arrays.

Performance / Capacity Ratios

Good performance like a lot of things is in the eye of the beholder.  When I think of what defines storage as being fast, its IOPS, throughput and latency.  Depending on your workload, more of one than the other may be more important to you, or maybe you just need something that can do ok with all of those factors, but not awesome in any one area.  To me, Nimble falls in the general purpose array, it doesn’t do any one thing great, but it does a lot of things very well.

Below you’ll find a break down of our workloads and capacity consumers.

IO breakdown (estimates):

  • MS SQL (50% of our total IO)
    • 75% OLTP
    • 25% OLAP
  • MS Exchange (30% of total IO)
  • Generic servers (15% of total IO)
  • VDI (5% of total IO)

Capacity consuming apps:

  • SQL (40TB after compression)
  • File server (35TB after compression)
  • Generic VM’s (16TB after compression)
  • Exchange (8TB after compression)

Compression?  yeah, Nimble’s got compression…

Nimble’s probably telling you that compression is better than dedupe, they even have all kinds of great marketing literature to back it up.  The reality like anything is, it all depends.  I will start by saying if you need a general purpose array, and can only get one or the other, there’s only one case where I would choose dedupe over compression, which is data sets mostly consisting of operating system and application installer data.  The biggest example of that would be VDI, but basically where ever you find your data being mostly consistent of the same data over and over.  Dedupe will always reduce better than compression in these cases.    Everything else, you’re likely better off with compression.    At this point, compression is pretty much a commodity, but if you’re still not a believer, below you can see my numbers.  Basically, Nimble (and everyone else using compression) delivers on what they promise.

  • SQL: Compresses very well, right now I’m averaging 3x.  That said, there is a TON of white space in some of my SQL volumes.  The reality is, I normally get a minimum of 1.5x and usually end up more along the 2x range.
  • Exchange 2007: Well this isn’t quite as impressive, but anything is better than nothing,   1.3x is about what we’re looking at.  Still not bad…
  • Generic VM’s: We’re getting about 1.6x, so again, pretty darn good.
  • Windows File Servers: For us its not entirely fair to just use the general average, we have a TON of media files that are pre-compressed.  What I’ll say is our generic user / department file server gets about 1.6 – 1.8 reduction.

Show me the performance…

Ok, so great, we can store a lot of data, but how fast can we access it?  In general, pretty darn fast…

The first thing I did when we got the arrays was fire up IOMeter, and tried trashing the array with a 100% random read 8k IO profile (500GB file), and you know what, the array sucked.  I mean I was getting like 1,200 IOPS, really high latency and was utterly disappointed almost instantly.    In hind sight, that test was unrealistic and unfair to some extent.  Nimble’s caching algorithm is based on random in, random out, and IOmeter was sequential in (ignored) and then attempting random out.  For me, what was more bothersome at the time, and still is to some degree is it took FOREVER before the cache hit ratio got high enough that I was starting to get killer performance.    Its actually pretty simple to figure out how long it would take a cold dataset like that to completely heat up, divide (524288000k/9600) or 15 hours.  The 524288000 is 500GB converted to KB.  The 9600 is 8k * 1200IOPS to figure out the approximate throughput at 8k.

So you’re probably think all kinds of doom and gloom and how could I recommend Nimble with such a long theoretical warm up time?  Well let’s dig into why:

  • That’s a synthetic test and a worst case test.  That’s 500GBs of 100% random, non-compressed data.  If that data was compressed for example to 250GB, it would “only” take 7.5 hours to copy into cache.
  • On average only 10% – 20% of you total dataset is actually hot.  If that file was compressed to 250GB, worst case you’re probably looking at 50GB that’s hot, and more realistic 25GB.
  • That was data that was written 100% sequential and then being read 100% random.  Its not a normal data pattern.
  • That time is how long it takes for 100% of the data to get a 100% cache hit.  The reality is, its not too long before you’re starting to get cache hits and that 1,200 IOPS starts looking a lot higher (depending on your model).

There are a few example cases where that IO pattern is realistic:

  • TempDB: When we were looking at Fusion IO cards , the primary workload that folks used them for in SQL was TempDB.  TempDB can be such a varied workload that its really tough to tune for, unless you know your app.  Having a sequential in, random out in TempDB is a very realistic scenario. 
  • Storage Migrations:  Whether you use Hyper-V or VMware, when you migrate storage, that storage is going to be cold all over again with Nimble.  Storage migrations tend to be sequential write.
  • Restoring backup data:  Most restores tend to be sequential in nature.  With SQL, if you’re restoring a DB, that DB is going to be cold.

if you recall, I highlighted that my IOmeter test was unrealistic  except in a few circumstances, and one of those realistic circumstances can be TempDB, and that’s a big “it depends”.    But what if you did have such a circumstance?  Well any good array should have some knobs to turn and Nimble is no different.  Nimble now has two ways to solves this:

  • Cache Pinning: This feature was released in NOS 2.3, basically volumes that are pinned run out of flash.  You’ll never have a cache miss.
  • Aggressive caching: Nimble had this from day one, and it was reserved for cases like this.  Basically when this is turned on, (volume or performance policy granularity TMK), Nimble caches any IO coming in or going out.  While it doesn’t guarantee 100% cache hit ratios, in the case of TempDB, its highly likely the data will have a very high cache hit ratio.

Performance woes:

That said, Nimble suffers the same issues that any hybrid array does, which is a cache miss will make it fall on its face, which is further amplified in Nimbles case by having a weak disk subsystem IMO.  If you’re not seeing at least a 90% cache hit ratio, you’re going to start noticing pretty high latency .  While their SW can do a lot to defy physics, random reads from disk is one area they can’t cheat.  When they re-assure you that you’ll be just fine with 12 7k drives, they’re mostly right, but make sure you don’t skimp on your cache.  When they size your array, they’ll likely suggest anywhere between 10% and 20% of your total data set size.  Go with 20% of your data set size or higher, you’ll thank me.  Also, if you plan to do pinning or anything like that, account for that on top of the 20%.  When in doubt, add cache.  Yes its more expensive, but its also still cheaper than buying NetApp, EMC, or any other overpriced dinosaur of an array.

The only other area where I don’t see screaming performance is situations where 50% sequential read + 50% sequential write is going on.  Think of something like copying a table from one DB to another.  I’m not saying its slow, in fact, its probably faster than most, but its not going to hit the numbers you see when its closer to 100% in either direction.  Again, I suspect part of this has to do with the NL-SAS drives and only having 12 of them.  Even with coalesced writes, they still have to commit at some point, which means, you have to stop reading data for that to happen, and since sequential data comes off disk by design, you end up with disk contention.

Performance, the numbers…

I touched on it above, but I’ll basically summarize what Nimble’s IO performance spec’s look like in my shop.  Again, remember I’m running their slightly older cs460’s, if these were cs500’s or cs700’s all these numbers (except cache misses) would be much higher.

  • Random Read:
    • Cache hit: Smoking fast (60k IOPS)
    • Cache miss: dog slow (1.2k IOPS)
  • Random Write: fast (36k IOPS)
  • Sequential
    • 100% read: smoking fast (2GBps)
    • 100% write: fast (800MBps – 1GBps)
    • 50%/50%: not bad, not great (500MBps)

Again, its rough numbers, I’ve seen higher number in all the categories, and I’ve seen lower, but these are very realistic numbers I see.

Ease of use:

Honestly the simplest SAN I’ve ever used, or at least mostly.  Carving up volumes, setting up snapshots and replication has all been super easy, and intuitive.  While Nimble provided training, I would content its easy enough that you likely don’t need it.  I’d even go so far as saying you’ll probably think you’re missing something.

Also, growing the HW has been simple as well.  Adding a data shelf or cache shelf has been as simple as a few cables and clicking “activate” in the GUI.

Why do I say mostly?  Well if you care about not wasting cache, and optimizing performance, you do need to adapt your environment a bit.  Things like transaction logs vs DB, SQL vs Exchange, they all should have separate volume types.  Depending on your SAN, this is either common place, or completely new.  I came from an Equallogic shop, where all you did was carve up volumes.  With Nimble you can do that too, but you’re not maximizing your investment, nor would you be maximizing your performance.

Troubleshooting performance can take a bit of storage knowledge in general (can’t fault Nimble for that per say) and also a good understanding of Nimble its self.  That being said, I don’t think they do as good of a job as they could in presenting performance data in a way that would make it easier to pin down the problem.  From the time I purchased Nimble till now, everything I’ve been requesting is being siloed in this tool they call “Infosite”, and the important data that you need to troubleshoot performance  in many ways is still kept under lock and key by them, or is buried in a CLI.  Yeah, you can see IOPS, latency, throughput and cache hits, but you need to do a lot of correlations.  For example, they have a line graph showing total read / write IOPS, but they don’t tell you in the line graph whether it was random or sequential.  So when you see high latency, you now need to correlate that with the cache hits and throughput to make a guess as to whether the latency was due to a cache miss, or if it was a high queue depth sequential workload.  Add to that, you get no view of the CPU, average IO size, or other things that are helpful for troubleshooting performance.  Finally, they role up the performance data so fast, that if you’re out to lunch and there was a performance problem, its hard to find, because the data is average way too quickly.


Besides disk failures (common place) we’ve had two controller failures.    Probably not super normal, but none the less, not a big deal.  Nimble failed over seamlessly, and replacing them was super simple.

Customer Support:

I find that their claim of having engineers staffing support to be mostly true.  By in large, their support is  responsive, very knowledgeable and if they don’t know the answer, they find it out.  Its not always perfect, but certainly better than other vendors I’ve worked with.


I think Nimble scales fantastically so long as you have the budget.  At first when they didn’t have data or cache shelves, I would have said they have some limits, but now a days, with their scale in any direction, its hard not to say that they can’t adapt to your needs.

That said, there is one area where I’m personally very disappointed in their scaling, which is going up from an older generation to a newer generation controllers.  In our case, running the cs460’s requires a disruptive upgrade to go to the cs500’s or cs700’s.  They’ll tell me its non-disruptive if I move my volumes to a new pool, but that first assumes I have SAN groups, and second assumes I have the performance and capacity to do that.  So I would say this is mostly true, but not always.

Value / Design:

The hard parts of Nimble…

If we just take face value, and compare them based on performance and capacity to their competitors, they’re a great value.  If you open up the black box though and start really looking at the HW you’re getting, you start to realize Nimble’s margins are made up in their HW.    A few examples…

  • Using Intel sc3500’s (or comparable) with SAS interposers instead of something like an STEC or HTST SAS based SSD.
  • Supermicro HW instead of something rebranded from Dell or HP.  The build quality of Supermicro just doesn’t compare to the others.  Again, I’ve had two controller failures in 2 years.
  • Crappy rail system.  I know its kind of petty, but honestly they have some of the worst rails I’ve seen next to maybe Dell’s EQL 6550 series.  Tooless kits have kind of been a thing for many years now, it would be nice to see Nimble work on this
  • Lack of cable management, seriously, they have nothing…

Other things that bug me about their HW design…

Its tough to understand how to power off / on certain controllers without looking in the manual.  Again, not something you’re going to be doing a lot, but still it could be better.  Their indicator lights are also slightly mis-leading with a continual blinking amberish orangeish light on their chassis.  The color is initially misleading that perhaps an issue is occurring.

While I like the convince of the twin controller chassis, and understand why they, and many other vendors use it.  I’d really like to see a full sized dual 2u rack mount server chassis.  Not because I like wasting space, but because I suspect it would actually allow them to build a faster array.  Its only slightly more work to unrack a full sized server, and the reality is I’d trade that any day for better performance and scalability (more IO slots).

I would like to see a more space conscious JBOD.  Given that they over subscribe the SAS backplane anyway, they might as well do it while saving space.  Unlike my controller argument, where more space would equal more performance, they’s offering a configuration that chews up more space, with no other value add, except maybe having front facing HDD’s.  I have 60 bay JBODs for backup that fit in 4u.  Would love to see that option for Nimble, that would be 4 times the amount of storage in about the same amount of space.

Its time to talk about the softer side of Nimble….

The web console, to be blunt is a POS.  Its slow, buggy, unstable, and really, I hate using it.  To be fair, I’m bigoted against web consoles in general, but if they’re done well, I can live with them.  Is it usable, sure, but I certainly don’t like living in it.  If I had a magic wand, I would actually do away with the web console on the SAN its self and instead, produce two things:

  • A C# client that mimic’s the architecture of VMware.  VMware honestly had the best management architecture I’ve seen (until they shoved the web console down my throat).  There really is no need for a web site running on the SAN.  The SAN should be locked down to CLI only, with the only web traffic being API calls.  Give me a c# client that I can install on my desktop, and that can connect directly to the SAN or to my next idea below.  I suspect, that Nimble could ultimately display a lot more useful information if this was the case, and things would work much faster.
  • Give me a central console (like vCenter) to centrallly manage my arrays,  I get that you want us to use infosite and while its gotten better, its still not good enough.  I’m not saying do away with info site, but let me have a central, local, fast solution for my arrays.  Heck, if you still want to do a web console option, this would be the perfect place to run it.

The other area I’m not a fan of right now, is their intelligent MPIO.  I mean I like it, but I find its too restrictive.  Being enabled on the entire array or nothing is just too extreme.  I’d much rather see it at the volume level.

Finally, while I love the Windows connection manager, it still needs a lot of work.

  • NCM should be forwards and backwards compatible, at least to some reasonable degree.  Right now its expected that it matches the SAN’s FW version and that’s not realistic.
  • NCM should be able to kick off on demand snaps (in guest) and offer a snapshot browser (meaning show me all snaps of the volume).
  • If Nimble truly want to say they can replace my backup with their snapshots, then make accessing the data off them easier.  For example, if I have a snap of a DB, I should be able to right click that DB, and say (mount a snapshot copy of this DB, with this name) and the Nimble goes off and runs some sort of workflow to make that happen.  Or just let us browse the snaps data almost like a UNC share.

The backup replacement myth…

Nimble will tell you in some cases that they have a combined backup and primary storage solution.  IMO, that’s a load of crap.  Just because you take a snapshot, doesn’t mean you’ve backed up the data.  Even if you replicate that data, it’s still not counting as a backup.  To me, Nimble can say they’ve solved the backup dilemma with their solution when they can do the following:

  • Replicate your data to more than one location
  • Replicate your data to tape every day and send it offsite.
  • Provide an easy straight forward way to restore data out of the snapshots.
  • Truncate transaction logs after a successful backup.
  • Provide a way of replicating the data to non-Nimble solution, so the data can be restored anywhere.  Or provide something like a “Nimble backup / recovery in the cloud” product.

Continued Innovation:

I find Nimble’s innovation to be on the slow side, but steady, which is a good thing.  I’d much rather have a vendor be slow to release something because they’re working on perfecting it.  In the time I’ve been a customer, they’ve released the following features post purchase:

  • Scale out
  • Scale deep
  • External flash expansion
  • Cache Pinning
  • Virtual Machine IOPS break down per volume
  • Intelligent MPIO
  • Cache Pinning
  • QOS
  • RestAPI
  • RBAC
  • Refreshed generation of SANS (faster)
  • Larger and larger cache and disk shelves

Its not a huge list, but I also know what they’re currently working on, and all I can say is, yeah they’re pretty darn innovative.

Conclusion and final thoughts:

Nimble is honestly my favorite general purpose array right now.  Coming from Equallogic, and having looked at much bigger / badder arrays, I honestly find them to be the best bang for the buck out there.  They’re not without faults, but I don’t know an array out there that’s perfect.  If you’re worried they’re not “mature enough”,  I’ll tell you, you having nothing to fear.

That said, its almost 2016 and with flash prices being where they are now, I personally don’t see a very long life for hybrid array going forward, at least not as high performance mid-size to enterprise  storage arrays.  Flash is getting so cheap, its practically not worth the saving you get from a hybrid, compared to the guaranteed performance you get from an all flash.    Hybrids were really filling a niche until all flash became more attainable, and that attainable day is here IMO.  Nimble thankfully has announced that an AFA is in the works, and I think that’s a wise move on their part.  If you have the time, I would honestly wait out your next SAN purchase until their AFA’s are out, I suspect, they’ll be worth the wait.

Backup Storage Part 5: Realization of a failure

No one likes admitting they’re wrong, and I’m certainly no different.  Being a mature person means being able to admit you’re wrong, even if it means doing it publicly, and that is what I’m about to do.

I’ve been writing this series slowly over the past few months, and during that time, I’ve noticed an increasing number of instances where my storage space virtual disks NTFS would go corrupt.  Basically, I’d see Veeam errors writing to our repository, and when investigating, I would find files not deleting (old VBK’s).  When trying to manually delete them, they would either throw some error, or they would act like they were deleted (they’d disappear), but then return only a second later.  The only way to fix this (temporarily) was to do a check disk, which requires taking the disk offline.  When you have a number of backup jobs going at anytime, this means something is going to crash, and it was my luck that it was always in middle of a 4TB+ VM.

Basically what I’m saying, that as of this date, I can no longer recommend NTFS running on Storage Spaces.  At least not on bare metal HW.  My best guess is we were suffering from bit rot, but who knows since storage spaces / NTFS can’t tell me otherwise, or at least I don’t know how to figure it out.

All that said, I suspect I wouldn’t have run into these issues had I been running ReFS.  ReFS has online scrubbing, and its looking for things like failed CRC checks (and auto repairs them) .  At this point, I’m burnt out on running storage spaces, so I’m not going to even attempt to try ReFS.  Enough v1 product evals in prod for me :-).

Fortunately I knew this might not have worked out, so my back out plan is to take the same disks / JBODS and attach them to a few RAID cards.  Not exactly thrilled about it, but hopefully it will bring a bit more consistency / reliability back to my backup environment.  Long term I’m looking at getting a SAN implemented for this, but thats for a later time.

Its a shame as  I really had high hopes for storage spaces, but like many MS products, I should have known better than to go with their v1 release.  At least it was only backup’s and not prod…

Update (09/13/2016):

I wanted to add it bit more information.  At this point it’s theory, but just incase this article is or is not dissuading you from doing storage spaces, it’s worth noting some additional information.

We had two NTFS volumes, each being 100TB in size.  One for Veeam and one for our SQL backup data.  We never had problems with the SQL backup volume (probably luck), but the Veeam volume certainly had issues.  Anyway, after tearing it all down, I was still bugged about the issue, kind of felt really disappointed about the whole thing.  In some random google, I stumbled across this link going over some of NTFS’s practical maximums.  In theory at least, we went over the tested (recommended) max volume size.  Again, I’m not one to hide things and I fess up when I screw up.  Some of the storage spaces issues may have been related to us exceeding the recommended size, and NTFS couldn’t proactively fix things in the background.  I don’t know for sure, and I really don’t have the appetite to try it again.  I know it sounds crazy to have a 100TB volume, but we had 80TB of data stored in there.  In other words, most smaller companies don’t hit that size limit, but we have no problem at all exceeding that.  If you’re wondering why we made such a large volume, it really boiled down to wanting to maximize both contiguous space as well as not wasting space.  Storage spaces doesn’t let you thin provision storage when its clustered, so if we for example would have created five 20TB LUNS instead, the contiguous space would have been much smaller and ultimately more difficult to manage with Veeam.  We don’t have that issue anymore with CommVault as it can deal with lots of smaller volumes with ease.

Anyway, while I would love to say MS shouldn’t let you format a volume larger than what they’ve tested (and they shouldn’t without at least a warning), ultimately the blame falls on me for not digging into this a bit more.  Then again, try as I may, I’ve been unable to validate the information posted on the linked blog above.  I don’t doubt the accuracy of the information, often I find fellow bloggers do a better job of explaining how to do something or conveying real world limits than the vendor.

Best of luck to you, if you do go forward with storage spaces, and if you do have questions, let me know, I worked with it in production for over a year, at a decent scale.

Quicky post: Centralized task management and workflow

Just wanted to post this quick, and I may have time to do a more through post later, but this will do for now.

In our backup solution, we had to join together two solutions, CommVault and Veeam.  Veeeam backed up the data to disk and CommVault backed the data up to tape.  The problem with this solution is the systems were unaware of each other.  We really needed a solution that could take disperate systems and jobs, and basically link them together in a workflow.  After a lot of searching we found this AWESOME product from MVP systems called “JAMS (Job automation management scheduler).

If you’re managing tons of scheduled tasks (or rather not able to manage them), want alerts, job restarts, automatic logging, support for cross platform, cross system, cross job workflows, there’s not a product out there that I’ve seen can beat it.

They’re not paying me, nor have they asked me to comment on the product.  This isn’t an official review, its just a “hey, you should really check these guys out” sort of post.

Backup Storage Part 4a: Windows Storage Spaces Gotcha’s

The pro and the con of a software defined storage is that its not a turn key solution.  Not only do  you have the power to customize your solution, you also have no choice but to design your solution.  With Storage Spaces, we figured most of this stuff out before selecting it as our new backup storage solution.  At the time, there was some documentation on storage spaces, but it was very much a learning process.  Most “how to’s” were demonstrated inside labs, so I found some aspects of the documentation to be useless, but I was able to glean enough information, to at least know what to think about.

So to get started, if you end up here because you want to build a solution like this, I would encourage you to start with this FAQ’s that MS has put together.  A lot of the questions I had were answered here.

I want to go over a number of gotcha’s with WSS before you take the plunge:

  1. If you’re used to and demand a solution that notifies you automatically of HW failures, this solution may not be right for you.  Don’t get me wrong, you can tell that things are going bad, but you’ll need to monitor them yourself.  MS has written a health script, and I myself was also able to put together a very simple health script (I’ll post it once I get my GitHub page up).
  2. WSS only polls HW health once every 30 minutes.  You can’t change this.  That means if you rip an power supply out of your enclosure it will take up to 30 minutes before the enclosure goes into an unhealthy state.  I confirmed this with MS’s product manager.
  3. Disk rebuilds are not automatic, nor are they what I would call intuitive.  You shouldn’t just rip a disk out when its bad, plop a disk in and walk away.  There is a process that must be followed in order to replace a failed disk.  BTW, this process as of now, is all powershell based.
  4. Do NOT cheap out on consumer grade HW.  Stick to the MS HCL found here.  There have been a number of stability problems listed with WSS, and its almost always has to do with not sticking with the HCL and not WSS’s reliability.
  5. This isn’t specific to WSS, but do not plan on mixing SATA and SAS drives on the same SAS backplane.  Either go all SAS or go all SATA, avoid mixing.  For HDD’s specifically, the cost is so negligible between SATA and SAS, I would personally recommend just sticking with SAS, unless you never plan to cluster.
    1. Do NOT use SAS expanders either, again, stop cheaping out, this is your data we’re talking about here.
  6. Do NOT plan on using parity based virtual disks, they’re horrible at writes, as in 90MBps tops.  Use mirroring or nothing.
  7. Do NOT plan on using dedicated hot spares, instead plan on reserving free space in your storage pool.  This is one of many advantages to WSS.  it uses free space and ALL disks to rebuild your data.
    1. If you plan on using the “enclosure awareness”, you need to reserve a drives capacity of free space * the number of enclosures you have.  So if you have 4 enclosures and you’re using 4TB drives, you must reserve 16TB of space per storage pool spanned across those enclosures.
  8. Plan on taking point 7, and also ensuring there’s at least 20% free space in your pool.  I like to plan like this.
    1. Subtract 20% right of the top of your raw capacity.  So if you have 200TB raw, that’s 160TB.
    2. As mentioned in point 7\1.  If you plan to use enclosure awareness, subtract a drive for each enclosure, otherwise, subtract at least one drives worth of capacity.  So in we had 4 enclosures and they had 4TB drives, that would be 160TB – 16TB = 144TB usable before mirroring.
  9. I recommend thick provisioned disk, unless you’re going to be VERY diligent about monitoring your pools free space.  Your storage pool will go OFFLINE if even one disk in the pool runs low on free space.  Thick provisioning prevents this, as the space will only allocate what it can reserve.
  10. Figure out your strip width (number of disks in a span) before building a virtual disk.  Plan this carefully because it can’t be reversed.  This has its own set of gotcha’s.
    1. This determines the increments of disks you need to expand your pool.  If you choose a 2 column mirror, that means you need 4 disks to grow the virtual disk.
    2. Enclosure awareness is also taken into account with columns.
    3. Performance is dependent on the number of columns.  if you have 80 disk, and you create a 2 column mirrored virtual disk, you’ll only have the read performance of 4 disks, and write performance of 2 disks (even with 80 disks).  You will however be able to grow at 4 disk increments.  However, if you create a virtual disk with 38 columns, you’ll have the read performance of 76 disks, and the write performance of 38, but you’ll need 76 disks to grow the pool.  So plan your balance of growth vs. performance.
  11. Find a VAR that has WSS experience to purchase your HW through.  Raid Inc. is who we used, but now other vendors such as Dell have also taken to selling WSS approved solutions.  I still prefer Raid Inc due to pricing, but if you want the warm and fuzzies, its good to know you now can go to Dell.
  12. Adding storage to an existing pool does not rebalance the data across the new disks.  The only way to do this is to create a new virtual disk, move the data over, and remove the original virtual disk.
    1. This is resolved in server 2016.

Those are most of the gotcha’s to consider.  I know it looks big, but I’m pretty confident that every storage vendor you look at, has their own unique list of gotcha’s.  Heck, a number of these are actually very similar to NetApp/ZFS, and other tier 1 storage solutions.

We’ll nerd out in my next post on what HW we ended up getting, what it basically looks like and why.

Backup Storage Part 3d: Traditional SAN / NAS

Of all the types of storage we looked at, this was the one type that I kept coming back to.  It was a storage technology that I had the most personal experience with, and having just finished implementing my companies first SAN, I was pretty well versed in what was out on the market.

There were other reasons as well.  Traditional storage is arguably the most familiar storage technology out there.  Even if you don’t understand SAN, you probably have a basic understanding of local RAID and to some degree, that knowledge is far more transferable to traditional storage, than say something like a scale out solution.  More disks generally = faster, RAID 10 = good for writes and 7k drives are slower than 10k drives.  I’m over simplifying on purpose, but you can see where I’m coming from.    The point is, adopting a storage solution where existing skills can transfer is a big win when you don’t have time to be an expert in everything.  On top of that, you can NOT beat the cost per GB of traditional storage nor the performance edge of traditional storage compared to all its fancier alternatives.  I’m not saying there aren’t cases where deduplication targets will offer better costs per GB, nor am I saying scale out or cloud don’t have their other winning use cases.  However, as a general purpose, do a little bit of everything good and few things great, you can’t beat traditional storage.

By now, its probably pretty apparent that we went with a traditional storage solution, and I suspect your curious as to whose solution.  Well, I think you’ll find the answer a little surprising, but let first dig in to who/what we looked at.

  1. Dell Equallogic (6510’s):  I actually have a very good bit of experience with Equallogic.  In fact at my previous employer we had been running Equallogic for a good 6 years, so there wasn’t a whole lot of research we needed to do here.  Ultimately, Equallogic, while a great SAN (during its hay day), just could’t compete with what was out in the 2014 time frame.
    1. 15TB LUN limit in the year 2014 is just stupid.  Yes I know I could have spanned volumes in the OS, but that’s messy and frankly, I shouldn’t need to do it.
    2. The cost while better than most, was still on the pricey side for backup.  Things like the lack of 4TB drive support, not being able to add trays of storage, and instead being forced to buy an entire san to expand, just make the solution more expensive then it was worth.
    3. I didn’t like that it was either RAID 10 or RAID 50.  Again its 2014, and they had no support for RAID 60.  Who in their right mind would use RAID 50 with drives greater than 1TB?  Yes I know they have RAID 6, but again, who is going to want a wide disk span with a RAID 6?  That might be fine for 24 400GB drives, but that’s totally not cool with 48 (-2 for hot spares) 3TB drives.
  2. Nimble CS 200 series:  This is our current tier 1 SAN vendor and  still my favorite all around enterprise SAN.  I REALLY wanted to use them for backup, but they ultimately weren’t a good fit.
    1. If I had to pick a single reason, it would be price.  The affordability of them as a storage vendor isn’t just performance, its also that they do inline compression.  The problem is, the data I’d be storing on them would already be compressed, so I wouldn’t be taking advantage of “usable” capacity price point and instead would be left with their raw price point.  I even calculated adding shelves of storage, 100% maxed out, and the price point was still way above what we could afford.  Add to all of that, that in reality, for backup storage, they really didn’t have a ton of capacity at the time.  24TB for the head SAN, and 33TB for each shelf, with a 3 shelf maximum.  That’s 123TB usable, at 100% max capacity.  It would have taken up 12 rack units as well, and that’s if we stacked them on top of each other (which we don’t).
    2. Performance *may* have been an issue with them for backup.  Most of Nimble performance point is based on an assumption that your working data set lives in flash. Well my backup’s working data set is somewhere between 30TB and 40TB over a weekend, good luck with that.  What this means is the spindles alone would need to be able to support the load.  While the data set would be mostly sequential, which is generally easy for disks, it would be A LOT of sequential workloads in parallel, which Nimbles architecture just wouldn’t be able to handle well IMO.  If it was all write or even all read, that may have been different, but this would be 50% to even 75% write, with 25% – 50% reads.  Add to that, some of our work load would be random, but not accessed frequently enough (or small enough) to fit in cache, so again, left to the disks to handle.  There’s a saying most car guys know, which is “there’s no replacement for displacement”, and in the case of backup, you can’t beat lots of disks.
  3. ZFS:  While I know its not true for everyone, I’d like to think most folks in IT have at least heard of ZFS by now.  Me personally, I heard about it almost 8 years ago on a message board, when a typical *NIX zealot was preaching about how horrible NTFS was, and that they were using the most awesome filesystem ever, and it was called ZFS.  Back then I think I was doing tech support, so I didn’t pay it much mind as file system science was too nerdy for my interests.  Fast forward to my SAN evaluation, and good ‘ol ZFS ended up in my search results.  Everything about it sounded awesome about it, except that it was not windows admin friendly.  Don’t mis-understand me, I’m sure there are tons of windows admins that would have no problem with ZFS, and no GUI, but I had to make sure whatever we implemented was easy enough for average admin to support. Additionally, I ideally wanted something with a phone number that I could call for support,and I really wanted something that supported had HA controllers built in.  After a lot of searching, it was clear there’s at least 50 different ways to implement ZFS, but there were only a few what I would consider enterprise implementations of ZFS.
    1. Nexenta:  I looked at these guys during my SAN evaluation.  Pretty awesome product, that was unfortunately axed from my short list almost right away when I saw that they licensed their storage based on RAW capacity and not usable capacity.  While there was tiered based pricing (the more you buy, the cheaper it gets) it was still way too expensive for backup grade storage.  For the price we would have paid for their solution, plus HW, we would have been better off with Nimble.
    2. TrueNAS:  I had heard a lot about and even tried out FreeNAS.  TrueNAS was an enterprise supported version of FreeNAS.  Unfortunately I didn’t give them much thought because I honestly couldn’t find one damn review about them on the web.  Add to that, I honestly found FreeNAS to be totaly unpolished as a solution.  So many changes required restarting services, which led to storage going offline.  Who knows, maybe these services were replaced with more enterprise frinedly services in the TrueNAS version, but I doubted it.
    3. Napp-IT:  I tried running this on OpenIndiana, and some Joyent distribution of Unix (can’t recall its name).  In either case, while I had no doubt I could get things running.  I found the GUI so un-intuitive, that I just reverted to the CLI for configuring things.  This of course defeats the purpose of looking at Napp-IT to begin with.  The only postive thing I could say about it, is that it did make it a little easier to see what the lable were for the various drives in the system.  On top of all this, no HA was natively built in to this solution (not without a third party solution) so this was pretty much doomed to fail from the begining.  if we were a shop that had a few full time *nix admins, I probably would have been all over it, but I didn’t know enough to feel 100% comfortable supporting it, and I couldn’t expect others in my team to pick it up either.
    4. Tegile:  Not exactly the first name you think of when you’re looking up ZFS, but they are in fact based on ZFS.  Again, they were a nice product, but way too expensive for backup storage.
    5. Oracle:  Alas, in desperation, knowing there was one place left to look, one place that I knew would check all the boxes, I hunkered down and called Oracle.  Now let me be clear, *I* do not have a problem with Oracle, but pretty much everyone at my company hates them.  In fact, I’m pretty sure everyone that was part of the original ZFS team hates Oracle too for obvious reasons.  Me, I was personally calling them as a last resort, because I just assumed pricing would be stupid expensive and huge waste of my time.  I thought wrong!  Seriously, at the end these guys were my second overall pick for backup storage, and they’re surprisingly affordable.  As far as ZFS is goes, if you’re looking for an enterprise version of ZFS, join the dark side and buy into Oracle.  You get enterprise grade HW 100% certified to work (and with tech support / warranty), a kick ass GUI, and compared to the likes of Tegile, NetApp and even Nexenta, they’re affordable.  Add to that, you’re now dealing with a company that owns ZFS (per say) so you’re going to end up with decent development and support.  There were only a few reason we didn’t go with Oracle
      1. My company hated Oracle.  Now, if everything else was equal, I’m sure I could work around this, but it was a negative for them right off the get go.
      2. We have a huge DC and rack space isn’t a problem in general, but man do they chew up a ton of rack space.  24 drives per 4u, wasn’t the kind of density I was looking for.
      3. They’re affordable, but only at scale.  I think when comparing them against my first choice, we didn’t break even until I hit six JBOD filled with 4TB drives.  And that assumed that my first choice was running brand new servers.
      4. Add to point 3, I couldn’t fill all their JBODs.  They either have 20 drive options or 24 drive options.  I needed to allow room for ZIL drives, which means at a minimum, one JBOD would be only populated with 20 4TB drives.
      5. Those that work with ZFS know you need LOTs of memory for metadata/caching.  In my case, we were looking at close to 400TB of raw capacity, which meant close to 400GB’s of DRAM or DRAM + SSD.  Oracle specifically in their design doesn’t use shared read cache and instead populates each controller with read cache.  In my case, that meant I was paying for cache in another controller that would rarely get used and those cache drives weren’t cheap.
    6. Windows Storage Spaces (2012 R2):  I know what you’re thinking, and let me be clear, despite what some may say, this is not the same solution as old school Windows RAID.  Windows Storage Spaces is surprisingly a very decent storage solution, as long as you understand its limitations, and work within them.  It’s very much their storage equivalent of Hyper-V.  It’s not as polished as other solution, but in many cases its likely good enough, and in our case, it was perfect for backup storage.  They ultimately beat out all the other storage solutions we looked at, which if you’ve been following this series is a lot.  As for why?  Here are a few reasons:
      1. We were able to use commodity HW, which  I know sounds so cliche, but honestly, its a big deal when you’re trying to keep the cost per TB down.  I love Dell, but their portfolio was not only too limiting, but its also expensive.  I still use Dell servers, but everything else is generic (LSI, Segate) HW.
      2. With the money we saved going commodity, we were able to not only buy more storage, but also design our storage so that it was resilient.  There’s not a single component in our infrastructure that’s not resilient.
        1. Dual Servers (Failover clustering)
        2. Dual port  10g NICS with teaming
        3. Dual Quad port SAS cards with MPIO for SAS failover in each server
        4. A RAID 10 distributed across 4 JBOD’s in a way that we can survive and entire JBOD going offline.
      3. Because these are servers running windows, we could also dual purpose them for other uses, in our case, we were able to use them for Veeam Proxy’s.  Storage Spaces is very low in memory and CPU consumption.
      4. Storage spaces is portable.  if we find that we grow out of the JBODs or the servers, its as simple as attaching the storage to new servers, or moving the disks from one JBOD into a newer one.  Windows will import the storage pool and virtual disk, and you’ll be back and running in no time.
      5. Storage spaces offered a good compromise of enabling us to build a solution on our own, but still allowed us to have a vendor that we could call in the event of a serious issue.  Sure there is more than one neck to choke now, but honestly, its not that much different than Vmware and Dell.
      6. Its surprisingly fast, or at least windows doesn’t cause overhead like you might think.  I’ll go more into my HW spec’s later, but basically I’m limited by my HW, not Windows.  As a teaser, I’m getting over 5.8GBps sequential read (that’s Bytes with a big B).  Normally I’m not a big fan of sequential IO as a benchmark, but for backup, most of what goes on is sequential IO, so its a pretty good IO pattern to demonstrate.  Again, its not the fastest solution out there, but its fast enough.

This is the final post on the types of storage we looked at.  All in all, it was pretty fun to check out all of the solutions out there, even if it was for something that was kind of on the boring side of IT.  Coming up, I’m going start going over how we tackled going from nothing, to a fairly large, fast and highly resilient storage spaces solution.

Backup Storage Part 3c: Cloud Storage

Anyone who’s heard the word “cloud” as it pertains to technology knows its a fairly nebulous term.  About the only thing people seem to grasp is that its means “not in my datacenter or home”.  When we talk about “cloud storage” the term is still fuzzy, but at least we know it has something to do with storage.  For this post, I’m going to be writing about two specific types of storage outlined below.

  • Online / Realtime storage:  This is just a name I’m going to use to describe the type of storage that you’d normally run cloud hosted VMs on.  I like to think of it as cloud SAN.  This type of storage is great for primary backup storage, both because of its performance capability and always being online.  However, do to its price, it may not be the most prudent cloud storage option for long term retention or for secondary copies.
  • Nearline or Archive storage: This storage is a kind of like tape (and depending on the vendor may be tape). You don’t write or read to this storage directly, and instead use either a virtual appliance, or some form of API / SW.  Archive and nearline storage are great options for secondary storage or for hosting secondary copies of your backups.  Other than how you interact with the storage, the only downside tends to be the potential of longer recovery times.

With two great options, what’s not to love?  You have the perfect mix of short term, fast storage for your more recent backups, and your cheap and deep storage for your secondary and extended retention copies.

Initially it sounded great, until we started digging deeper into it.  Ultimately we ended up having to pass on the solution for a number of reasons below.

  1. Cost:
    1. Let’s face it, unless you’re one of the lucky few, like us you have a tough enough time getting budget for things that might actually make your company more productive.  Trying to get your company to invest heavily in something they might never need to use, or rarely need to use is not likely to happen.
    2. This ones a toss up, but if your company prefers capex over opex, cloud is not going to be an easy battle for you.  In our case, capex is preferred, which gave cloud storage a big black eye.
    3. The storage is only one small piece of the cost of cloud options.
      1. Your network pipe is probably not big enough for all the data you need to send.  This mostly depends on your backup solution as a whole (software optimized backups with dedupe and compression may help), but I suspect you don’t have enough to send your weekly full, let alone try to recover a weekly full.  So on top of the never ending cost of cloud storage, you’re going to need to add a more expensive pipe to get the data there, and maybe even pull the data back.
      2. Potentially in addition to the network, or maybe instead of the network, you’ll need to invest in a cloud gateway or some other backup data optimizing software.  Either way, you’re going to be investing even more capex on top of your increasing opex.
  2. DR: Our new DR plan wasn’t finalized yet, which made it really difficult to pick a solution that we weren’t sure would make sense in two years.  For example, if we move our DR solution to the cloud, some of the considerations in point 1 go away, as we’ll be running our secondary site in the cloud anyway.  However, if we decided to stay in a colocation unit, while the cloud technically speaking could still work, it wouldn’t make sense to send our data to the cloud compared to just sending it to our DR site.
  3. Speed:  While point 1/3/1 (Network) if invested in correctly shouldn’t pose a bottleneck, its tough to argue against copying our data to tape from a performance perspective.  We easily saturate 5 LTO6 tape drives in parallel (off our new backup storage solution to be discussed in an upcoming post).  There’s no way that it would be cost effective to get that same level of performance out of cloud storage.
  4. Integration:  While more and more backup vendors are integrating cloud storage API’s, they’re not always free, and they’re not always good.   Veeam as an example, had a cloud solution for their product, but it was terribly inefficient (as I recall).  There was no backup optimization to the cloud.  Veeam was simply copying files for you to the cloud.  Its simple, but not efficient.  This also goes back to point 1/3/2.  We could have worked around this limitation with different backup SW, or cloud gateways, but again, we’re talking about adding cost to an already limited budget.

At the end of the day, I really wanted cloud, but financially speaking it doesn’t make much sense to backup to the cloud.  While the price of the storage its self is quiet reasonable, the cost of the pipe or SW to get the data there is what kills the solution.  I’m not saying it doesn’t make sense for others, but for us, our data was too big and the the cost would have been too high.

Factors that would change my view are below:

  1. if ISP’s prices were dropping at the same rates as cloud vendors, I could see this making cloud more affordable.
  2. If the cloud providers themselves started offering deduplication targets as part of their storage offering I think this would make a big difference.  Perhaps instead of charging what we physically consume (per GB) they could instead charge what we logically consume.  This way they still win, and so do we.

Backup Storage Part 3b: Scale Out Storage

While evaluating EMC’s DataDomain solution, our EMC rep suggested we also take a look at their Isilon product.  Not being a person to turn down a chance to check out new technology I happily obliged.

Before we get into my thoughts on Isilon, let’s go over a simplistic overview of what scale out storage is, and what makes it different from say an EMC VNX.

SANs like the EMC VNX are what I’ll refer to as more traditional storage.  Typically they’re deployed with two controllers, and each controller shares access to one or more JBODs.  Within this architecture, there are a few typical “limitations”.  I’m putting limitations in quotes as in a lot cases most folks don’t run into these limitations when a system is properly designed.

  1. A controller is really jut an x86 server and like any server it is only going to have so many PCI slots.  Just like you would in any server you need to balance the use of those PCI slots.  In most cases, you’re going to be balancing how many slots are used for host uplinks (targets) or use for storage connections (initiator).
  2. While some high end SAN’s support true active / active (and more than two controllers BTW), your average configuration is going to have one controller active and the other one waiting to take over.  Meaning, 50% of your controllers are doing nothing all day long.
    1. Some people will divide their storage up, and have both controllers hosting LUNs.  Meaning if there is 50 disks, 25 disks may be active on controller 1, and the remaining 25 will be active on controller 2.  In the case of a two controller configuration, your net worst case performance will be that of a single controller.
  3. When you max out the number of disks, CPU, or target/initiator ports that a SAN can host, you need to deploy another SAN.  At this point, its likely you’re leaving some amount of resources on an island never to be utilized again.  Maybe its free spaces, maybe its the CPU, no matter what, something is being left under-utilized.
    1. This has a negative effect when we start talking about things like file shares.  Now this means spreading your file system over non-related storage units, it can get messy quick.

This is the way most shops run, and honestly its not a bad thing.  Having some extra of something isn’t always a negative, as long as its the right extra.

So how does scale out fit into all of this?  What does any of this have to do with backup?  Well first, lets start with the basics of scale out storage.  There are few things to keep in mind.

  1. Scale out no longer uses shared disk.  There is instead 1 controller to 1 set of disks and this forms a node or a brick.  You then connect multiple nodes together over a backbone network (think a private network used for node to node communication only). This forms a scale out cluster.  Over the scale out cluster you typically have one file system, or rather one contiguous allocated amount of space.
  2. The number of nodes in a cluster is mostly limited by the number of backbone connections that can be provisioned.  In the case of a 48 port infiniband switch, that means you basically have room for 48 nodes in a single cluster.
  3. Resiliency within the cluster is maintained by providing copies or parity at not only and individual disk level, but also a node level.
  4. Typically (not always) scale out is accessed over SMB/NFS and not iSCSI or FC.

So what makes this architecture so great, and why did we look at it for backup?  Well I’m going to highlight what we learned from speaking with EMC about Isilon.

First, what makes the architecture slick.

  1. Theres a lot less waste (at scale) in this architecture.  You don’t have pools of capacity spread across multiple SANs being unused.  With scaleout, its one massive pool of storage.
    1. A cluster is made up 3 or more like nodes
    2. Node can be divided up into tiers based on their performance.  Each performance tier must have a minimum of three nodes to be added into a cluster.
  2. To grow capacity, you simply add a node into the cluster.  Depending on your resiliency scheme, you may end up adding 100% of the nodes capacity into the cluster.
    1. One big win with scaleout that most traditional SAN’s don’t have, is when you do add capacity, the data in use is re-balanced across all nodes, thus ensuring every node is being evenly used.
  3. Isilon offer some pretty slick management capabilities.  For example, being able to control the resiliency settings at not only the cluster level but also down to the folder / file level.  Maybe that archive you have doesn’t need the same resiliency as your active files.   This same management capability also applies to performance polices.  Your archive data can reside on slow spinning disk and your active data can reside on their 10k RPM tier.
  4. Performance increases linearly as you add nodes.

So what makes this potentially great for backup?

  1. The lack of wasted space means you’re able to suck 100% of the capacity from your storage (after parity).  This means no space being wasted across luns  or different SAN units.  100% space is allocated across these devices.
  2. Due to the simplicity of adding nodes hot and thus adding capacity, if you start running out of space, there’s no need to do crazy things like expand luns, move data to new luns, migrated data within a storage pool so that its rebalanced.  All this stuff goes away with scale out.
  3. The amount of space in general supported by this solution should be able to hold a lot of your data for a long period of time.
  4. Throughput is pretty good on this architecture and it only gets better as you add nodes.

All this said, no architecture is perfect and Isilon is no exception.  There were a few negatives that ultimately led to us passing on it at the time of evaluation.

  1. We planned to backup to the Isilon unit and then backup the Isilon unit to tape.  The only way to do this (correctly) was to purchase their NDMP appliance.  This wouldn’t have been an issue per say, except that the NDMP appliance IIRC has some pretty terrible backup throughput numbers.  I don’t recall the exact numbers, but I want to say something like 200MBps.  We are going to be backing up over 100TB of data, and 200MBps wasn’t going to cut it.
  2. Isilon is cool and it has a hot price tag along with it.  While I’m sure at scale the cost per GB would be decent, it just wasn’t something we could afford.  You have to buy three nodes at a minimum and we’d only have one nodes capacity.  it would have blown our storage budget and we would have ended up with a lot less capacity than other solutions.
  3. Performance does scale as you add nodes, but it takes A LOT of nodes to equal the performance of some other solutions we were looking at.  So again, at scale, this may have been a contender, but not at the levels we were thinking about.
  4. They had no cloud solution, meaning I couldn’t replicate to an Isilon unit in AWS.  This wasn’t a deal breaker but it wasn’t ideal either.
  5. Like most EMC solutions, it was mired with all kinds of licensing features and various ways to nick and dime you to death.  Half of the cool features we heard about were a la carte license features, each jacking the price up more and more.

All in all, a very cool solution that’s unfortunately too expensive, and too limiting for us.  I think if we were contending with PB’s of data, this solution would be the only right way to tackle it, but in our case, a few hundred TB can easily be managed by plenty of other solutions.