Bob Madaio

Hey EMC, Welcome To The Party

Blog Post created by Bob Madaio on Aug 5, 2015

Thank you, EMC.


It’s not often that those words are written here in the HDS Community.  But this time, they are well earned.


When a vendor does something that is in the best interest of storage customers, it should be applauded, no matter how belatedly it comes or whatever the motivations. And for a long time, we’ve suggested that vendors should publish to industry standard benchmarks, and in particular the Storage Performance Council (SPC), to provide a reasonable (if imperfect) basis of comparison.


So just like when it felt strange to thank NetApp for stepping up with its latest performance benchmarking (as I did a in a post called The Tale of the Magic Benchmark), it’s now appropriate to welcome EMC to the party!


NerdParty.png


This is also one of those times where everyone should be very willing to forgive prior transgressions and allow a company to make a U-turn.  Because unlike NetApp, until these new submissions came out, EMC has stayed far away from publishing SPC benchmarks.

tumblr_inline_ngn969Paab1spdppr.jpg


The depth of EMC’s disinterest in (or distaste for?) SPC benchmarking was highlighted as far back as this July 2007 post by Chuck Hollis, EMC – and now VMware – blogger-extraordinaire entitled Does Anyone Take The SPC Seriously? 

Since then other vendors tested and posted EMC results for them (yikes) and in 2011 EMC got everyone in the industry quite excited (see articles/posts here, and here, and here…) by joining the SPC, only to do nothing.


That all changed with the recent publication of an all-flash configurations of VNX 8000 all flash (but with HDDs as system “vault drives”) and VMAX 400K. The VNX result is here, the VMAX result is here.


Competitor or not, it's fair to say that EMC posted good performance numbers.


I could pick nits about what configuration choices were made and the real world applicability of them or why they chose those particular benchmarks to cover, etc., etc., and this will undoubtedly come out in competitive sales conversations.


But all vendors choose configurations they think will showcase them in the best light.

iiwii.jpg


So congrats to EMC for posting solid numbers.  Because what’s great about such benchmark disclosures is that they provide rich information about the configurations, system response times, scalability and more that customers can use in deciding what technology best fits their needs.  Different customers will find different things most important.


Let me explain, starting with a quick look at the new EMC benchmark results. On its EMC Pulse blog (here), EMC says its VMAX 400K SPC-2 performance results bested all other systems by "close to 30 percent" on the SPC-2 benchmark.

 

You know what? For that particular metric, it's true.


Unfortunately, HDS has not posted any SPC-2 performance numbers for our new VSP G Series of storage systems. However, HP did post the 2nd highest (after EMC) SPC-2 MBPS result (here) using its version of our high-end VSP G1000 platform.  (And if you believe that there are significant differences between the HP XP7 and the Hitachi VSP G1000, I have a bridge for sale I’d like to talk to you about…)


Now, at Hitachi we don't typically celebrate second place, but as I mentioned, each customer will find different metrics valuable, so I created a quick set of graphs of a few that I found interesting and which tell a fuller story. (You may choose others.)


EMC’s system was indeed faster as an overall number (by 28%) than the HP entry, but the HP entry was over 14 times less expensive on a per TB basis and needed half as many cabinets. Maybe some customers are interested in a 28% advantage for a 14X price-per-TB concession... but I doubt you'll find many.


VMAX3 SPC-2.jpgThe good news is, that now a customer can make a more informed choice.


The same is true for the VNX SPC-1 submission. Again, strong results.


Frankly, to best the VNX 8000’s overall SPC-1 IOPs number with a “midtier” solution we will need to wait until we post our VSP G800 results. (The VSP G800 was announced in April and will begin shipping to customers this month, SPC-1 results are planned, but will take at least a few months to be published.)


However, as the VMAX/HP discussion shows, there’s more to the story than just the singular MBPS, or in this case IOPS, number. Is a customer more latency-centric or IOPS-centric?  Are they more worried about cost or performance?


For instance, if we compare the “all flash” VNX 8000 to the Hitachi HUS VM with Hitachi Accelerated Flash (our most similar published result, found here), we will find that while the maximum SPC-1 IOPS is indeed higher for the VNX solution, it is NINE TIMES more expensive on a $/TB  basis as configured.


Also, if a customer is more latency-centric, you’ll note that HUS VM does very well versus the VNX in that regard. Interestingly, both systems crossed the exact same value of .70 ms response time as the workload increases, and HUS VM had 30% more IOPS at that (admittedly arbitrary) response time threshold.


VNX8000 SPC-1.jpg

In fact, considering these are theoretically "midrange" systems, perhaps customers will deploy them further away from full saturation, and the fact that the HUS VM write response time was about ONE QUARTER that of the VNX when both systems are 50% busy would be important to a customer.


Perhaps none of the details I raise is interesting to a particular customer and they will review these results and prefer the EMC solutions. That’s entirely possible. The key is, we cannot know the exact requirements of every customer and providing more information can and should lead customers closer the the right decision FOR THEM.


Hopefully if it can help customers focus on bigger issues about how a vendor can best help their business or even just, well, I don't know, whether or not they will accept dissimilar and incompatible architectures between midtier and high-end storage solutions or would rather a common storage operating system and functionality across both.


So thank you, EMC. We might not want you at everyone one of our parties, but we're glad you showed up to this one.

Outcomes