Bob Madaio

The Tale of the Magic Benchmark

Blog Post created by Bob Madaio on Jan 30, 2015

When buying storage, getting a real sense of system performance can be a very tricky proposition. And let’s be clear, that’s how many vendors seem to like it.


To be fair, we all brag about huge IOPS "hero" numbers. We then all say, yes, but what you really care about is the response time. Or, many will offer that "with flash, performance is all pretty similar," and tell customers they should really focus on “X”… with “X” of course, being whatever thing that particular vendor thinks is the most difficult thing for the next guy to have to speak about.


For customers, getting real answers must be rather frustrating.


4411497087_89ac3ba69d_o.png

There are organizations that have tried to help, by creating uniform, audited and public benchmarks, notably Standard Performance Evaluation Corporation (SPEC) and the Storage Performance Council (SPC) that can provide a common point of review for storage systems (and other devices, in the case of SPEC.)


SPC, in particular, has created storage benchmarks that emulate real-world OTLP and high-bandwidth application environments. It may not be perfect, but it’s representative, it’s audited and it’s public.


The good news is that many vendors, including HDS, participate, which helps provide a useful basis of comparison across the many options in the market. 


That’s why I strayed into unusual territory this week on Twitter, by thanking a competitor for participating.

I'd like to thank competitor @NetApp for posting SPC-1 numbers for its new EF560. Big vendors should help customers w/ real 3rd party data.

— Bob Madaio (@bmadaio) January 28, 2015


Because the big (or the noisy) vendors in this industry should participate. We should do whatever we can to help demystify the performance discussion and provide a basis for rational comparisons.


For instance, you can now compare that brand new NetApp EF560 system with the Hitachi HUS VM all flash configuration that’s been posted for some time. Customers can see things like how the response time maps to IOPS as the system gets busier, like in this (quite crude) graph I created. (Both the NetApp and Hitachi results are found here.)

SPC-NetApp.JPG

It’s only a quick comparison, of course, but you can see that both systems are strong performers.  And that Hitachi HUS VM offers more IOPS at a similar latency as the NetApp system.  Or alternatively, can provide the same IOPS as the NetApp system's maximum, but at a noticeably lower response time.


Does that mean a customer should buy ours vs. theirs? Without knowing more about the customer's needs, I couldn't really say. Performance is clearly only one metric of many to consider.  But customers can now ask a lot better questions, I’d think.


So congrats to NetApp for stepping up to the plate. More should do it.


Because, while it’s fine for a vendor to try to educate customers on things like “Why IOPS don’t matter” as PureStorage does in a blog post, it may be better to attempt such education on what matters while also completing industry standard tests to establish a baseline of credibility. 


So here’s an idea, why not run an application-emulating public benchmark like SPC-1 on top of your attempts to educate customers on what matters? Heck, PureStorage, you're great at viral videos, turn your SPC information explanation into some fun video exercise.  I'll be sure to watch.


But PureStorage is hardly the only or worst offender.  Heck, they don’t really sell on ultimate performance anyway.


My new favorite example of vendor performance bamboozling must be EMC’s claims about the VMAX3.


As a very interested competitor, we pay pretty close attention as to what claims are being publicly made about EMC systems. And it was clear at its recent launch that the EMC marketing engine had done a masterful job pre-briefing its community with slides, positioning and performance claims prior to the announcement.


The most powerful of the systems could do 3.2M (marketing) IOPS. We saw that result in online-posted presentations from EMC Forum events posted on emc.com, and here, and here, and here, and here, and here.


These guys were prepped and ready. Their posts included cool code-names. They posted slick corporate slides. They were getting out the message.


OK, OK, we get it.  3.2M IOPS.


Except for one thing, on launch day EMC’s executives then told a different story.  6.3M IOPS. Really?  Somehow the system performance that was pervasive throughout its content was suddenly off by around 3 Million IOPS?


As expected, customers are dubious on these claims, especially given the fact that zero performance information has been publicly provided to back up them up. This simply adds to frustration and confusion in the high-end customer base and puts all storage vendors' claims in doubt.


Despite EMC's having a famous magician on stage with them at their launch, magic seems like an implausible explanation. EMCmagic.JPG


Now look, we at HDS, are sometimes imperfect in getting out technical descriptions of our performance metrics, and yes, we also lead with silly 100% random back-end read numbers as well.  But we try to help provide as much background as possible as quickly as possible.


When we launched the Hitachi Virtual Storage Platform G1000 (VSP G1000), our press release had a mixture of best-case claims and 3rd party benchmark information.  It said: “Continuing Hitachi leadership in high-performance, trusted infrastructure, the system can start small and scale block-storage throughput performance of well over 3M IOPS, over 48GB/sec of usable bandwidth, and NFS ops/sec performance of over 1.2M in unified configurations.”


Well, that “well over 3M IOPS” probably just should have said 4 Million IOPS, as we later got deeper information proving it was 3.9M+ IOPS. We didn’t provide immediate details on where it came from on announcement day, but did follow up with configuration and testing details via Hu Yoshida's pretty clearly titled blog post: “Where are the performance specs for VSP G1000?


The NFS ops/sec performance numbers were publicly verifiable on the SPEC.org website and an accompanying whitepaper, so that part was pretty straight forward.


We think this helps, and we think customers should demand this level of transparency from vendors expecting to run the most critical and high-performance applications in a customer's environment.


So while I thanked NetApp, I suppose I should also give kudos to another vendor who’s been busy with testing their systems. HP.

 

A line I don't always expect to tweet: Hey @HPstorage, congrats, that's some FAST storage you got there. http://t.co/8IPGACMF65

— Bob Madaio (@bmadaio) December 15, 2014


HP put the XP7 through the SPC-2 (bandwidth-heavy application emulation) results testing and smashed the record that was in place.  Results here.


Now, I’ll admit, my happiness for them is more than a little self serving, since as many of you know, the XP7 is, umm, REALLY similar to the VSP G1000. We’ll get around to posting our own version of that SPC-2 benchmark soon.


But even sooner, we’ll post the VSP G1000's SPC-1 OLTP emulation benchmark.  And we think it's going to be very interesting for our customers.


While we don't have magicians on staff, we do have some great engineering talent and they sure do know how to make some performance magic happen.


Stay tuned...

Outcomes