There are several theories rolling around , Can some one tell exactly , what is maximum throughput possible on 8 Gbps on VSP storage ?
Just need to know is it 800MB/s OR 1600 MB/s? . Is there any document or article on this ?
You are 100% correct regarding the impact of block size on the throughput of serial channels.
However you should never expect to reach the rated speed of the channel itself. You will always have some overhead due to frames that maintain the connection, and unused bandwidth due to short frames. I would not plan on getting better than 95-98% of the channel speed as actual data rate.
Thus fur an 800MB/sec FCP channel I'd expect a best case in the range of 760 to 784 MB/sec data transfer rate.
Remember it is 10 bits per byte, so 8Gb is 800MB/sec or 800,000Kb/sec. It is not 800 MiB/sec.
raga sai, It's 800/825MBps.
If you consider a 8Gb FC, this uses 8b/10b encoding, This means 8-bit of data gets encoded to 10-bits of transmitting data. i.e the 2 bits will be used for data integrity.
In an actual scenario a 1Gb of FC is 1.0625Gb/s.
Calculate the usable bandwidth as follows :-
8*1.0625 = 8.5Gbp/s
8.5* .80 = 6.8Gb
Well, depends also on the type of IO profile , what are you looking at , 100% read, 100% write, sequential/random ?
I would say if all is configured point to point and all the cabling is set well, it could be 800MBps but normally I would say that might be asking for too much, I have been testing the equipment's performance both on storage controllers and fabrics but I have never seen the true potential 800MBps , even in the test environments where the array is not at all loaded. The max on my meter would be around 550-650MBps..I tested this via actual Load simulators and test equipments like Medusa with different IO profiles + bit error insertions, not to forget the actual production like situations might have several bit errors.
On papers the values might be different, but not correct always.
Thanks for your quick response !.
For my Question, Rahul is closer .
- I don't think , we should mix i/o profile (read/write) with total throughput (bandwidth) possible on 8Gbps storage port ?
till now, my understanding is 8 Gbps port approximately at max can transfer (theoritically) 800MB/s
But , there is another argument come across in recent days is a full duplex 8 Gbps storage port can theoritically support upto maximum of 1600 MB/s
I tried through different channels , but i am not getting any confirmed answer. This was the reason for my question in this forum to see atleast some one from hds , confirm the correct value.
Hope some one , confirms / provides some official document /pointer.
Block size is the key to getting Throughput and hitting 800MB/s. It's very easy to hit 800MB/s as long as the block size is large (> 1024k).
As to the question initally asked. Typically just shy of 800,000 KB/s or 800MB/s is the practical throughput limit of an 8 Gb FC port.
760-784MB/sec falls into my definition of "Just Shy of 800"
The baud-rate of the link accommodates for the 8b/10b overhead and the throughput rate is based on the net 8bit pre-encoding/post-decoding. The limiting factor will always be the processing speeds of the back-end FPGA/ASICs/MPB's, cache and disk (irrespective of SAS/FLASH/SATA etc...) in addition to the capability of the host to process the data both ways. Having an IO size that fits right into a single frame doesn't really help. The best would be to have an IO size that spans the maximum sequence count in a single exchange whereby the FCP layer can accommodate for the entire data-transfer phase so the XFER_RDY on writes comes-back with the entire IO-size. That removes the necessity for the half-duplex chattiness inherent to SCSI where transfer of initiatives often needed on split sequences.
A FC frame itself has a max of 2112 bytes (excluding extended headers) of which 24 are header, 4 are crc, 4 SOF and 4 EOF which results in an overhead of ~0.017%
Thus on an 8G link the net theoretical speed for "user-date" would be around 799MB/s one way. The same 799 MB/s could be going the other way.
Now, obviously these are just MS-Excel theoretical numbers and I've never seen those in real life, not even close.
Just my $0.02
Same here. Real Life I saw: large block, between 128KB and 1,024KB, fiber efficiency found at 85 to 90%. Blocksize 16KB to 32KB, drops to 75% or less (worse case 66%). Dream world is nice... but only in movies such as Avatar or Terra Nova. Either ways... T-rex awaits you, then any attemtps to "even mimic the beginning of a spur" of SLA with a customer is more dangerous than Avatar or Terra Nova. ...so long. My Great Leonopteryx is awaiting me for a joy-ride.
Like your sense of humor
I also don't see many applications using bigger block sizes , mostly they use 8k -32k, 64@max, I mean most of them. So with that and FC overhead , getting the optimal values even close to 600-650 is a challenge, that's what my experience tells me. Never commit an optimal value when you calculate performance.
Hi Rahul, ...my great leonopteryx is havng lunch now, thus I can rest and respond. (R&R )
We do have internal groups discussion approach to Performance SLAs stmt.
Always a tricky mind-set.
1. Can not be rigid.
2. has to be relative to the workload types, industry-known such as:
- ...and a few others
3. Always produce SLA with a pair of thresholds
4. ...then can go even as down to looking at DIMM sizes in cache, as, smaller DIMM (not in fashion anymore), instead may produce twice the paralellism to and for cache extraction.
Thus workload cohab (see above) will create at greater or lesser degrees "turbulences", further possibly diminishing the PCB opportunities to flush fast (either ways).
My leonopteryx calls... got to go.
Nice talking to you.
I see you have been to Nomura. Well done.
Thanks Erwin and all for a good discussion on this topic !
To summarize and close this discussion , and for clarity i ignore the over head numbers:
because of limitations on back end PCB/ASIC/MPB 's etc..,
If my above conclusions are correct, I will close this thread .
The biggest limitation is the half-duplex nature of SCSI. This, in addition to the huge dependency on the type of workload, will show that the rate of MB/s would be little short of 800MB/s.
You cannot just look at the rated speed of a channel and expect it to get that as throughput. There is command and protocol latency to consider, as well as short frames, the MP speed at both ends of the channel, and the capacity of the PCB hosting the port to consider.
I recommend the whitepaper “Hitachi Virtual Storage Platform(VSP™), High Performance FICON(zHPF™), and IBM z196 Express 8S Channels a Primer” for a discussion on Fibre Channel speeds.
This is a FICON paper, which is ESCON over Fibre Channel. SCSI over Fibre Channel may have some differences, but the gist of your question is covered here. For a single port processing 4KiB blocks our VSP can reach 85K useful IOPS, or 340MB/sec. This is far short of rated speed of 800MB/sec, but this is because of the additional overhead of processing short blocks. At larger blocks the throughput increases 700MB/sec as there is inherent efficiency in larger data lengths.
The tests in this paper do not cover the single port duplex throughput, but it does show that the PCB is limited to <1200MB/sec. From tests not shown here we know that 1200MB/sec is the net capacity of a PCB due to the speed of the backplane, and this is all that you can expect from one or four ports on a PCB. Read and write workload will be loaded proportionally across the 1200MB/sec.
That means one port can do 700MB/sec of reads, but two ports can only process 1200MB/sec of reads. Alternatively two ports can do 300MB/sec of reads each, and 300MB/sec of writes each for a total of 1200MB/sec.
Therefore, don’t expect to get 1600MB/sec full duplex out of a single port. You won't be the first or last to be fooled by the burst rate.
In theory "bi-directional" = 800 MB/sec each way, or 1,600 MB/sec total.
Theory aside, my real world practical experience has been to not expect more than 800 MB/sec per 8Gb port. We have VSP's that hit 700-800 MB/sec at times, sometimes maybe 810 MB or 815MB/sec, but not sustained at above 800 MB/sec.
My background is in storage benchmarking and array testing as a vendor and as a user; I have never been able to drive an 8Gb HBA to anywhere near 1,600 MB/sec. In real life you need to size to what you can actually expect - your workload will probably not be lab tools or benchmark applications.
Thanks for your clarifications, i too in this field from past several years, but this is special requirement at one of the customer place .As you already seen from the above discussions , theoretically it might possible to reach 1600 MB/s,
But practically, we all agree the max throughput ranges from 500 - 820 MB/s. (I also agree with these numbers).
Before i close this discussion entirely , i will wait some more time to see we get some article / pointers on the above subject.
We know if there is document/article, it will be more easy to convincing the customer..
Else, i will go with Ron point reaching the back-end PCB limit, might be the possible reason !
Check the attached Graph in this thread. You will find it interesting.
Retrieving data ...