A Cost Effective High Bandwidth System

One of the biggest catalysts to computer hardware development in recent years has been the video game industry. Video adapters and GPU's are the most bandwidth hungry devices in computing today. In fact, the fastest peripheral slot in any shipping system is a Generation 1 x16 PCI Express slot. And the most common adapters that run x16 are video cards.

So, it should really be no surprise that some workstations have the capability for extraordinary external system bandwidth. One of my favorite systems to use for benchmarking is based on the Tyan S2915 (aka the Thunder n6650W). If you're feeling brave, you could build one yourself, or you can always take the route of having a number of reputable whitebox builders create one for you.

I like the S2915 for the following reasons:

1) With dual Opteron processors, this system takes full advantage of the integrated memory controllers and the two memory subsystems--one attached to each CPU via AMD's Direct Connect technology. You can either choose to use a NUMA strategy, or you can configure this system via POST setup to also use either node or bank interleaving. This in itself is pretty nice as I have observed some applications that appear to be much faster under a NUMA strategy, while others prefer the system interleaved options.

2) This system has an amazing 56 PCI Express lanes that run from the primary and secondary Nvidia chips to allow for extraordinary levels of peripheral bandwidth. This layout is even greater than a number of shipping enterprise server systems.

3) The PCI peripheral slots allow for lots of upgradeability. The system has two x16 and two x8 slots. In addition, there are two legacy PCI-X slots that can run at 133 MHz with a single adapter or 100 MHz if both slots are populated.

So, how fast is this system? Really fast in terms of bandwidth. The Sisoft Sandra memory benchmarks show that this system can transfer data at over 12,000 MB/s with 667 MHz DDR2 memory (this is processor to CPU bandwidth).

I was very fortunate to have two of these systems with me at a recent Storage Networking World Conference where I was showing off some pretty impressive Fibre Channel performance. Using four Emulex LPe11002 4 Gb/s Fibre Channel adapters in each system, I was able to transfer data between two S2915 systems at over 5.6 GB/s. This in itself shows the amazing performance of this system. However, these adapters were only running x4 PCI Express.

I can't wait for the next generation of these adapters that will run 8 Gb/s Fibre Channel and x8 PCI Express. It is overly optimistic to believe that this system can deliver at twice this level of performance with faster adapters. After all, this would be 11.2 GB/s of bandwidth, which is more than 93% of the benchmarked CPU to memory bandwidth of this system. And, that would be a stretch... However, I'd love to see how fast this system can be pushed. I don't believe I've reached its limits yet.

So, in summary, if you need a high bandwidth solution, but don't want to pay for an enterprise server, there are several workstation options out there for you to investigate. Based on my experience, the Tyan S2915 blows away the vast majority of shipping servers in bandwidth and it is a solid choice for bandwidth fanatics. However, it is not the only choice in this area, so please do your own research if you're in the market for this type of system.

I would never advise using a solution such as this for any server type applications. Although quite fast, workstations are simply not held (or tested) to the same standards as servers. They do great for demonstrations (or for any application where a reboot isn't the end of the world).