More on Virginia Tech G5 Cluster: 17.6 Tflops

daveschroeder writes "BBC World's Click Online has a video report (with text transcript) on Virginia Tech's new 1100-node dual 2.0 GHz G5 Terascale Cluster. The report quotes the performance as 17.6 Tflops. As a point of reference, the cluster would be number 2 on the most recent June Top 500 list, behind only Japan's Earth Simulator, and considerably more than doubling the performance of the current number 3 1152-node dual 2.4 GHz Xeon MCR Linux cluster. Assuming the performance figure accurately reflects the LINPACK score (which it should; since the deadline for submissions for the upcoming list of Oct 1 has already passed, one would imagine VT would quote that figure), and depending on new entries for November's upcoming list, the cluster should almost certainly rank in the top 5 - all for only US$5.2 million. The video report is available in Windows Media 9 and Real formats; the relevant portion starts at 13:00."

Better links for Windows Media

[Mwongozi]

You can watch just the report itself, no skipping required, by following the links on this page:

http://www.bbcworld.com/content/template_clickonli ne.asp?pageid=666&co_pageid=3 [bbcworld.com]

Re:Can the results be trusted?

[sakusha]

They have previously discussed this, they use error correction algorithms, no ECC RAM necessary.

Re:Twice as fast...?

[adam872]

The two clusters are different enough that making accurate comparisons is difficult. The new G5's have a more recent PCI architecture, they use Infiniband as the interconnect and it's possible that they made use of the AltiVec (though I hear that this may not be the case because of 32 bit limitations). I believe none of these apply to the Xeon's. In high speed computing, the interconnect is vital, so that alone may push this cluster ahead for the time being. I don't doubt that the individual G5 processor are bloody quick (and as a Mac user and fan, I'm kinda glad) though.

New top-500 list will be announced around Nov 18

[slyfox]

The new "top 500" list will be announced right before SC2003 [sc-conference.org] and discussed in detail at a session of SC2003 on November 18 [sc-conference.org].

Look for another (less speculative) story on Slashdot around then.

Re:It runs MacOS X !!!

[paulthomas]

Ugh.

It has been said thousands of times by now I'm sure.


Running Mac OS X does not mean running FreeBSD Mac OS X is a system of frameworks running on top of a Mach Kernel. The only thing that relates Mac OS X to FreeBSD is the userland. In addition to the userland you have: Cocoa, Carbon, Aqua, Java, etc. The FreeBSD portion is minimal.

And yes, if you want you can run this lower level unix without the rest of Mac OS X. It is called Darwin [apple.com]. It runs on Intel and PPC if you're wondering. No, this doesn't mean that Mac OS X runs on both or ever will.

Here is a short description of the BSD families [daemonnews.org].

Infiniband

[Anonymous Coward]

10 Gbps Infiniband from Mellanox

Each machine has a PCI-X Infiniband card, interconnected with several 96-port switches.

Re:I don't give a flying f*** about you

[cshotton]

A "real" troll realizes that the term derives from the verb "to troll" and not the noun for a hairy creature that lives under a bridge and eats farm animals. The operative definition of "to troll" as it applies to Internet messages is #3.

Main Entry: troll
Pronunciation: 'trOl
Function: verb
Etymology: Middle English
Date: 15th century
transitive senses
1 : to cause to move round and round : ROLL
2 a : to sing the parts of (as a round or catch) in succession
b : to sing loudly c : to celebrate in song
3 a : to fish for by trolling b : to fish by trolling in (troll lakes) c : to pull through the water in trolling (troll a lure)
intransitive senses

If you want to use the term to refer to a message that invites one to respond or otherwise lures you into a discussion, you want this definition of the noun, from Webster's:

Main Entry: troll
Function: noun
Date: 1869
: a lure or a line with its lure and hook used in trolling

For those of you who were in diapers when the Internet was created, a "troll" is a message designed to lure you into responding, to rise to the bait, so to speak. Please learn this bit of Internet lore before we have to start the canings again.

Project leader speaking at conference Oct 28

[daveschroeder]

The project leader, Dr. Srinidhi Varadarajan, will be speaking at a session entitled Building Virginia Tech's G5 Supercluster [oreillynet.com] on Oct 28 at the upcoming O'Reilly Mac OS X conference [oreillynet.com].

He'll probably reveal some of the technical details, such as the version of Mac OS X used, at that session.

Also, according to a blog [oreillynet.com] at O'Reilly:

Next year, all the little known details [about the cluster] will be revealed in a new book. By that time we'll know what the project means for supercomputing and for Apple.

Re:But it doesnt add up...?

[Aardpig]

Would anyone care to shed some light onto this?

I can shed light to this extent: a linear scaling between processors and processing power is only realized in the most idealized of situations (those known as 'embarrasingly parallel'), where each job is small and completely independent of other jobs. The funny thing about embarrasingly parallel tasks is that they do not need a fancy parallel computer; they can just as easily be accomplished on N separate 486 machines, if N is sufficently large.

The upshot? If they claim a purely-linear scaling, they are either lying, or only considering those jobs for which one can get by on a (large) Beowulf cluster of shit machines. My head is not turned by this news...

Re:Twice as fast...?

[mangu]

17.6 Tflops in 2200 processors results in 8 Gflops/processor. I don't know about the Xeon, but I have benchmarked my own 2.4 GHz Pentium4 at 6 Gflops, multiplying two 1000x1000 random matrices using Lapack. So, yes, 8 Gflops at 2.0 GHz is faster than 6 Gflops at 2.4 GHz, but only slightly. Also, there is the overhead in the matrix multiplication. The peak performance in the 2.4 GHz P4 would be 9.6 Gflops, so one can say there's no magic other than Apple marketing in the G5. The diference in performance between those two clusters probably comes from other factors than processor power.

numbers ok....reading is wrong...

[djupedal]

The 'project' uses the same amount of electricity as 3,000 average sized homes. There are many more devices deployed than just the 1100 G5s. The cooling system alone is a major power eater. Read the articles :)

Re:Twice as fast...?

[Halo-]

Yeah, it might be the same argument, but either way it is fairly pointless. This is why benchmarking is such a controversial subject. Do you measure pure operations per second? If so, which ops? Or do you measure the actual wallclock time it takes for real world programs to execute a set of common tasks? Again, which programs and which tasks? (Which doesn't even begin to get into the all-to-real problem of vendors adding hacks to screw with the benchmark ala the video card arms race....)

I don't know jack about Macs, but it wouldn't surprise me if some marketing drone had claimed that. They might have even had a few examples to back it up.

Ultimately, "speed" isn't really a singularly quantifiable entity. I can think of at least three different ways to measure speed:

1) Pure CPU operations per second. Good for hard math, but only part of the equation

2) Hardware speed across the entire system. If the CPU screams, but the memory subsystem drags ass, the usage speed is slower. I doubt VT's cluster would be very cool if it was using 9600 baud modems for interconnects. :)

3) Perceived speed. How fast does it feel like it runs? For example the Transmeta chips which "learn" and optimize will feel slower or faster depending on if the code has been run recently, but the hardware speed hasn't changed.

Re:Can the results be trusted?

[afidel]

No, ECC ram typically is just made with faster internals. As an example most ECC comodity ram is CAS2 latency whereas most generic ram is CAS3, so the ECC ram will perform exactly the same as the non-ECC ram. You can buy CAS2 non-ECC ram but it's nearly as expensive as the ECC ram. If you have a simple idiot check at the end of a complex calculation then saving the cost of going with ECC may be worth it but most clusters this large will be used on too many different projects to assume that all of them will have such checks. For an idea of how important ECC is read (a href="http://www.ibm.com/servers/eserver/pseries/c ampaigns/chipkill.pdf">This IBM whitepaper on their chipkill ECC scheme. Even normal SEC ECC ram (what most ECC ram is today) will have aproximately 900 failures per 10TB per three years. I think that IBM is right and that eventually all ram will be RAID-M, that is a RAID5 style array of redundant memory banks that are composed of ECC banks. At future densities this will be necessary because a single high energy particle will have the ability to scramble an entire memory word including it's ECC checking bits.