More on Virginia Tech G5 Cluster: 17.6 Tflops

daveschroeder writes "BBC World's Click Online has a video report (with text transcript) on Virginia Tech's new 1100-node dual 2.0 GHz G5 Terascale Cluster. The report quotes the performance as 17.6 Tflops. As a point of reference, the cluster would be number 2 on the most recent June Top 500 list, behind only Japan's Earth Simulator, and considerably more than doubling the performance of the current number 3 1152-node dual 2.4 GHz Xeon MCR Linux cluster. Assuming the performance figure accurately reflects the LINPACK score (which it should; since the deadline for submissions for the upcoming list of Oct 1 has already passed, one would imagine VT would quote that figure), and depending on new entries for November's upcoming list, the cluster should almost certainly rank in the top 5 - all for only US$5.2 million. The video report is available in Windows Media 9 and Real formats; the relevant portion starts at 13:00."

Better links for Windows Media

You can watch just the report itself, no skipping required, by following the links on this page:

http://www.bbcworld.com/content/template_clickonli ne.asp?pageid=666&co_pageid=3 [bbcworld.com]

Can the results be trusted?

This cluster has a lot of RAM, none of which is ECC.

Can the results be trusted?

Re:Can the results be trusted?

They have previously discussed this, they use error correction algorithms, no ECC RAM necessary.

Re:Can the results be trusted?

Yep, and they're going to be top 5. Between you and them I wonder who has the best knowledge of how to build a cost efficient cluster?

Re:Can the results be trusted?

So they bog down the software doing something that could be done in hardware?

Just because it's in hardware doesn't mean it's free. The ECC logic is going to add a small delay to each of trillions of memory accesses. Plain memory can most likely be tuned to run faster than ECC memory.

If you're running a constrained problem and can verify the results at the end, a single error check in software could consume far less overall time than the continuous ECC hardware checks. The software check would probably catch other types of errors as well (including many errors caused by software bugs).

Re:Can the results be trusted?

No, ECC ram typically is just made with faster internals. As an example most ECC comodity ram is CAS2 latency whereas most generic ram is CAS3, so the ECC ram will perform exactly the same as the non-ECC ram. You can buy CAS2 non-ECC ram but it's nearly as expensive as the ECC ram. If you have a simple idiot check at the end of a complex calculation then saving the cost of going with ECC may be worth it but most clusters this large will be used on too many different projects to assume that all of them will have such checks. For an idea of how important ECC is read (a href="http://www.ibm.com/servers/eserver/pseries/c ampaigns/chipkill.pdf">This IBM whitepaper on their chipkill ECC scheme. Even normal SEC ECC ram (what most ECC ram is today) will have aproximately 900 failures per 10TB per three years. I think that IBM is right and that eventually all ram will be RAID-M, that is a RAID5 style array of redundant memory banks that are composed of ECC banks. At future densities this will be necessary because a single high energy particle will have the ability to scramble an entire memory word including it's ECC checking bits.

Heist

Surley they only need 1099 G5s.

Yes, but,

can it defeat an iMac in Apple's Photoshop benchmarks ;-)?

Twice as fast...?

...considerably more than doubling the performance of the current number 3 1152-node dual 2.4 GHz Xeon MCR Linux cluster.

If I understand this correctly, it's saying that a G5 is more than twice as fast as a dual 2.4 GHz Xeon? (1152 dual 2.4 GHz Xeons vs 1100 dual 2.0 GHz G5s -- there are fewer G5s and they run at a slower clock speed.)

This is a pretty staggering statistic. I hadn't really believed the hype about how fast the new G5s were.

oh my god...

The BBC page of the devil!!

(look at the link URL)

There was a reception yesterday

For all the people involved in it's construction. I helped in the construction. Wouldn't have known about it if not for the Slashdot article. I don't doubt the Linpack score. Allthough the e-mail I got about the reception said the score wasen't supposed to come out for another week or so. Anybody know what went on in the reception?

mck

How fast can it copy a 17mb file?

That is the true test.

Cluster problem.

I don't wan't to start a holy war here, but what is up with you G5 cluster zealots? Ive been sitting at my freelance gig of a G5 cluster (1200 Dual G5s) for twenty minutes as it attempts to copy a 17 megabyte file from one folder to another! Twenty minutes! At home my Beowulf cluster of 100 Celerons running Linux, the same operation takes just 2 minutes, if that. While this is happening, Itunes won't work, and everything else grinds to a halt. Even vim is struggling to catch up as I type this.

Yes, I do have DMA enabled, and I am using 25K SCSI disks. My old 3.2 Ghz 64 way xeon runs faster than this G5 cluster at times. G5 zealots, flame me if you like, but I'd rather hear some intelligent reasons why I should use G5s over faster, cheaper clusters.

New top-500 list will be announced around Nov 18

The new "top 500" list will be announced right before SC2003 [sc-conference.org] and discussed in detail at a session of SC2003 on November 18 [sc-conference.org].

Look for another (less speculative) story on Slashdot around then.

Are they all running Panther?

What's been left behind is wether or not these systems are using Panther as the OS. It would seem that with this kind of performance, an Apple supplied OS -- as opposed to Yellow Dog would -- only be capable of performing well on the G5 since Panther has processor optimizations for the G5.

If the original XServers were too costly and low performance (since they came with a G4) wouldn't a G5 server (since the performance is apparently much better) be a great option for small/medium size businesses for a web/mail/database server?

Project leader speaking at conference Oct 28

The project leader, Dr. Srinidhi Varadarajan, will be speaking at a session entitled Building Virginia Tech's G5 Supercluster [oreillynet.com] on Oct 28 at the upcoming O'Reilly Mac OS X conference [oreillynet.com].

He'll probably reveal some of the technical details, such as the version of Mac OS X used, at that session.

Also, according to a blog [oreillynet.com] at O'Reilly:

Next year, all the little known details [about the cluster] will be revealed in a new book. By that time we'll know what the project means for supercomputing and for Apple.

Doubling the second fastest?

considerably more than doubling the performance of the current number 3 1152-node dual 2.4 GHz Xeon MCR Linux cluster.

Humm, isn't the second fastest ASCI Q at Los Alamos has now been measured with 13.88 TF/s?
As mentioned here: http://www.top500.org/lists/2003/06/top5.php [top500.org]

High speed interconnect?

MCR uses Quadrics - What does this cluster use?
http://doc.quadrics.com/Quadrics/QuadricsHom e.nsf/ DisplayPages/7C18E51DBC215D3E80256D5900535959

5.2 million

Could someone who knows the going rate of these top 5 supercomputers please tell me how much less expensive 5.2 million is? I know that it sounds like a lot of money to me, but I'm guessing that it is orders of magnitude cheaper than the other top computers.

Pains a Mac user

The video is available for either Windows Media Player or RealOne Player -- but not QuickTime! [apple.com]

Ouch!!!

But it doesnt add up...?

I didn't see anywhere the source of these benchmarks - I believe what they have done is multiply the SPECfp_rate_base2000 for a dual processor 2GHz G5 (which according to Apple is 15.7) [apple.com] and multiply it by 1100.

The thing is, that only comes to 17.27TFLOPS, and in addition it does assume that the original spec scores were accurate. [theregister.co.uk]

Would anyone care to shed some light onto this?

How does it not add up?

Easy: you yourself point out that 1100 * 15.7 = 17.27 ... not 17.6.

Since the call for papers for the new Top 500 list was Oct 1, and the BBC show aired on Oct 9 with a companion BBC News story [bbc.co.uk] dated Oct 12, you'd hope that VT was simply regurgitating the figure that has already been sent to the Top 500 organization.

And why are you trolling around with one of those super-old benchmarking stories? We've already established that every manufacturer does what they can to show their products in the best possible light. At least Apple documented their test [veritest.com] results [apple.com] and methods in full.

So acually, your logic doesn't make any sense: you jump to the conclusion that it's not real results - even though real results already exist and have been submitted, and the entire story is pretty much about that process, making performance figures a critical piece to get accurate - and that they must have just multiplied some benchmark number by 1100. Then, even though the subject of your own post indicates your recognition that "it doesn't add up", you still apparently assume that the results are somehow doctored, this time for the worse, and you manage to weave in one of the stories that tries to make it look like Apple lied with its benchmarks - which it didn't - which is unrelated to the current issue! How does it "assume" the original scores were accurate?? YOU are assuming that they're just multiplying. You might have been onto something if the multiplication actually came out, but it doesn't, meaning that is NOT what they did.

Bravo, +1 Troll.

Not that simple...

Quoting one paragraph out of context of the contract tells you jack squat.

The section could start off with "The following will not apply". Unlikely, but I've seen one contract that flipped negatives 3 times.

In most of the contracts I've seen, the key section would be the one on termination which outlines what survives and what doesn't. And the definitions section is usually pretty important too. Like the definition of SOFTWARE PRODUCT. I'd expect that to be the Unix source code that AT &T provided to them.

All this says is that they could terminate the contract.So, excuse me if I don't take the word of a public relations dude.

(No, IANAL, but I've had to deal with some of that stuff).

The most important benchmark

How many seti packets can it do in an hour?

I wonder...

what the frames per second would be on Q3A...Has anyone tried it yet?

For those on APPLE computers, the REAL file

http://www.bbcworld.com/content/template_clickonli ne.asp?pageid=666&co_pageid=20 [bbcworld.com]

Skip ahead to 13:00

Since the freakin' Windows Media files won't play on OS X.

simon

Power consumption

The power supply was another huge challenge. The Supercomputer uses the same amount of electricity as 3000 average sized homes.

How do 1100 dual G5s end up using the same amount of eledtricity of 3000 average sized homes? Even with the cooling and networking, this number seems way too big.

What will they do with it afterwards?

As I've asked before [slashdot.org], what will they *do* with it once they've posted their performance benchmarks?

Japan's Earth Simulator [jamstec.go.jp] will be used for climate studies. What will VT use it for?

Chip H.

BJs for Geeks

My favorite comment:

In fact the heat is so intense that ordinary air conditioning units would have resulted in 60 mph winds [bbcworld.com]

Interesting math

The VT cluster cost about $5.2 Million and get approx. 17TFlops - The NEC Earth Simulator gets 35TFlops and cost one billion dollars. That makes it 192 times more expencive. So you can build 192 VT Clusters. And then in theory get. 3.2PFlops for the same amount of money. If you detract performance for cable lenght etc. - You will most definitly get around 1PFlops.

So, you supercomputerusers out there - build a 1PFLOPS cluster NOW!

Bleaksburg

Supercomputer is #2 & Football team is #3
-Go Hokies!

Non-upgradable?

Supercomputers, like PCs, become old in technology terms within about 3 to 5 years. But whereas you or I would go to a shop and buy one, Virginia tech has to start again from scratch. That's why they are building a new multi million dollar facility opening in 2006 for the next generation of Supercomputer.

So, they are just going to junk this setup in 3 years and start over? Doesn't make sense to me. The PowerMac is always had the better processor upgrades then PC's. I'm sure PowerLogix or Sonnet would love to make 1000+ CPU upgrade cards for them. Sure, the bus speed would be slightly behind a new system at that time, but the cost of buying an upgrade versus a whole new machine is worth looking at. Maybe after the first round of upgrades they should consider scrapping it.

-m

FINALLY!!

Virginia Tech's new 1100-node dual 2.0 GHz G5 Terascale Cluster. The report quotes the performance as 17.6 Tflops.

Finally I'll be able to play SimCity 4!!

Yea but...

does it run windows?

17.6 TFLOPS is Rpeak, not Rmax!

Simple equation: 4 FPops/cycle (IBM-PPCs) * 2GHz * 1100 G5s * 2 CPUs/G5 = 17.6 TFLOPS!

No *real* Rmax linpack scores are known yet, and from what i figured the submissions on Oct 1st are just for *inclusion* in the list, real Linpack scores can be submitted till shortly before (or even on!) the conference mid-November..

This article is BS and should be removed...

P.S.: 4 FPops/cycle per clock with 2 FPUs i hear you scream - Impossible! - That's due the Multiply/Add FMAC thing that counts as 2 FPops!

If you think about it

It's amazing what a bunch of turkeys can accomplish when we set our minds to it. This makes us #3 in the nation in football and #2 in the world for computing power. Dont mess with the Hokies ;-)

When's a Tflop not a Tflop? Dan does the math

The only skepticism I have regarding this figure (and I have this skepticism as a current G3 owner who's waiting for a dual G5 that had #$%#% better ship by Tuesday like they said it would! ;) is that in the past, Apple has published figures based on single-precision floating point performance, and TOP500 rankings use a benchmark that requires double-precision.

I had this pointed out to me in May of 2002 when Apple introduced the Xserve... in this Slashdot thread [slashdot.org].

Apple's page about the G5's execution core [apple.com] states that its two FPU's are double-precision, and that it can thus "complete at least two 64-bit mathematical calculations per clock cycle." That's obviously an improvement over the G4, but let's see...

They've got 1,100 machines. Each has 2 CPU's for a total of 2,200 CPU's. Each CPU has 2 FPU's for a total of 4,400 FPU's. Each FPU is capable of doing a 64-bit (double-precision, I presume) floating-point calculation each clock cycle. The machines are running at 2GHz, or 2 billion clock cycles per second. So... 4,400 * 2 billion = 8.8 trillion 64-bit floating-point calculations per clock cycle.

To me, that's 8.8 TeraFlops, which is conveniently precisely half of the 17.6 figure. Did I forget to multiply by two, or did someone else multiply by two an extra time? Or... is the situation (most likely) that there are situations under which the G5's additional mathy bits can actually turn out a couple more FLOPs per cycle, but that's not going to be the case most of the time, and probably won't be the case running LINPACK?

Of course, it is important to point out that this is a theoretical maximum, and that things like interconnects, RAM limitations, chip failures, entropy and unauthorized use of the cluster to render hot babes in "Poser 5" will detract from it.

So... I don't think this will be the second-most-powerful machine in the world. Even relative to the previous list. I would not be at all surprised to see it in the top 5, though, and would be surprised if it didn't make the top 10.

666

I suppose I should mention that the pageid for this thing is '666' :P

1100 G5 dual 2Ghz 2*1152 Xeon dual 2.4 Ghz...

...and considerably more than doubling the performance of the current number 3 1152-node dual 2.4 GHz Xeon MCR Linux cluster.

It's a good day to be a Mac man.

2006 Supercomputer

Anyone have any idea where he was standing at the end of the video and mentioned a building would be built to house a new supercomputer? I go to VT. I know all the MACs are over at the corporate research center but I can't figure out exactly where he's standing at the end of the video.

benchmark of mathematica

if you have g5 and mathematica, can you guys go here
http://www.scientificweb.de/mathstef3.html
http://smc.vnet.net/timings40.html

download the benchmark file and report the result?

thanks.

How 'bout $ per Tflop?

Seems to me what matters most is simply how fast is it and how much did it cost?

$300,000 per Tflop - I doubt any other on the list is even close...

17.6 TFLOPS NOT Apple-BS-Marketing but just Rpeak!

To all the ones who like to scream "cheating" and "lying" trying to blame Apple:

The formula FPops/cycle * MHz * number of CPUs is the formula for all Rpeak-Values in the Top500! It's the way Rpeak is calculated! Try it yourself by dividing the listed Rpeak-numbers down and you'll end up with the FPops/cycle for each CPU-Type! ;-) Intersting: IBM-PPCs (Power4, Power3, G5) have 4 FPops/cycle, Itanic dito, x86 has 2 FPops/cycle and Alpha (!) aswell! Yes, Alpha, the FPU-Monster! ;-)

This 4 FPops/cycle "FMAC-feature" (besides the price and Altivec!) probably explains why they went with G5s in the first place!

P.S.: Did you know that test by the german Magazine C't using IBMs XLC/XLC and P4s/Xeons with Intels ICC basically proved what apple claimed with their GCC comparison? That the G4 is faster than a 3Ghz P4/Xeon Dual 3GHz in FPU but slower in Integer? And the IBM-Compiler is still in beta, mind you! ;-)

So much for cheating lying Apple, hehe! Ofcourse, Slashdot, The Register, the Inquirer, Wired and all the rest would NEVER write an article about that, a supposed "Mac-User" who's accusing Apple of cheating without having the glance of a clue is just so much better, and who are they to admit they made a mistake?

the real question...

is it snappier?

More tech info in previous interview

Hassan Aref, Dean of Engineers at Virginia Tech, answered a lot of questions about the setup of the computer during a 20 minute interview last month. Go to the Your Mac Life archives page [yourmaclife.com] and pull down the MP3 or AAC archive of the Sep 10 show. The interview itself is 1:15-1:35.

Re:Imagine...

oh wait.

Re:It runs MacOS X !!!

Ugh.

It has been said thousands of times by now I'm sure.


Running Mac OS X does not mean running FreeBSD Mac OS X is a system of frameworks running on top of a Mach Kernel. The only thing that relates Mac OS X to FreeBSD is the userland. In addition to the userland you have: Cocoa, Carbon, Aqua, Java, etc. The FreeBSD portion is minimal.

And yes, if you want you can run this lower level unix without the rest of Mac OS X. It is called Darwin [apple.com]. It runs on Intel and PPC if you're wondering. No, this doesn't mean that Mac OS X runs on both or ever will.

Here is a short description of the BSD families [daemonnews.org].

Re:How much power ?!?!

Air Conditioners, switches, etc....

more than just CPUs

Re:I don't give a flying f*** about you

A "real" troll realizes that the term derives from the verb "to troll" and not the noun for a hairy creature that lives under a bridge and eats farm animals. The operative definition of "to troll" as it applies to Internet messages is #3.

Main Entry: troll
Pronunciation: 'trOl
Function: verb
Etymology: Middle English
Date: 15th century
transitive senses
1 : to cause to move round and round : ROLL
2 a : to sing the parts of (as a round or catch) in succession
b : to sing loudly c : to celebrate in song
3 a : to fish for by trolling b : to fish by trolling in (troll lakes) c : to pull through the water in trolling (troll a lure)
intransitive senses

If you want to use the term to refer to a message that invites one to respond or otherwise lures you into a discussion, you want this definition of the noun, from Webster's:

Main Entry: troll
Function: noun
Date: 1869
: a lure or a line with its lure and hook used in trolling

For those of you who were in diapers when the Internet was created, a "troll" is a message designed to lure you into responding, to rise to the bait, so to speak. Please learn this bit of Internet lore before we have to start the canings again.

Re:17.6 Tflops sounds more like an Altivec number

And which part of the score was released by apple?

Re:17.6 Tflops sounds more like an Altivec number

And why would it be misleading? If Xeons would be used, surely the researchers would optimize for SSE2, as x87 on the Pentium 4 has been crippled [arstechnica.com].
Scientific Computing has always emphasized numerical linear algebra. An entire strain [wikipedia.org] of supercomputer processors was developed to support such requirements.

Besides, the final "score" will be produced by benchmarking with LINPACK. It's not merely a matter of taking manufacturer supplied numbers, multiplying them together, and claiming a spot.

Re:How much power ?!?!

They meant 3,000 prison-cell sized dorm rooms.

What?

1 Hollywood = 100 Oscars 1 Oscar = 100 flops therefore 100 Oscars = 10,000 flops Therefore 1 Hollywood = 10,000 flops

numbers ok....reading is wrong...

The 'project' uses the same amount of electricity as 3,000 average sized homes. There are many more devices deployed than just the 1100 G5s. The cooling system alone is a major power eater. Read the articles :)

Re:Quicktime

Excellent use of the blur filter to pretend that some of the footage was shot poorly. Wait a minute...it WAS shot poorly.
Thanks for the link. I have no immediate plans to install WMP and Real.

VT Power Plant

Yep, the numbers must be wrong.

And if not, they have a 6 megawatt coal-fired power plant [vt.edu] within spitting distance of the engineering buildings. That's 5 kilowatts per computer; plenty for either a house or a computer.

Incidently, after years of putting up with coal dust, I thought we would finally see the benefits of living next to a huge coal pile. We had a severe ice storm and most of blacksburg lacked power. but so did campus!.

SUPERCOMPUTER for the rest of us

Slap together 4-5 G5 (conceivable to be together in an office) and you have your home-brewn supercomputer cluster.

Then there's only the need of coming up with the applications to use it (besides research), apart from gobbling up seti@home data snippets for breakfast.

However, XCode and Rendezvous enable something in this front, enabling distributed seamless compilation of big projects.

Re:How they calculated the 17.6 tflops number..

Cool post. This is almost always what goes into numbers
quoted by IT groups that don't know anything about the machines they're running (I know; it's a redundant sentence). Just take the marketing drivel from the manufacturer, multiply by how many you bought, and quote it! Everything I've read from VT appears to have been synthesized that way.

I'm running a G5 at home now. I'm still wondering how a machine with a 32-bit OS "breaks the 4 GB barrier", given that it can't do anything that a 32-bit Xeon with more than 4GB of RAM can't do.

But it hasn't quadrupled my home energy bills, at least, and that include air conditioning.

Re:Behold The POWER of Apple and tremble Penguinis

blimey ... someone got hold of some cheap speed, didn't they?

Re:Not the double speed myth again.

If they went to RackSaver (http://www.racksaver.com/shop/UserCustomize3.asp? lSystemID=70) they could have created a super computer with over 20 TFlops of peak performance for that same 5 million.

Re:Video?

Posting video to slashdot, isn't that a bad idea?

And soon to be considered a felony in a courtroom near you.

Re:The press release mentions Linux for some reaso

The slide-show says MacOSX which sounds more likely, They could even mean just Darwin(The command-line totally open source part of MacOSX). I imagine that the press people meant GNU software. The slide-show also says that they are using IBM's compiler that was just released for MacOSX.

http://don.cc.vt.edu/tcfslides.pdf
page 13 of the slide has the interesting stuff.

On a side note, without ECC Ram this thing can not be trusted to give the right answer all the time, I wonder what the researchers will do about that.