Slashdot Log In
Drooling Over VA Tech's 1100-Node G5 Cluster
Posted by
pudge
on Sat Sep 20, 2003 11:46 PM
from the more-pics-wanted dept.
from the more-pics-wanted dept.
Mr. Slurpee writes "Virginia Tech's 1100-node dual 2 GHz Apple G5 Terascale Cluster is getting racked up and ready to roar. If you're a penniless geek like me, at least there's some tech pr0n for us to drool over. There's 1100 of them ... think they could part with one?" Update: 09/22 02:55 GMT by T : Matt submits a link to this full mirror of the photos, writing "The page owner's comment on the original mirror being taken down due to bandwidth? 'Bring it on!'"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
think massive (Score:4, Funny)
Re:think massive (Score:3, Funny)
Re:think massive (Score:5, Funny)
Parent
It's slow now. Complete mirror. (Score:5, Informative)
1100 reality distortion field generators (Score:5, Funny)
Imagining each one of those came with just a little bit of Steve Job's Reality Distortion Field, someone from NASA might want to head over there and make sure that some kind of tear in space/time doesn't occur right there. With that many G5s, we don't know what level of destruction could happen.
space.. (Score:4, Insightful)
Re:space.. (Score:4, Funny)
Parent
Re:space.. (Score:4, Funny)
Yeah, except for the fact that the front is covered in a mesh of holes, with fans sucking air through them. Maybe that's why you can see clear plastic hanging down in some of the photos.
Parent
I love it (Score:4, Funny)
And I'm a Mac guy, too. I wouldn't mind wandering through that room for a while myself... though I probably would keep my pants on.
-fred
Parent
Re:HAHAHAHAH MOD PARENT UP! (Score:3, Funny)
I try to look at the slides and what do I get? (Score:5, Interesting)
Why make a website "optimized for IE", when the content of the said website is of interest to people who are probably not running IE or Windows?
Re:I try to look at the slides and what do I get? (Score:3, Interesting)
The simply created a PowerPoint presentation and converted it to "HTML". Actually, it doesn't even appear to *try* to be HTML - I have no idea what this crap is. Through their laziness, those morons managed to make their very simple page inacessible to any browser but Internet Explorer (well, neither Opera 7 nor Moz 1.4 render the page, from what I can see, maybe it works in KHTML-based browsers, but I doubt it). Oh well, Search Engines won't be able to index the content, and that'll be their loss
Re:Why expect reason in this case? (Score:3, Informative)
How long... (Score:3, Insightful)
FooGoo
Where is mine? (Score:4, Insightful)
They've got 1100+...where's mine? I ordered a Dual 2.0 GHz G5 in July....still no sight. Supposed to ship on Tuesday....but online time will tell....
Sigh...Maybe they'll loan me one if mine gets delayed!
PS--anyone got the rest of these pics? There were a TON of them...Mirror? COMPLETE?
Re:Where is mine? (Score:4, Funny)
Parent
Re:Where is mine? (Score:5, Funny)
Parent
Re:Where is mine? (Score:3, Funny)
Hm. (Score:5, Interesting)
For example, I've yet to figure out a way to effectively get a computer lab with 30 eMacs installed and configured the same way. DHCP/Netboot is slow because we only have 100mbit switches. Split CD images are slow, and Jaguar doesn't yet have free software that does that yet (besides the dd of course). I'm not sure how to keep them all updated either.
I really hope they describe how they maintain the operating system on them.
NetBoot is slow? (Score:5, Insightful)
Parent
Re:Hm. (Score:4, Informative)
Easy way to do it.
1. Set up 1 machine how you want it.
2. Get a bunch of firewire cables.
3. Hook the eMac's together using the cables. (If you can't reach with the cables, get some portable firewire drives, iPods work well with this too.)
4. Use Carbon Copy Cloner 2.2 (http://www.bombich.com/software/ccc.html) and move down the line of machines until they're all the same.
5. Go in and change HD, Network, etc names.
6. Smile because you just did something in 4-5 hours that it would take Windows users a week to do.
If you have questions, feel free to email (sethmath @ mac.com) me about it. I can walk you through if necessary.
Parent
Re:Hm. (Score:3, Informative)
1. Boot to floppy
2. Press menu option for image to install
3. Boot machine
4. Change HD, Network, etc names
I don't know what the average user/site would encounter with the WinXP authorization, but I know larger sites get blanket installation without the c
Re:Hm. (Score:4, Informative)
Except, with Ghost, you could install 1000 machines in 30 minutes - using multicast.
A couple might fail and you'd have to redo them, but if you have a 100Mbit switched network (or gig, even better) then its about 30 minutes to blast a Windows 2000 install to any number of machines.
I love macs, typing this on a PB17", but all the apple zealots out there really make me ashamed sometimes.
Macs are strong in some areas, and weak in others. If it wins in something, DONT RUB PEOPLES FACES IN IT. They don't care.
Get over it.
Parent
Re:Hm. (Score:5, Informative)
I run a lab with about 50 macs (assorted models, from 350mhz iMacs through 800mhz eMacs, and a few 1ghz G4's) - I spent a good bit of time on a solution, and it's really not as hard as this thread makes it sound.
First, I build one system and set it up *Exactly* the way I want all the others to be. I have some run-once script voodoo to set the IP address of each machine based on its Mac address, and to munge some ByHost user preferences for the built-in guest account. Then, I use Carbon Copy Cloner">Carbon Copy Cloner [bombich.com] to create an image of that machine's hard drive.
Once I have an image of the machine, I use NetRestore [bombich.com]NetRestore (by the same guy as CCC) to create a netboot image that will automatically install the master machine's HD image onto each client.
I am fortunate to have a MacOS X Server machine on which to run the NetBoot server - which is independent of the subnet's master DHCP server, I might add - but it is possible to netboot macs from other Unix machines with a bit of patching to dhcpd.
Anyhow, all in all I don't find it any more difficult to netinstall Macs than it is to do the same for Windows machines. Building the master clone image is time consuming and annoying, but it always will be for any platform.
Feel free to email me if you are interested in my machine setup voodoo script. I had to borrow some binaries from OS X Server in order to make it work. It's slowly turning into something useful as I add more functionality to it.
Parent
Interesting... A Light Just Clicked On... (Score:4, Interesting)
He had to admit they didn't have any in stock, and weren't expecting to get any from Apple for some time.
I guess I know where the dual-G5 systems are all going. Ah, well, it's all for a good cause. I hope.
Re:Interesting... A Light Just Clicked On... (Score:5, Funny)
Parent
Cocktease (Score:4, Interesting)
The pics at chaosmint are a small selection of what was originally on the site.
But to be on topic I'm suprised that Apple didn't get them Xserve G5's for the cluster. While the desktop G5's look cool it's really unneeded to use up all that space.
Re:Cocktease (Score:3, Insightful)
Plus, these guys were on a pretty strict deadline, the cluster has to be functional by November IIRC. So, they wouldn't have wanted to wait any longer than they did.
drool over this, baby! (Score:5, Informative)
1024 nodes
2048 cpus
1024 power cables
1024 Myrinet network cards
2048 fiber cables (8.8 miles)
3072 Myrinet switch ports
4096 sticks of RAM (2 Terabytes)
7168 fans
1 hard drive
1 CDROM drive
Not only do they have pictures of its assembly, they have movies [lanl.gov].
Check the web page for more stats and better quality movies.
Oh, yes, it's unclassified
Video cards... (Score:5, Funny)
Ummmm.... (Score:3, Funny)
* - I really, really hate people who make gratutious use of the word "professional" as some sort of elitist mark of supremecy
** - I would like to run in there, see if the machines are locked down, and grab as many as I can hold.
(And yes, I'm just joking, I don't want to steal anything from them and I neither have the plans nor the means to do so, it's just a joke people)
my first tech pr0n (Score:5, Interesting)
Now some statistic pr0n:
There were about 5 1/2 million Apple IIs sold [interesting-people.org], so at an average of 64k each (just a guess), that would be 343 GB of memory total. Adding up the couple of computers in the office (it's a 4 person company), we're about 1/70 of the way there. Assuming 2 140K floppy drives per computer, that would be 1.5TB of disk storage -- that would be 6 hard drives, and they would occupy less space than a single pair of old floppy drives.
Tshirts given to Terascale Volunteers (Score:5, Funny)
It is quite the fashion statement [vt.edu] :)
(Excuse the blurriness and poor lighting - crappy cam and crappy dorm lighting)
Re:Tshirts given to Terascale Volunteers (Score:5, Funny)
"Someone shelled out the cash for 1100 G5's and all I got was this lousy t-shirt"
Parent
Do you really want to know why these were cheaper? (Score:5, Informative)
The real answer is that the problems that are going to be solved with this cluster are easily parallelizable. That's the IDEA, right? 1100 machines, each running one chunk. Well, the G5, and more specifically the Altivec vector processing section of it, is SO MUCH better for processing big bites of easily parallelizable data at a time than any of the alternatives that it can run rings around any Intel or AMD machine you care to name with fewer than double the number of processors. (And in the cases of some particular kinds of calculations, it beats those, too. But you can't count on that for all your problems.)
We've seen this before a number of times... I seem to recall a gene sequencing program that was running five or six times faster on a G4 than it was on a Pentium IV of the same speed. And then there's SETI@home, which runs much faster, cycle-for-cycle, on the Mac, and doesn't even USE altivec. (Though I believe it does take advantage of the 'multiply-and-add' instruction of the PPC, which is another nice little feature.)
Altivec is an astonishingly clean and usable interface for an amazingly powerful vector processor that is, in 99% of the Macs out there, underutilized to the point that if it suddenly disappeared, most people wouldn't notice any difference at all. It's kind of a pity, really.
Basically, Intel came out with MMX (and all the later developments) in order to have a talking point on a slide presentation about their processors, about the time when competitors like AMD were starting to come forward: functionally, an awful mess, and impossibly difficult to program. (In fact, for the first few years, Intel would send programmers out to work with companies to implement MMX, because otherwise none of them would bother.
AMD came up with something that was a little less hacked together in a very short period of time, as a response to Intel. But it still wasn't pretty, at least partially because of the limitations of the archetecture, and the performance wasn't *that* much better than just doing without.
Apple (who really designed a lot of the basics themselves when it comes to Altivec, so don't think this was a Motorola invention) said, 'Hey, wow, we need something like that, in order to compete.' First they decided on a coprocessor, but that didn't fly any better with the PPC than it did with the older Macs (840av, 660av) with DSPs in them. So they sat down and came up with a really *good* spec for a set of multimedia extensions. And they've only gotten better since.
I've toyed with altivec code, and I can tell you that in one application that I wrote, one instruction (vector permute) did the work of ten or more non-altiveced instructions on four times the data per cycle. Mind you, I just did it for fun, I don't know enough about parallel computing problems to come up with anything useful... but there's some interesting stuff under the hood.
Of course, nobody is going to believe this, because as fashionable as it is to like MacOS X on slashdot these days, nobody wants to admit that, for *some* subset of problems, Mr. Jobs's reality distortion field might not be quite as much of a distortion as you might think...
-fred
Look out, Cavaliers (Score:5, Funny)
Re:Why G5s? (Score:5, Informative)
Dell - too expensive [one of the reasons for the project being so "hush hush" was that dell was exploring pricing options during bidding]
Sun (sparc) - required too many processors, also too expensive
IBM/AMD (opteron) - required twice the number of processors and was twice the price in the desired configuration; had no chassis available
HP (itanium) - ditto
Apple (IBM PPC970) - system available with chassis for lowest price
Parent
Re:Why G5s? (Score:5, Funny)
Parent
Re:Why G5s? (Score:4, Funny)
Parent
Re:Mod Parent Up (Score:4, Informative)
Parent
Re:Expensive processor vs. inexpensive processors (Score:5, Informative)
Parent
Re:Expensive processor vs. inexpensive processors (Score:5, Interesting)
Y'know, I saw this presentation a few days ago. I wasn't there, I saw it on the net. Anyway, this bullet point stuck out then - like, what are they talking about?
For one, how come it required twice the number of processors? From the benchmarks I've seen Opterons normally whup the G5, or are at least very competitive on paticularly G5 optimised code. Certainly not out by a factor of two, anyway.
And no chassis? What the hell does this mean? You can get 1U, 2U and 4U beast Opteron boxes from the likes of, well, IBM for one. As mentioned above.
It's not even like the kinda ropey nature of 64 bit Linux comes into play either because, well, there is no 64 bit OS X - unless VT know something we don't (which is always possible).
So, yeah, I think someone decided to buy all the G5's made for a month and just set up the project to make it happen. This "achitectural options" thing is horseshit.
Dave
Parent
Re:Expensive processor vs. inexpensive processors (Score:5, Informative)
Perhaps for their benchmarks, the G5 was 2x the performance of the Opteron. Have you taken into consideration the Altivec processor, which happens to be 128bit in size? Any vector processing will be enhanced greatly by the powerful nature of the G5 in general, and especially when using Altivec optimized code. Couple this with IBM's XLC auto-vectorizing C compiler, and I wouldn't be surprised if Altivec did wipe SSE2/3D!Now; it's been discussed before that Altivec is a superior solution to MMX/MMX2/SSE, and SSE2, so there's no reason to doubt that when you pump up the FSB from 167MHz->1GHz, pump up the CPU from 1.4GHz->2.0GHz, on the PowerPC architecture, that Altivec doesn't become the most powerful SIMD solution in commodity computing.
Re: Chassis
It may be a time of research vs time to market discrepancy; IE, at the time VT was requesting bids, there were no Opteron chassis announced or available, whilst Apple may have had at 95% completion, barring an actual press release and announcement. Like, simultaneous to the release of the G5 there are no IBM PPC 970 machines, yet both companies use the same CPU.
Re: OS X
Yeah, there is a 64 bit X. It's called OS X Panther, and there's a 64 bit aware X called 10.2.7, and the libraries for Altivec have been 128bit for years now, so all 10.2.7 really added was... 64 bit pointers and memory addresses, really.
To recap: Altivec makes a big difference. Having immediately available machines makes a difference. Having a lower price point per performance per machine makes a difference (each node, including AC + networking + ram only costs about $4,727, which is $1,600 lower than an identically specced stock dual G5 with 4GB of ram!), as well as supportability of OS X vs Linux or, heaven forbid, Windows 2k... And yes, OS X for these machines are at least 64 bit enough to address 8GB of ram, and the OS has *always* been able to manipulate 128 bit data, as well as 64 bit data.
Parent
Re:Expensive processor vs. inexpensive processors (Score:5, Interesting)
- I don't know a whole lot about a blade center, but there doesn't seem to be a place to plug in the high-speed interconnects. Also, it runs on Intel chips that run hotter and do less work than the G5, especially when AltiVec gets involved, which is usually why you build a computer this size; vector processing. I'm also guessing the required configuration needed resale value to students at the end of life for the project/system.
- That's absoloutly true. When you need technical details about Linux you have to dig. When you have a question about OS X's guts, I'd guess you call Apple and have a conference call with all the coders (at least at this level of purchase/prestige). Could you imagine trying to get Linus, and all the other code writers for Linux and the supporting libraries and utilites on the phone at once?
Parent
Re:Expensive processor vs. inexpensive processors (Score:4, Insightful)
No one in their right mind would try to argue that one couldn't build a home-grown system for less. But with optical ports? FW 400 and 800? Gigabit ethernet? USB 2.0? And with said home-grown machines, when the NIC goes bad in one, or a memory slot goes bad in another, who do you call? The NIC or mainboard manufacturer? So you what, keep a list of all your machines, give 'em i.d. numbers or whatever, itemize the guts and who made what (mainboard, NIC, RAM, CPU, HDD, etc.) of each, and hope to make sense of it all when stuff starts to fail? Me, if I was in charge of it, it would make sense to me to farm it all out to one company, and then when something breaks there is one number that I have to call.
Also, lets not forget that this is probably going to be used for research, and if it involves vectors, then AltiVec is the SIMD for you.
Of course, being human, my opinion is suspect.
(tig)
Parent
Re:Small Schools (Score:3, Insightful)
When you choose between going to a large, research-oriented school and going to a smaller school, you're essentially making a trade-off between resources and personal attention. Bigger schools have more and deeper resources, but it can be tough for undergrads to have much significant interaction with professors, particularly in the first year or two. Smaller schools may not offer the same variety of courses, or get huge
Re:Don't believe the hype (Score:3, Informative)
Depends what you mean by "as well as"... That only applies if you aren't talking about heat output, power requirements, cooling required, decent case design, ease of servicing. Then, for the programs you are using, things like an extra-fast bus, large CPU cache, and posibility of huge ammounts of RAM, must not be important at all to you.
So, sure, if those 8 things are not to be considered at all, then sure, you can say that
Re:Don't believe the hype (Score:3, Interesting)
The g5 is the IBM power4 in a lighter configuration. ITs the fastest desktop level cpu.
You could probably build a dual smp pIV which will perform close but the G5 has better fpu's and is cheaper in an smp system then xeon based top of the line PIV. This is why they chose the G5.
Also I do not
Re:Joke Goes Here (Score:3, Funny)
NO! DON'T SHOOT!
*BANG*
Re:Waste of Space (Score:4, Insightful)
Parent