Slashdot Log In
Xgrid Agent for Unix
Posted by
pudge
on Wed Jun 23, 2004 06:22 PM
from the beowulf-strikes-again dept.
from the beowulf-strikes-again dept.
mac-diddy writes "Someone on Apple's mailing list for Xgrid, Apple's clustering software, just announced an 'Xgrid agent for Linux and other Unix platforms' available for download. There are still some issues being worked on like large file support, but it does allow you to simply add a Unix node to your existing Xgrid cluster. Just goes to show that when companies embrace open standards and code, the world doesn't fall apart."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
My Experience (Score:5, Interesting)
(http://artlu.net/ | Last Journal: Wednesday June 09 2004, @02:32PM)
Time to find the download.
GroupShares Inc. [groupshares.com] - A Free Online Investment Community
Re:My Experience (Score:5, Informative)
(http://www.white-mountain.org/jamie/)
Problem type: The problem may not be well suited to running on a bunch of PCs (especially when the agent app isn't allowed to take 100% of the machine's resources to accimplish the task) over typical office networks. Basically if the app needs to communicate frequently with other nodes, or if a huge data set is involved (or both), latency or bandwidth issues might outweigh the possible advantage of putting more CPUs to work.
Security: The data may be highly sensitive, in which case you might not want to put it on ordinary desktop PCs that might have untrustworthy users, spyware, etc.
Configuration: The configuration of your office's PCs may vary enough to make the cost of getting a companywide desktop cluster working unacceptably high. You'd have to pick a few target configurations and settle for that. Hopefully drivers and such wouldn't matter as much as CPU, RAM, disk, and OS version, but there are still companies that are just now getting their desktops updated to Win2K. There's also the headache of installing yet another required application on a large number of heterogeneous machines, which is virtually guaranteed to result in confusing installation problems. Oops, our app crashes if the user has this or that service pack installed. Oops, our app requires strong encryption. You could build your app on top of some sort of moderately portable framework or VM or whatever but that will have system requirements too, and probably will have some surprising gotchas when deployed in a real-world environment.
How many clusters (Score:5, Interesting)
Re:How many clusters (Score:5, Funny)
Re:How many clusters (Score:5, Funny)
You forgot, "as long as they don't run Windows!"
Re:How many clusters (Score:4, Informative)
Re:How many clusters (Score:5, Informative)
imagine (Score:4, Funny)
Re:imagine (Score:5, Funny)
Re:imagine (Score:5, Funny)
Mixed Company (Score:4, Insightful)
(http://home.san.rr.com/jasno | Last Journal: Wednesday February 25 2004, @07:51PM)
Re:Mixed Company (Score:5, Informative)
(http://www.crfh.net/ | Last Journal: Tuesday November 14 2006, @02:47PM)
Not really. Everyone uses network byte order for communication, so you won't have more overhead in a mixed system than you would in a homogenous system.
Re:Mixed Company (Score:5, Informative)
(http://x86vmm.blogspot.com/ | Last Journal: Thursday June 24 2004, @01:34PM)
The real tragedy is when you have homogenously little endian machines; e.g., a network that only has PCs on it. An integer gets byteswapped twice to end up in exactly the same byte order it was all along.
Re:Mixed Company (Score:5, Interesting)
(http://www.bgb.cc/garrett/)
Re:Mixed Company (Score:4, Informative)
(http://slashdot.org/~joe_bruin/ | Last Journal: Wednesday April 14 2004, @09:25PM)
not quite.
first, i think you mean "ntohs" (and ntohl and friends).
second, they are not macros. they are, in fact, real functions (in glibc, bsd libc, and windows' winsock library). i'd imagine it's the same on macs.
third, a macro that does nothing is not expanded to a NOP, it is simply removed by the preprocessor.
so, assuming the macs are conforming to bsd networking standards, ntohs is required to be a function, so there is still a function call per conversion (which is much more costly than doing the actual byteswap).
The real tragedy is when you have homogenously little endian machines; e.g., a network that only has PCs on it. An integer gets byteswapped twice to end up in exactly the same byte order it was all along.
a real high performance implementation (ie, the kernel) would not use ntohl, it would implement a similar byteswap macro. a byteswap can be done on x86 in one instruction, so it is fairly trivial to do.
Re:Mixed Company (Score:4, Informative)
Plenty of power to be had.... (Score:4, Interesting)
(http://www.easyautoinsuranceguide.com/)
http://www.rtp.org/index.cfm?fuseaction=in_the_ne
--
9 Gmail invitations availiable [retailretreat.com]
Kinda Cool (Score:5, Insightful)
Everything is better clustered...
How would that work? (Score:5, Funny)
[Me] Good morning, computer. How are you?
[iMac] PRETTY GOOD. I SOLVED A VEXING SCIENTIFIC PROBLEM LAST NIGHT.
[Me] Oh is that so.
[iMac] YEAH. I FOUND A SOLUTION TO THE HEISENBERG-BERTELLSMAN PROTEIN FOLDING DELIMMA.
[Me] Huh.
[iMac] THE ANSWER TURNED OUT TO BE 42.
[Me]
[iMac] OK
Re:How would that work? (Score:5, Funny)
[Me] Ok. Also, could you not use so many caps? It's like YELLING.
I've been dying to know.... (Score:5, Interesting)
(http://www.numbski.com/ | Last Journal: Thursday May 26 2005, @10:44PM)
If I use XGrid on the two, what kind of performace could I use it for day to day?
Faster compiles of applications would be the first thought. Any usefulness, say running photoshop? How about Quake? MAME?
Re:I've been dying to know.... (Score:5, Insightful)
Xgrid's main benefit is in "grunt work" calculations that aren't necessarily needed immediately. Things like SETI@Home or Folding@Home would be the sort of thing Xgrid excels at: throw some data out, have it processed, get it back when it's done.
While Apple has made clustering drop-dead easy, it's really not targeted at the home or small-business user, and the potential uses are pretty limited in that field.
Re:I've been dying to know.... (Score:5, Informative)
Not so, not so.
If your problem is embarrassingly parallel, chances are you can use Xgrid to run it right now.
For example, let's say you're rendering a 3D animation. (I haven't done real 3D work since the PowerAnimator days, so pardon me of some of my jargon is antiquated.) You've got a scene file on which you can run a render command. A command-line argument tells the renderer which frame to render.
No problem. Just use use Xgrid's Xfeed plugin. Xfeed lets you set up a job that runs a single command with a variety of command-line arguments. You tell Xfeed that you want to run the "render" command with "-f" and the numbers 1 through 720.
Xgrid goes to the first available machine on the grid and says, "Run render -f 1." Then it goes to the second machine and says, "Run render -f 2." And so on, until there are no available machines. Then it waits until a machine becomes available and says, "Run render -f n."
As each output file (a frame, in this case) becomes available, Xgrid (the client application itself, I mean) collects them in whatever directory you specified when you submitted the job.
The cool part comes when you realize that this isn't a cluster. It's a grid. That means machines can come and go as they please. If this job is running overnight, when I come in the next morning and sit down at my workstation, the agent on my computer stops the job and de-registers itself. The job goes back in the controller's queue for processing on whatever the next available machine is.
And you don't have to have any special software for this. It can be done right now with the tools that already exist in Preview 2.
Re:I've been dying to know.... (Score:4, Interesting)
(http://xydyx.com)
Re:I've been dying to know.... (Score:5, Informative)
So could someone please inform me (Score:2, Insightful)
Large corporations then download and use these products to increase productivity, get better results without paying a cent, but possibly making themselves even richer in the process. This isn't a troll, i'm just after an answer. I'm not saying OSS is bad, but i'm curious as to what motivates developers.
In general it makes life easy (Score:5, Informative)
After a while they cease to become fun to write, and you'd rather just get on with writing code that does something instead of infrastructure. By using and contributing to OSS projects, you can use the same code no matter what company you end up at. Because the code is portable it can become part of the package you can offer to a potential employer - they not only get an employee but potentially one that can producive almost right away because they are familiar with the tools they'll be using, with no cost to the company for said tools.
So it makes life easier for you, less re-work. And it makes life easier for employers, as they get richer products sooner. And if the employee becomes really proficient at a widely used OSS project they can write their own way through consulting or training.
Re:So could someone please inform me (Score:5, Interesting)
(http://www.tupps.com/)
Instead of buying a product that is 95% of what I want I can take a OSS package that is 90% of the way there and pay a developer to customise it to exactly my needs. Now I have a solution that is perfect for my business, maybe given something back to the OSS community. While if I had bought the product I would probably have to change my business to use the product. The company now is also free of licensing and upgrade issues. Also they do not have to worry about the vendor going out of business or introducing a new version with no support for the old version.
If you think of software as tools for business rather than something that a developer trys to sell OSS makes a lot more sense.
Re:So could someone please inform me (Score:5, Insightful)
The mistake I see in every Microsoft attack on OSS and the great fallacy behind every purchased white-paper that predicts that OSS will destroy the economy is that writing and selling software is only a very small part of the economy! Most of the economy is involved in creating real, tangible things like cars and planes and food, etc, etc. Most of the economy is not involved in endlessly copying and selling the same pattern of bits.
OSS creates tools that promise to improve the creation of many, many things on this planet and improve the prosperity of all. The only ones threatened by this are those that have made a business of monoplizing ideas. Ideas that are so easy to duplicate or recreate that they are deliberately trying to setup and use the force of law to keep people from producing ideas on their own.
OSS is really a "paradigm shift". This phrase has been used so emptily so many times by senseless marketing droids that it has lost impact over the years. But it is here, it is now, and it is unstoppable. How can they stop it? We have the source!
Re:So could someone please inform me (Score:5, Insightful)
(http://eddy.me.uk/ | Last Journal: Thursday July 08 2004, @01:59AM)
In the proprietary model the software is becoming worth less and less. 5 years ago run time licenses accounted for over 80% of the income of commercial software provider companies, now you will be lucky to see it account for 40% and it is going down rapidly. The rest being made up of support, training and other services.
However, the cost of producing software is the same, and what is more, it is an upfront cost. You cannot get money for it until after you have paid a programmer to write it.
Open source takes the above to a logical conclusion. As software is becoming relatively worthless (as far as run-time licenses go) you do not lose by giving the software away for free, and if you Open Source it you have available a 90% solution from free software out there before you begin thus cutting down on the production costs.
It is not about "giving stuff away" or people "not paying a cent" to use your software, it is about facilitating an extremely cost effective way for which software companies can provide services to the customer by using open source predecessors, and passing the benefit on to successors.
Re:So could someone please inform me (Score:5, Insightful)
For instance, I need a special library for an app. And none of the off-the shelf ones exactly match it. So I write it.
Now, I find out that other people have a similar problem. So I think to myself "well, I already got my ROI, so to speak. I solved my problem. So now I'll put this software out as open source and see what happens".
And people use the software in ways I didn't think of. They give suggestions on how it might be better. A few send in patches. Suddenly my solution is an even better solution, at no cost to me.
On the flip side: I download an open source library. It works okay, but there are some bugs and it needs a little refactoring. It will cost $1000 in labor to fix this library, vs. $5000 to write it from scratch. So I do it and send the patch to the author. The author is happy (free patch), I'm happy ($4000 worth of code for free), and I don't have to re-do my fixes in the next version! I sure wish commercial software worked that way.
A lot of folks make it seem that OSS is a bunch of people working for others, for free, like communists or something. Not true
Of course, you don't have to explain *how* OSS works. Just look and see that it exists and is self-sustaining, that's enough to prove that *something* about it works!
It Doesn't Show That At All! (Score:3, Insightful)
Just goes to show that when companies embrace open standards and code, the world doesn't fall apart.
Don't get me wrong, I support open standards/code, but it doesn't show any such thing if this linux client has only just been released. I bet Apple, and others for that matter, will be watching sales of Mac machines for use in clusters. If they drop because everyone starts using linux PCs, then Apple will probably not try this again.
Re:It Doesn't Show That At All! (Score:5, Insightful)
Re:It Doesn't Show That At All! (Score:5, Informative)
Home cluster (Score:4, Interesting)
but.... (Score:5, Funny)
But the world hasn't fallen apart using Microsoft either...oops, I said that outloud....
Re:but.... (Score:5, Funny)
GridEngine (Score:2, Interesting)
http://gridengine.sunsource.net
Why another technology (Score:1, Flamebait)
(http://www.qpsf.edu.au/~jc118215)
It would fit right in as it can run on top of JBoss and prefers a *nix environment.
Either that or OpenMosix... but not another new one.
great job (Score:2, Insightful)
Probably a silly question but... (Score:2, Interesting)
(http://www.tyler.mcadams.com/)
Re:Probably a silly question but... (Score:5, Informative)
(http://www.novajo.ca/)
(I wrote the xgridagent).
As the other poster said, XGrid does not care what the binary does (so it can be smp aware, multi-threaded, whatever). However, the xgridagent itself is not explicitly smp aware, but it is multi-threaded. Each task is started in its own thread and depending on the OS(?) I guess they could spread to other CPUs. The other aspect of the question is "Does the Unix XGrid agent support MPI like Apple's GridAgent for OS X?". It does not and I can't say for sure how difficult it would be to support it. However, since all communication is done via the XGrid protocol, I don't see what would prevent it from being implemented. BUt other things need to be done first.
The most pressing issue is to fix the annoying "large message" issue which makes the agent hang (while it waits forever for the controller to accept more frames). I am convinced it is trivial, I just don't know enough about BEEP to fix it. I am hoping somebody who knows BEEP will take a look at xgridagent-profile.c and fix the xgridagent_SengMSG() function and send me the patch.
Daniel Côté
Why bother? (Score:2, Informative)
There are many other open source cluster/queuing systems available.
The one I prefer is OpenPBS [openpbs.org]. It works very well for engineering compute clusters, and there are many different resource schedulers available which use the PBS job and node management system.
It's for ad hoc cluster creation... (Score:5, Informative)
Good for home use too. (Score:5, Interesting)
Considering Apple's ease-of-use for heavyweight *NIX apps this would empower more people to have more computing resources available rather than the big fish out there - schools with low budgets would be able to stretch their capabilities that bit further. And so on.
Re:Good for home use too. (Score:5, Interesting)
(http://haven.parodius.com/)
embracing? (Score:5, Insightful)
I'm not disputing that Apple released Darwin source code. But before you start cheering, keep in mind that Darwin started out as open source: the CMU Mach kernel and bits and pieces of BSD. And it's not like Apple made a big sacrifice in releasing a kernel that looks and feels like half a dozen other open source kernels.
Re:embracing? (Score:5, Informative)
(Last Journal: Wednesday March 10 2004, @01:15PM)
If Apple breaks this intentionally (meaning not for adding significant, enhanced functionality) in their next release, I will stand with you as an anti-Apple nay-saying zealot and deride them all up and down /.
-Potentially recovering Mac zealot (it's so hard with WWDC right around the corner :-( )
wake up! (Score:2, Insightful)
shall I quote from the download page? yes, yes I shall
Quote:
Several notes on compilation:
1. If you use this for anything other than testing, you are insane.
2. The configure script isn't great: it does not check for all compatibility issues and might even fail to run properly without telling you.
I'll assume that Pudge is just another Michael in disguise, endlessly posting over hyped BS articles that are easily refuted.
Car Thief (Score:2)
(http://slashdot.org/~Doc%20Ruby/journal | Last Journal: Thursday March 31 2005, @01:48PM)
GarageBand (Score:1)
A chance to nominate an app for parallelisation (Score:1, Informative)
Apple and a third-party partner are looking to target a few key
applications with the hope of developing parallel versions that would
benefit from computational clusters. As many of you know,
embarrassingly parallel algorithms like BLAST are easily written to
take advantage of clusters. There is a large set of problems, however,
where this is not the case. We would like to find some of these more
difficult applications and find a way to parallelize them using some
interesting technologies developed by our partner.
I'd like to solicit feedback from the members of these mailing lists
with respect to choosing two or three "killer applications" that, if
parallelized, would present an immediate value to their respective
users. We have a few in mind, but I'd like to leave the question
open-ended. Any science is equally applicable -- bioinformatics,
molecular dynamics, physics, engineering, etc. We would prefer to work
with open source applications.
Feel free to reply to me directly, or to the entire list.
Regards,
Matt
--
Matt MacInnis
Research and HPC Manager
Higher Education
Apple Computer, Inc.
Office 408-974-6322 / Mobile 408-203-1001
Re:it's about time... (Score:1, Funny)
Re:Apple embraces opensource? (Score:5, Informative)
Re:Apple embraces opensource? (Score:5, Informative)
(http://www.mothership.co.nz/)
I wouldn't say that - I find it pretty amusing you've been registered at ./ for so long and are still so wrong.
p.s. I know I should reference - how about 'MS owns fuck all anymore' - will this [aviationhumour.co.uk] do?
Re:Apple (Score:4, Informative)
(http://slashdot.org/)
Well, there's Darwin, their (improved, IMnsHO) version of BSD.
Rendezvous is their (improved) version of ZeroConf.
Safari runs on the KHTML engine. Apple made some improvements and gave them back to the KHTML people, who thanked and praised Apple.
They've worked to improve gcc on PPC-based compilers.
They also provide the standard tools like apache, perl, python, etc etc etc, with OS X. I don't know if they have worked on these specifically, but it wouldn't surprise me in the least.
TROLL! (Score:2)
(Last Journal: Tuesday April 15 2003, @11:00PM)
Someone else already mentioned Rendezvous etc., but another point is that Apple is letting a whole slew of people tinker with UNIX without having to reboot into a foreign, newbie-unfriendly, OS. I've written a few shell scripts myself. Using a BSD foundation in itself helps the open-source community.