Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Software Businesses Apple

Xgrid Clustering Software and Demo 290

Posted by michael from the please-mod-down-beowulf-jokes-kthx dept.
no_demons writes "Along with a selection of other goodies, Apple also unveiled their Xgrid clustering technology from their advanced computation group today. Xgrid can turn a number of networked Macs into a supercomputer, detects nodes automagically via Rendezvous, and can run in or out of a screensaver mode. You can download a technology demo (including a BLAST test app) here."
This discussion has been archived. No new comments can be posted.

Xgrid Clustering Software and Demo More

Xgrid Clustering Software and Demo

Comments Filter:

  • small scale? (Score:5, Interesting)

    by SHEENmaster ( 581283 ) <travis@uUUUtk.edu minus threevowels> on Tuesday January 06, 2004 @06:55PM (#7897041) Homepage Journal
    Does this work on the small scale as well, like OpenMOSIX? We have a few G4's at school that could benifit from clustering.

  • Will it work on legacy machines? (Score:5, Funny)

    by ActionPlant ( 721843 ) on Tuesday January 06, 2004 @06:57PM (#7897080) Homepage
    It'll be interesting to see if I can make this work with the stacks of old LC520s in my garage. I've been wanting to cluster them for a while. If Xgrid will work on those, Mac just saved me a ton of work. Not that I wasn't going to have fun with it....

    Damon,

    • Re:Will it work on legacy machines? (Score:5, Informative)

      by xactoguy ( 555443 ) on Tuesday January 06, 2004 @07:04PM (#7897156)
      Nope, it won't work on those. XGrid is based off of Apple's Rendezvous, which is OS X ( well, at least until someone ports it, seeing as it is open source [apple.com]. So, unless you plan to port it yourself, and to port XGrid as well ( if it is ever open sourced ), then you're out of luck. If this was a serious post that is. If it was a troll, then I bit, but you don't care ;)
      • Well, for what it's worth, people are compiling some very cool things these days; why not drop modules in the kernel until it works? I think it's worth a shot at least. Definitely a weekend project (or many, many weekends).

        If it works, just think of all the happy legacy owners!

        Damon,

      • Re:Will it work on legacy machines? (Score:5, Informative)

        by Ffakr ( 468921 ) on Wednesday January 07, 2004 @01:14AM (#7900225) Homepage
        "XGrid is based off of Apple's Rendezvous, which is OS X "

        No, Xgrid is based off of the Zilla project that ran on NeXT.
        Zilla was acquired by Apple when NeXT was purchased.
        Zilla was rechristened Xilla during development in honor of OS X.
        It's now called XGrid.. and yes it is cool.

        Now, XGrid includes support for Rendzvous.
        Rendzvous is Apple's release of ZeroConf, an OPEN SOURCE ad hoc IP based protocol.

        Someone else asked about running on other BSD's.
        XGrid runs in user space. It isn't a kext (kernel extension). It probably could run on other BSDs without too much work, but it is a carbon app so you'd have to totally port the interface to some other GUI API.. and you'd have to port it from Obj C to something more common I'd guess.
        Apple hasn't provided source (yet) though so I don't see anyone porting it soon. Maybe reverse engineering it...

        other stuff... it apparently makes use of XML too but I haven't gone through all the docs yet.

    • Re:Will it work on legacy machines? (Score:5, Funny)

      by Squid ( 3420 ) on Tuesday January 06, 2004 @07:10PM (#7897225) Homepage
      Of course Apple probably won't make this available for anything other than OS X. But even if they did... how many LC520-vintage Macs does it take, clustered, to equal the raw computing power found in, say, an iPod? (A modern Apple MOUSE probably has a more powerful CPU than some of the older Macs.)

      If there is a law of diminishing returns trying to cluster old hardware to keep it useful, I think an LC520 is well past it.

      Besides, if you want to do it, see if Linux will run on those 520s and... yes, you guessed it... build a Beowulf cluster out of 'em.
    • You should check out Pooch [daugerresearch.com] from Dean Dauger. He has adopted OS X for obvious reasons but older versions of Pooch ran on MacOS 9.

      Pooch was the original easy to use and implement solution for clustering Macs used originally at UCLA for physics modeling.

    • *strike!*
      Ok, original post shoulda been modded funny or troll, but there are cluster solutions for old Macs, so here goes:

      NetBSD/68k [netbsd.org], supports 68k cpus and various free cluster architectures.

      Appleseed [ucla.edu], supports OS 8.6 - 9.x on PPC.

      Quite a few older PCI Powermacs can be coaxed to run OS X 10.1.x using XPostFacto [macsales.com] and some patience. Won't support XGrid for now, but the other free suspects will work.

    • Launch Den Mother and Launch Den Puppy [daugerresearch.com] are a distributed computing application pair for older Macs. You can download a 68k version despite their claims of requiring a Power Mac.

      Its not as automagical as XGrid and you will have to write your own multiprocessing code, but it is a way to do distributed processing on old hardware. Clustering a bunch of LC520s may not be the best use of time and electricity, but then who said that a hack must be cost-effective.

  • Sounds good, but... (Score:4, Interesting)

    by moehoward ( 668736 ) on Tuesday January 06, 2004 @06:59PM (#7897101)
    What's the cost? I can't find any info on pricing, etc. Sounds like a great product if it works as advertised.

    But for me, the model I want is a broker model. I want to sell my processor time to a broker who will resell it on a day to day basis to whoever is the highest bidder. E-bay of grid computing, ya know. I don't want to pick projects, download clients, etc. I just want to pariticipate (i.e. make money) from whoever is willing to pay the most at any given moment.

    And when I feel like it, I'll volunteer x% to non-commericial stuff like SETI@home.

    • Re:Sounds good, but... (Score:5, Funny)

      by rampant mac ( 561036 ) on Tuesday January 06, 2004 @07:06PM (#7897181)
      "But for me, the model I want is a broker model. I want to sell my processor time to a broker who will resell it on a day to day basis to whoever is the highest bidder. E-bay of grid computing, ya know. I don't want to pick projects, download clients, etc. I just want to pariticipate (i.e. make money) from whoever is willing to pay the most at any given moment.

      And when I feel like it, I'll volunteer x% to non-commericial stuff like SETI@home."

      Ladies and Gentlemen, please welcome the 21st century pimp.

      "What do we get for 10 teraflops?"

      "Anyting you want!"

      "Anything?"

      "Anyting!"

    • Re:Sounds good, but... (Score:5, Informative)

      by Wesley Felter ( 138342 ) <wesley@felter.org> on Tuesday January 06, 2004 @07:34PM (#7897457) Homepage
      I want to sell my processor time to a broker who will resell it on a day to day basis to whoever is the highest bidder.

      Several companies tried this back in 2001 and discovered that the processor time on your computer is worth less than the overhead cost of using it. Sorry.
      • Several companies tried this back in 2001 and discovered that the processor time on your computer is worth less than the overhead cost of using it. Sorry.

        Well, sure, it's worth less to companies who are looking to save money in the process or your average 'netizen, who thinks it should be free by divine right). Now if you implemented it with something akin to Mojo [sourceforge.net] so that contributors to the grid could eventually use the grid for their own purposes, that's a different story.
        For example, Apple or IBM (who

        • Re:Sounds good, but... (Score:3, Interesting)

          by laird ( 2705 )
          In theory, distributing computing is an amazing resource -- most computers are 99% unused (2.4 GHz CPU waiting for the user to hit the next keystroke...). But it's very, very hard to actually take advantage of it.

          The problem isn't so much that the compute time isn't valuable as that there are relatively few problems that are amenable to this style of distibuted computing.

          There are two axis that I believe need to work out: the value of solving the problem (and to whom), and how parallelizable it is.

          In ter
      • Uh, someone forgot to tell United Devices [uniteddevices.com].

        The corporation is intimately tied to distributed.net [distributed.net] and has been selling distributed / Grid computing to corporations for YEARS.

    • Re:Sounds good, but... (Score:4, Informative)

      by andcarne ( 657052 ) <andcarne@mac.com> on Tuesday January 06, 2004 @07:37PM (#7897485)
      Its free software.
      "Anyone can download the technology preview today, which includes a kit that lets programmers add functionality to Xgrid for more advanced job control."
      The download can be found at: http://developer.apple.com/hardware/ve/acgresearch .html

  • Why limit this.. (Score:4, Interesting)

    by jaxdahl ( 227487 ) on Tuesday January 06, 2004 @07:04PM (#7897166)
    Why limit this to Macs only? Wouldn't this work even better if it were cross-platform -- like many other distributed computing solutions such as SETI, distributed.net, and the UD Cancer projects.

    • Re:Why limit this.. (Score:5, Informative)

      by sydney094 ( 153190 ) on Tuesday January 06, 2004 @07:21PM (#7897343)
      It isn't inherently limited to Macs... however, the only computers that they have written the client for is Mac 10.2.8 or better.

      (From the FAQ)

      Q: Can I use Xgrid with other UNIX-based computers?

      A: The short answer is no.

      The long answer is that Xgrid uses an XML property list protocol built on top of BEEP for all of its inter-computer communication and coordination, and because these protocols are open, it is possible a client, agent, or controller could be written to run on other UNIX-based computers and interoperate with Xgrid. However, no such programs have been written.
    • yes, apple could do that.. but then why would you buy a mac?

      • Re:Why limit this.. (Score:3, Insightful)

        by shaitand ( 626655 )
        You wouldn't, you'd buy Xgrid. It's simple math and microsoft certainly figured it out a long time ago. The markup on a piece of software is anywhere from let's see, $200 divided by $0.15, what does that come out to? Or you could sell the equivelent in hardware, $200 divided by $150. And for some odd reason you can easily manage to sell the software orders of magnitude more times than you'd manage to sell the hardware.

        Gee, I wonder.

  • Great, this sucks.. (Score:5, Funny)

    by QuantumRiff ( 120817 ) on Tuesday January 06, 2004 @07:05PM (#7897173)
    Now every single story posted on slashdot for the next 6 months is going to have a comment of "Imagine an XGrid of these!" The old incarnation was slowly dying too...

  • Anything similar for Linux? (Score:3, Interesting)

    by costas ( 38724 ) on Tuesday January 06, 2004 @07:11PM (#7897245) Homepage
    It's been a while since I played with Beowulf clusters, but I've been thinking of using a Beo-class cluster in an enterprise app. The problem there is that you want the IT people to have to do as little "deep magic" work as possible when it comes to grids. So, auto-detection of new nodes, transfer of jobs out of failing/crashed nodes, etc. are important features. Is there anything close to that for Linux? a few different solutions that integrate well together isn't out of the question either...
    • Mosix would probably be the best bet although it is still missing migratable sockets and distributed shared memory. The distributable shared memory problem has been at least partially addressed with migshm.

      Give it a try.

  • Been there, done that, Apple bought me lunch. (Score:5, Interesting)

    by antarctican ( 301636 ) on Tuesday January 06, 2004 @07:12PM (#7897249) Homepage
    Oh good, you mean I can actually talk about XGrid now after signing an NDA over 6 months ago? :)

    We had the second installation of XGrid, the only other group using it at the time was NASA. I haven't had much time to play with it personally but we had our coop do some genetic sequence analysis using it and he was quite impressed. Plus the speedometer-like gauge measuring performance just looks soooo cool. ;)

    • Re:Been there, done that, Apple bought me lunch. (Score:5, Interesting)

      by buckhead_buddy ( 186384 ) on Tuesday January 06, 2004 @08:18PM (#7897890)
      Back when Spindler ruled as Steward of Apple (well before the return of Jobs), I signed an NDA for an evaluation period with a piece of Apple hardware that was supposedly about 10-12 months from release.

      The specs of the machine have been outclassed by modern standards, and the neat software features not made irrelevant by Mac OS X have surfaced in Mac OS X Server. But there were two very cool things about the case though that I thought were of the "Geez, that seems so obviously handy!" variety.

      The gotcha was that it has never been released. I keep expecting new Mac hardware to one day come out with these two features but until that time I guess I'm still under NDA. Arrggghhhh!!!!!

      Be glad your NDA expired with the release of the product. Otherwise you'd be burning with the image of that cool speedometer in your mind and unable to tell anyone about it. :-)
  • I wonder how long it will take someone at MIT or some other tech university to figure out if I put all the Macs located on or around a college campus I could or University could make it's self a mini-Super Computer for very few dollars.

    • Re:mini me super cluster... (Score:4, Interesting)

      by antarctican ( 301636 ) on Tuesday January 06, 2004 @07:54PM (#7897643) Homepage
      I wonder how long it will take someone at MIT or some other tech university to figure out if I put all the Macs located on or around a college campus I could or University could make it's self a mini-Super Computer for very few dollars.

      We did this 6 months ago when we were first beta testing XGrid. I never had the opportunity to personally play with it, but a grid of a hundred EMacs was a pretty good start to a mini-supercomputer I was told. :)

  • db solutions? (Score:4, Interesting)

    by tobes ( 302057 ) * <(tobypadilla) (at) (gmail.com)> on Tuesday January 06, 2004 @07:18PM (#7897308) Homepage
    I don't know much about clustering, but maybe someone can fill me in. Would it be possible to run something like Postgres on a cluster of XServes? Would the clustering be transparent at the application level, so that any program could take advantage of the clustered resources or would specific distributed algorithm apps need to be written?

    I'm hoping that I could just get a stack of XServes and run an OSS db on it for free (as in no extra effort required), but I'm guessing that's not the way it works.
    • Depends on if it supports distributable shared memory. If not, like OpenMosix, it will have issues with a database. Now something that has been designed to be much less IO intensive (like audio/video processing, etc.) will really benefit.
    • Nope, postgres doesnt work, since it relies on shared memory to work, and doesnt support multiple postmaster processes per database.

      On the whole, the open-source databases' support for replication/clustering is poor to non-existent.
    • The usual ways to scale out databases is shared-everything data on one storage volume with crazy lock management or shared-nothing databases on separate volumes and a query manager doing distributed queries across the cluster. I believe oracle can shared-everything, and probably given their #1 spot on the tpc-c at the moment, the shared-nothing as well; DB2 and SQL Server can do the shared-nothing.

    • Nah, not really. (Score:5, Informative)

      by PCM2 ( 4486 ) on Tuesday January 06, 2004 @08:36PM (#7898070) Homepage
      Clustering databases has different issues/concerns than clustering computational problems. I wrote an article about database clustering a while back, available here, [newarchitect.com] if you're interested.

  • My TiVo has Rendezvous... (Score:4, Funny)

    by wembley ( 81899 ) on Tuesday January 06, 2004 @07:18PM (#7897309) Homepage
    I wonder if I could offload some of my compiles to TiVo...

    :-)
  • Big Mac Super Computer Movie [apple.com]

    If you want to see a behind the scenes vid of the making of the G5 supercluster then go to the link above.

    I wonder what, if any, input VT had on XGrid...

  • Apple vs Microsoft (Score:4, Interesting)

    by Archangel Michael ( 180766 ) on Tuesday January 06, 2004 @07:32PM (#7897446) Journal
    Here is another wonderful example of Apple beating Microsoft on more than one level.

    1. Over all Coolness AND Geek Factors.
    2. Usefulness of coolness factor.
    3. Ease of use of Geek factor.

    Microsoft hasn't done anything remotely like this in thier existance, rarely do they push the tech envelope beyond the what has become ordinary and benign. And when they do, they end up with over analized and engineered products nobody want (Bob?).

    Apple of the otherhand is OFTEN making stuff WAY before the curve. Newton/PDA, USB/Firewire, CDROMs, Floppylessness (is that a word), standard networking .......... and now Beowolf clustering out of the BOX!!!

    If Apple is smart with this, they could play against the new model of "ON DEMAND" services from companies like IBM.

    Imagine a corporation that could automatically timeshare, timeshift computing resources based on such technology using the workstations they currently own/purchase. Peak periods of processing could be syphoned from little or unused desktops "On Demand", back filling any need left over from the DP center.

    If this is NOT revolutionary it will be at least evolutionary.

    • Re:Apple vs Microsoft (Score:3, Informative)

      by jcbphi ( 235355 )
      Okay, I'll bite. Microsoft, for all its failings to innovate in its commercial projects, has been churning out interesting research for quite some time in its various labs:

      http://research.microsoft.com/research/topics/ [microsoft.com]

      Its a shame that so little of this work is making its way into products you can use, but them's the breaks.

      • Re:Apple vs Microsoft (Score:4, Interesting)

        by KrispyKringle ( 672903 ) on Tuesday January 06, 2004 @09:37PM (#7898586)
        While you are probably right, what is truly interesting about this is that it's Apple who are doing it.

        Microsoft, for all their research, always seem to have a ``me, too'' attitude. Their entire business approach (which is certainly highly successful in many ways) is essentially to jump into every possible market (gaming consoles, consumer electronics, embedded devices, low-end servers, enterprise computing, web application infrastructure, clustering, etc) with the hope of, if not succeeding, at least driving out potential competitors from succeeding instead.

        Apple, on the other hand, has seemingly ignored anything but their established market of home users and graphic artists, video editors, and the like for a long, long time. I have never even seen an Xserve in use, neat as they seem to be. They've made no real headway in commercial environments (other than ad agencies and the like), either as workstations or as PCs.

        And yet, here we have a product (for which a market may or may not really exist) which falls far outside Apple's traditional domain, but which seems to be clearly innovative. Where Microsoft only seems to promote technologies that it sees other companies already engaged in, Apple pioneers something somewhat more risky. Microsoft's model is probably more successful. But that may not be the point.

      • but then the question goes to: what would you rather have, innovation that leads to products, or innovation that leads to nothing? the latter can be more useful, when it is so innovative that it is simply too far ahead of its time, but i don't see microsoft having this. Apple's research has regularly turned out very new, creative and USEFUL applications and hardware.

  • Hardly a Supercomputer: Cluster computing 101 (Score:5, Informative)

    by deadline ( 14171 ) on Tuesday January 06, 2004 @07:41PM (#7897515) Homepage
    There are only certain problems that work well on LAN clusters. Those that have a lot of independent jobs (like BLAST) and those that require a small amount of communication like rendering.

    Read ClusterWorld [clusterworld.com] and you can figure this out yourself.

  • 90nm G5s (Score:5, Informative)

    by sergeantmudd ( 647674 ) on Tuesday January 06, 2004 @07:44PM (#7897542)
    What's more important is what it's clustering, 90 nanometer G5s. Apple and IBM are the first company to bring 90 nm processors to the market. Xserve White Paper [akamai.net]

  • batch queueing (Score:4, Interesting)

    by tolldog ( 1571 ) on Tuesday January 06, 2004 @08:01PM (#7897707) Homepage Journal
    Isn't this just a batch queueing system, much like Sun's Grid computing, Platform's LSF and the like.

    If it is, it is different in nature than the often joked about beowulf clusters and the mosix systems.

    I think its great that Apple has one too, the more ideas thrown into the pot, the better all of them become.

    -Tim
  • I thought they had an app that would let you start a compile (or something) on one machine, and if it got busy that machine would go to other Sun boxes on the network and say "Oi! Can I borrow some CPU time?" If the other boxes weren't too busy, they'd share the job.

    Maybe this is what Xgrid is doing now, I didn't read the article too well. ;)
  • what a phenomenal idea. to take advantage of the idle processing power of machines that sit unused at least 50% of their time. virginia tech is just the beginning.

    any college with a several ~25-machine labs can use this app to do supercomputer stuff, AND get the return on investment from normal users being able to utilize these machines during the day.
    • For a lot of problems, having a fast network interconnect between the computation nodes is essential to getting good performance. The interconnects in the lab may not be enough.

      That said, various software packages already exist to do opportunistic supercomputing using the spare resources in machines suitable for use in a lab -- Condor [wisc.edu] is one such package. We have it deployed in our computing labs, and several research groups in our department make use of it.

      Cheers,

      David
    • In a limited environment, it makes sense, but not campus-wide. For example, I know of a neuroscience lab which has a Linux computer lab where MRI data (which takes up terabytes) is stored on networked RAID arrays, and the actual processing is done by the Linux workstations.

      But to do it campus-wide makes many admins very very nervous; the cost of maintainance and upkeep may in fact rise, and even if the secretaries aren't using their P4's with hyperthreading most of the time, the admins don't want the has

  • ifdef Win32 at Apple? (Score:4, Interesting)

    by G3ckoG33k ( 647276 ) on Tuesday January 06, 2004 @08:21PM (#7897920)
    A code snippet from here [apple.com](!) reads:

    #ifdef _WIN32
    #pragma warning( disable : 4127 4706 ) /* disable conditional is constant warning */
    #endif

    Why would developers at Apple ifdef for Win32?!
  • ... is to make this available to *any* program that does intensive rendering. Using Toast's iMovie extension to export a 44 minute movie as an MPG for VCD took 2 hours 12 minutes at the best quality on a dual-G4/1250. A few other Macs laying around (like the other 22 in my department) could have cut that down quite a bit. Sure, it'd take time to send out 9 GB of DV, but less time than it took to encode it.
  • Anyone here seen POOCH? Been around for, what, the last year [daugerresearch.com] and offering the same thing? Now XGrid is big news?

  • Imagine (Score:2, Redundant)

    by rlp ( 11898 )
    Imagine a beowolf cluster of ...

    Oh! Never mind.

  • What it's not (Score:3, Interesting)

    by harywilke ( 621602 ) on Tuesday January 06, 2004 @08:48PM (#7898187)
    From Apples Website: http://www.apple.com/acg/ Xgrid Xgrid clustering software is intended primarily for scientific researchers who have a set of networked computers that are not yet being used to full collective CPU potential. The software provides a remote execution environment and file staging abilities that coordinate the running of tasks on distributed computing resources and ensure that each computer has access to all of the files necessary to execute the tasks, thus freeing both user and developer of such resources from this tedious work. Xgrid does not solve all clustering problems. It does not replace clustering software such as MPI or clustering hardware such as InfiniBand. Xgrid does not accelerate or "grid-enable" existing applications on your computer. For an application to take advantage of the Xgrid technology, you must update it to use the Xgrid APIs. However, if you currently run long computations in your Terminal windows using, say, an already-compiled executable, you should be able to use Xgrid out of the box to run batch jobs for this executable on your clusters. In this way Xgrid offers high-throughput computing with minimal development work on your part.

  • First impressions (Score:4, Interesting)

    by h3 ( 27424 ) on Wednesday January 07, 2004 @06:32PM (#7907375) Homepage Journal
    Coincidentally, I was working on setting up Sun Grid Engine on a couple of G5s here at work yesterday when I heard the news about XGrid. I dropped my work on SGE immediately.

    It was a little confusing at first to set up, but I eventually got 3 machines total configured as part of my grid - 2x2GHz G5s + 1x867MHz G4. Perhaps if I read the instructions, I would've better understood some of the terminology (agent v. controller v. client, etc.).

    The tachometer is sort of flaky. Sometimes it's stuck at zero on one machine while it it is actively moving around on others. Other times it's stuck at some non-zero position. Opening up 2 tachs on the same machine (XGridBlast has its own tach) will show different speeds. Though I should have in theory 8.867 GHz total speed, I could never get it to go over the default 8 "red line"- I was curious if it would rescale once it exceeded 8.

    The XGrid client (where you submit jobs) has some default demo type things (I've mostly been testing the Mandelbrot one as it runs in a continuous loop), but it also has a way to build "custom plug-ins" which allow you to submit arbitrary jobs. In other words, executables don't have to be modified per se. Of course for any kind of parallel execution, they do, but if you need to to run 1000 iterations of the same command with slightly different arguments, then it should be able to distribute that "run" pretty well. The GUI for building up such a run is pretty easy to use.

    One potentially big issue I've noticed is that once you submit a job, you have to keep the XGrid app open until it is done. For a run that may take days or weeks, I think this could be a problem. I'd like for you to be able to submit the job, quit, and log out (or shutdown the client even) then come back later to check your results.

    Also, there doesn't seem to be a queue manager where you can see a list of jobs and their states.

    I think that for any file-dependent commands, you need local copies of the files on each Agent node. At least that's how it appeared from perusing the documents for XGrid Blast - each node needs a copy of the database.

    FWIW, we're using SGE successfully on a bunch of RedHat based servers, but though OSX maybe *nix, the installation and config was turning to be a pain.

    Anyway, those are my initial impressions. I'm sure some of these things will change in the "release" version. In the meantime, I'll have to get working on getting some real work stuff to try out (I work in a university bio department).

    -h3
    • Status: Job failed (task: failed with status 255). I suspect that was because I was trying to run it off an xgrid consisting of one G3 iBook, and it needs the velocity engine (though nothing of the sort was documented). However if that is the case it shouldn't have failed, it should have waited to run off of other computers and just ignored the G3.

      I'm potentially seeing a situation where some well meaning non-compatible agent joins an open XGrid and flubs someone's whole job. There needs to be a way to

Slashdot Top Deals

Love may laugh at locksmiths, but he has a profound respect for money bags. -- Sidney Paternoster, "The Folly of the Wise"

Close