Apple Hardware VP Defends Benchmarks

Greg Joswiak, vice president of hardware product marketing at Apple, in a phone interview today, defended Apple's performance claims for its upcoming Power Mac G5, after they came under fire in the wake of yesterday's announcement. Read on for the details.

Joswiak went over the points in turn, but first said that they set out from the beginning to do a fair and even comparison, which is why they used an independent lab and provided full disclosure of the methods used in the tests, which would be "a silly way to do things" if Apple were intending to be deceptive.

He said Veritest used gcc for both platforms, instead of Intel's compiler, simply because the benchmarks measure two things at the same time: compiler, and hardware. To test the hardware alone, you must normalize the compiler out of the equation -- using the same version and similar settings -- and, if anything, Joswiak said, gcc has been available on the Intel platform for a lot longer and is more optimized for Intel than for PowerPC.

He conceded readily that the Dell numbers would be higher with the Intel compiler, but that the Apple numbers could be higher with a different compiler too.

Joswiak added that in the Intel modifications for the tests, they chose the option that provided higher scores for the Intel machine, not lower. The scores were higher under Linux than under Windows, and in the rate test, the scores were higher with hyperthreading disabled than enabled. He also said they would be happy to do the tests on Windows and with hyperthreading enabled, if people wanted it, as it would only make the G5 look better.

In the G5 modifications, they were made because shipping systems will have those options available. For example, memory read bypass was turned on, for even though it is not on by default in the tested prototypes, it will be on by default for the shipping systems. Software-based prefetching was turned off and a high-performance malloc was used because those options will be available on the shipping systems (Joswiak did not know whether this malloc, which is faster but less memory efficient, will be the default in the shipping systems).

As to not using SSE2, Joswiak said they enabled the correct flags for it, as documented on the gcc web site, so that SSE2 was enabled (the Veritest report lists the options used for each test, which appears to include the appropriate flags).

Who cares?

Really?

If you want OSX, you'll need to get the PPC.

If you want Windows, you'll get the x86.

If you want Linux, you can pick up 10 [slashdot.org] and build yourself a cluster for the price of one of these new machines.

Curious

This is the first time I think _I_ have seen slashdot with an article they wrote compltely on their own.

Did you recieve a phone call directly or something(Apple calling Slashdot)? If so did they act really aggressive wanting to make sure people don't become anti-G5 before it is even shipped?

Not too important you might say, but interests me.

Re:Curious

Eh, we do this sometimes, when it is appropriate. In this case, I have a PR contact at Apple who asked me last week if I wanted to talk to someone about WWDC, and we set up a call last weekend, for this afternoon. It just happened to coincide with the benchmark discussion, which Greg was eager to set straight (he had read the arguments and already compiled his responses :-). We also talked a bit about some other topics, but nothing of interest that you haven't read elsewhere.

Re:Curious

we do this sometimes, when it is appropriate. In this case, I have a PR contact at Apple who asked me last week if I wanted to talk to someone about WWDC

You know, I always thought that this would be a good idea for Slashdot. I mean, you guys must have some pretty interesting contacts by now, use some of them to do a "news" article or two on your own. I'd still keep the old Slashdot question/answer interview around because they are interesting and good for the people who don't have time to do a traditional interview.

Re:Curious

And why exactly is someone from /. *qualified* on this subject?

I hate to break it to you but many of the print reporters have very few qualifications to be covering the news but they are doing it anyways. Sure a Slashdot "professional" interview won't be extremely professional but I'm sure most people don't want it to be. I, for one, think I would enjoy a more "geek-on-the-street" kind of interview and I think that a Slashdot interview would provide that.

I may be wrong but it's at least worth a shot. It seems to have worked out pretty well in this story we are commenting on.

Re:Curious

(he had read the arguments and already compiled his responses :-)

Cheater! Dirty cheater, I say!!

What, did he use GCC to compile them?! Filth!!! DIE!

Re:Who cares?

The people who care are the zealots who don't understand, "Use the best tool for the job."

This means 3 things:

• Use a tool that is made for the task.
  
• Use a tool that you are comfortable with.
  
• The other tools don't suck.
  

People just have a hard time dealing with this whole "choice" thing.

Re:Who cares?

Personally I think the speed of the G5 and the validity of the benchmarks are both valid questions.

Does either of those questions alone determine whether you should get a G5 based system or not? No, but that doesn't mean the question isn't worth discussing.

I'm curious how fast the G5 is at certain kinds of tasks. Not because it helps me make a purchasing decision, but because I'm a geek and I'm interested in that kind of thing. This being slashdot, I'm sure I'm not the only one. Does superior floating point performance mean "better for photoshop"? Maybe not, but I'm more intersted in FP performance that PS performance.

I thought the original article was worth a read. I thought some of the comments are interesting. I thought this follow up was interesting. People like me are the ones who care. People who just want to know what kind of computer to buy, well yes, they are totally missing the point.

Re:Who cares?

The thing I wonder is what the purpose of this kind of company-sponsored benchmark is supposed to prove? Especially in a case like this where the results of the benchmark do not point to a clear winner (what with the questions surrounding the tests).

Apple may be a hardware company, but it isn't the hardware that is attracting customers. It's the software, stupid. If anything, Apple should be talking up the benefits of the OS and the "Apple System" (where everything works seamlessly) rather than the raw speed of the processor and leaving the benchmarking to review sites.

Apple's core competence is in making systems that are easy to set up and easy to administer and easy to use. It has never been in making "the fastest machine".

Re:Who cares?

>Apple's core competence is in making systems that are easy to set up and easy to administer and easy to use.

And that's not only thanks to the software, but also due to the great integration of software and hardware.

This integration ("it just works") is why people buy Apple. And therefore it's really hardware and software that attract customers (ey, and don't tell me I didn't buy my 17" PowerBook just for the software, I could have gotten an iBook if I only wanted to run OS X!)

Re:Who cares?

you know this and I know this but many trolls don't know this. I think Apple just got tired of hearing how PCs are faster and what not. Personally I was blown away by the keynote. Also, for anyone wondering I'm using the developer preview now and if the release of Panther is anything like the preview, holy crap. It is nice. There are a ton of tiny improvements here and there that really make it nice, even nicer than Jaguar. These are little things that weren't mentioned in the keynote.

Re:Who cares?

I just got a 450MHz G4 Cube (pre-owned, obviously).

I have used high-end workstation-class machines, both RISC and CISC, multi-GHz Intel machines, and Macs back to System 6. This Cube is without a doubt the best computer I have ever owned or used.

That having been said, I have seen Apple make some prety serious hardware and customer service mistakes. I would buy another Mac in a heartbeat, but I would wait for these systems to ship for at least six months before buying one of them. Wait until you can check Mac help forums. Find out what the problems are, if any. You don't want to spend $3000 on a computer, and have the paint chip off.

Re:Who cares?

There are 3 kinds of lies - lies, damned lies, and benchmarks.

I think Apple will have validity (in the performance arena) when AMD or Intel start publishing benchmarks against APPLE's systems.

Why won't Apple just use the AIX C compiler?

Let's face it: in their own way, Apple is being quite fair. Everybody in the free software community uses gcc, and publishing SPEC scores on x86 gcc is valid and useful.

However, IBM probably has C compilers for the POWER architecture that produce far more optimized code than gcc. Why hasn't Apple licensed and ported this technology?

Apple needn't resell such a C compiler, but critical system binaries (i.e. the kernel) could be recompiled for much better performance. Granted also that IBM is unlikely to support Objective C anytime soon, so such a compiler is only marginally useful.

However, Apple positively wastes these POWER chips without a vendor-optimized C compiler.

Re:Why won't Apple just use the AIX C compiler?

Apple's pulling off a miracle every day of the week by staying competitive and often moving ahead of the pack when it has such a small market share. When Apple has 10% of the market, they'll likely have the money to support such a project. But then again, why not just pour the same effort into gcc PPC optimizations? You get the same result (more hardware sales due to faster software) and you get kudos for contributing code.

Re:Why won't Apple just use the AIX C compiler?

If you had any brains you would have read the many refutations of the OSX on x86 idea and given up on this sad excuse for a business plan.

Most of the market wants to get their work done and doesn't care if it's x86, ppc, or some other chip that powers their computers. With Apple's unlimited client server licensing they're a cheaper solution for standard file and print servers than Windows. That's not as cheap as Linux but the hardware price difference very quickly gets swallowed up by Windows CAL costs. For small companies in the 10-100 employee range who don't want to have a full-time administrator Apple has a compelling enterprise product.

Re:Why won't Apple just use the AIX C compiler?

I'm writing a final proposal for a network redesign. I'm going to recommend that if they need to add a server in future it be a Mac OS X server box and swap out their current file and print box to handle the new application service they adopt. Why? Because their chief IT guy is also their chief accountant and their current IT consultants have been using fixed IP assignment instead of DHCP so that he needed to call them every time they had to install a machine. If you want to migrate a company like this off of Windows file and print, which makes more sense to you? Exactly.

You can't say hire a linux support person because their salary will be more than made up by the money you save on licensing and hardware over Mac. It just isn't. Even if you could get the chief accountant into a Linux class to take on an OS, is it really a wise use of his time and talents? No.

Macs are about as user friendly to administer as you can get and with 10.3 giving Active Directory integration with a dead simple GUI interface, it's a good choice for companies like this.

Photoshop WAS compiled with the AIX compiler!

I don't know if they still do, but for a while at the beginning of the PowerPC era Adobe was using the AIX compiler to generate its PowerPC binaries for Photoshop.

Admittedly, this was when the PowerPC was pretty new, and the choices were the IBM/AIX compiler which was robust and produced fast code but required an AIX box in addition to a Power Mac, or the nacent Metrowerks CodeWarrior compiler which run natively on the Power Mac, but generated poorly optimized code.

If I recall my history timeline correctly, after CodeWarrior came

• the Apple MPW "MrC" compiler (better code than CodeWarrior 1.0, but with a wacky command-line "IDE"), then
• gcc for PowerPC (cruddy code back then), then
• the Motorola PowerPC compiler (better code than Apple's compiler, with NO IDE - it plugged into the CodeWarrior or MPW IDE).
• Then Motorola inexplicably stopped selling their compiler.
• Later Motorola bought Metrowerks.
• Somewhere along the line, gcc learned to generate better PowerPC code.
• Eventually, Apple pretty much shelved their "MrC" compiler, and settled on using gcc for Mac OS X
• Monday, Apple released their "Xcode" environment -- still using gcc, I believe.

Apple's MPW tools are still available (free) here [apple.com] for Mac OS 7/8/9. The new Mac OS X tools including Xcode are available here [apple.com].

As a side note, it's really nice to see Apple giving away a full development suite for free, and continuing to put development time and effort into improving it.

-Mark

Actually, gcc is not so bad

It is quite difficult to produce better code than gcc, and my tests on powerpc (granted, those were a few years back using xlc on RS6000 with AIX 4) showed that xlc produced code of about the same quality -- sometimes worse, sometimes better.

The gcc "Haifa" scheduler was donated by IBM Haifa, by the way, so I think it's not surprising that gcc produces good code on powerpc.

On Intel it's quite the same, except that gcc does not vectorize code. From what I have seen, however, icc's vectorizer is not very useful either. I recently tested ogg-vorbis (which is a plain C floating point intensive benchmark) with icc 7 and gcc 3.3 and the gcc version was actually faster than the icc version (on my Athlon XP, target CPU pentium3) despite icc having vectorized several loops.

So all this "vendor-optimized C compiler" stuff is really besides the point. No C compiler will ever be able to match the quality of hand optimized assembler code, and the most important code (ffmpeg MPEG-2 decoder and MPEG-4 codec) has already been hand-optimized. You might be able to squeeze anoter 5 percent out of your code by using a vendor C compiler with insane optimizer settings, but what good is that if the end user is only going to use gcc anyway. I know I am, so I find the numbers for gcc actually more useful for comparison purposes than some vendor C compiler comparison.

Also, we don't want to encourage vendors to produce super vendor optimizing compilers, we want them to optimize gcc (so that everyone benefits, not just their users). So the more benchmarks are done using gcc, the better!

Re:Actually, gcc is not so bad

Interesting, when I took a course in Optimizing Compilers last year the concensus was that GCC is pretty awful when it comes to optimizations. Even general non-architecture dependent optimizations. The lecturers reason behind it was twofold.

First most research on compilers are being done at big corprorations. IBM being the single largest as I understood it. Naturally they put their optimizations in their own compilers first, the rest of the world have to implement them from their papers. (If they are lucky and the algorithms are not patented.)

Second if you were to put a good optimization in GCC it wouldn't take long before all other compilers had that optimization as well. GCC is OSS afterall.

We did comparisons between GCC and SunCC on UltraSPARC. SunCC minimal optimizations (O1) beat GCC with maximum optimizations (O4).

I'm just finished a course on vectorizing/parallelising compilers. There the situation is that even the best commercial compilers are pretty much equivalent to junk. Implementing the vector algorithms is a lot harder though. Even compared to complex SSA-form optimizations.

Re:Actually, gcc is not so bad

On the research level gcc is not as bleeding edge as other compilers. So if you run example code that shows the merits of a particular optimization, gcc may look not so good. But in practice, it's quite good.

My experiences with UltraSPARC are also a few years old, but gcc was faster and produced better code than Sun CC back then. You have to make sure to set -march=ultrasparc, of course. And I'm not sure about UltraSPARC but normally gcc -O4 does not do more than -O3, which basically is -O2 with function inlining. You can also get some boost with profile based optimization with gcc.

In summary, gcc produces very good code, but you might have to use some little known options for it. For example, gcc on Athlon XP and Pentium >= 3 may gain significant floating point performance with -mfpmath=sse,387 (I got >10% speed-up on lame, gcc's code was even faster than icc's with vectorizer). Another option worth knowing is -malign-double and the regparm attribute.

Another thing you have to keep in mind is: recent optimization advances normally are not big breakthroughs but small incremental advances. Many of them only help in a handful of special cases. gcc 3 has many more optimizations than gcc 2.95.3 and they were so proud of it that they said "much faster code on x86", and then there was whining and gnashing of teeth when most software was unaffected or even slower.

The only platform where I really would prefer the vendor cc is HP-UX on PA-RISC. The HP CC consistently produced 10-30% faster code than gcc (although that may have changed, I haven't used gcc > 2.7 on HP-UX).

Re:Who cares?

This is what makes me dislike benchmarks, they're treated too much like sports (everything has to be fair). We're talking about computers here. We should use the optimized version, of everything. Compile with the Intel compiler, compile with the ibm compiler, I don't care which one wins 'on even ground' I want to see what can go the fastest, period. If the G5 wins with gcc, but if you use inte's compiler and the P4 completely blows it away (or the other way around) then I want to know that, as opposed to thinking something else because 'the benchmark had to be fair'

Re:Who cares?

I'm sick and tired of people saying this crap about "use the right tool for the right job". It's like a geek mantra or something. I'm a geek but I don't subscribe to this theory that a computer or software or a programming language is a regular tool to be confused with a hammer or something.

Computers, software, and programming languages are tools, I'll give you that. But they are not single purpose tools like a hammer or screwdriver. A computer can do a multitude of tasks. It's malleable and can do just about anything. Since programming languages drive the computer they also fall into the same category. No matter what computer or what programming language, you have a all powerful system (well, as far as any electronic piece of equipment can be).

Picking the "right tool for the right job" when you're talking computers isn't like deciding whether to use a pair of pliers or axe to cut down a tree. It's like having a box of super tools and each one can do just about anything. Which one do you pick? Well, that answer isn't so easy when just about any of them can do the same tasks just maybe in a different fashion.

I also believe because of the flexibility of computers and specifically programming languages that it is in fact possible to create a more perfect language than anything currently existing. There is no perfect programming language, but there could be.

Sorry if that came out confusing. This only just now hit me. I'll have to organise my thoughts as I think about this some more.

Re:Who cares?

I think the key lies in your second-to-last paragraph: "There is no perfect programming language." Yes, exactly -- and moreover, some languages are more nearly perfect (not that I think any existing language is anywhere near perfect) than others, and how close a given language comes to perfection very often depends on the task you're trying to accomplish.

I used to write image-processing software. I wrote it in C, because writing it in a higher-level language would have been absurd. These days, I write database and Web interfaces, and I use the "P" languages (PHP, Perl, and Python) because writing it in C, while certainly possible, would be a huge pain in the ass. I like all of these languages, but it's indisputable that each of them is the right tool for some tasks but not for others.

The same is true of computers in general -- processors, architectures, OS's, etc. It would be great if you could set up one system that was clearly better than all others, or even equally good, for all tasks you might want to use it for. But you can't. The difference might not be quite as dramatic as that between pliers and an axe, but it's real.

I'm very happy with my iBook. It does many things I want to do very very well, and everything else I want to do at least passably. But I'm well aware of its limitations, and chafe at them fairly often. And this would be true of any system -- laptop, desktop, handheld, whatever -- I could possibly buy. I chose it because overall it offered the best fit for what I want to do. If my requirements change, well, then, so will my computer.

Re:Who cares?... Geeks do!

I'm becomming really dissapointed with what seems to be the majority of posts on /. (and especially in this thread). I was anticipating a lot of /.ers going on and on about how sweet the tech specs are on the PowerMac G5 hardware. It shouldn't matter what religion you are (M$, Sun, *NIX, Mac, IdogAppleToSoundSmart...), the hardware freeking rocks! Just like my attitude towards BeOS - I don't necessarily care whether the thing will gain 82% market share, its just cool shit.

Re:Who cares?... Geeks do!

The weakest point of Mac systems for many years has been slow bus speeds. Nobody's challenging the bus speeds and they're much, much faster. If you had a bus this fast on the G4 systems, they would dramatically improve their real-world performance.

RAM capacity is also not under challenge. So, for 23999 I can get a system that would permit up to 8GB of ram on the system.

Just those two unchallenged figures make this much more than just another boring speed bump hardware upgrade.

If they're providing the actual compiler flags they used and the flags used disprove one of the doubter's claims (no SSE2 use) then maybe Apple is *not* just making stuff up?

Re:Who cares?

For some people, e.g. physicists who do numerical "experiments", the benchmarks are crucial, or, at least, a large factor when considering which machine to buy.

Sure, one could buy 10 Linare boxes and Beowulf them together, but if you're a lone physicist with relatively little funding--Beowulf clusters take lots of time, money, and space to feed and maintain--you might care about being able to run floating-point intensive jobs quickly while being able to use MS Word or PowerPoint or some such.

In fact, I already know one astrophysicist who will be getting a G5 in the fall when her new research grant begins. She also happens to be one of the 3 physicists I managed to convince to switch to Mac and get a PowerBook.

Re:Who cares?

Most of the physicists I knew run *nix for their simulations, use LaTeX (not word) to do word processing and DTP, and use pdf files for presentations.

Not a flame, just a note on how things were when I knew physicists. Now I'm stuck with bio types :) Maybe things have changed and physicists are moving toward macs; I don't claim expertise in this area.

Honesty

At least everything that they did seemed to be amply documented.
I found that to be refresing especially in light of all the recent benchmark tests that have not been so forthright with all their methods and procedures.

Re:Honesty

What about the new 3.2's?

You mean the new chips from Intel that were announced the same day as the G5s?

Shit, some people you can never please.

Re:Honesty

That's the dangers of making a comment like "fastest PC on earth"......it's a claim that one has to be VERY careful about making.

Considering that you couldn't get either system yesterday (G5 or 3.2GHz Xeon (I hate it when slashdotters write 'zeon')), I see the whole thing as moot.

It's marketing, folks, not the bible. Greg backed up the test parameters with his data. I think the world would be a less stressful place if Windows went away, but I'm not stressing over a minute saved over a week of Photoshop work.

We spend more time thinking about what to do next in Photoshop than could possibly be saved by a faster processor/architecture/whatever.

That being said, I'll continue to buy Macs because the extra up front cost is well worth the knowledge that one company (and a rather well-run one these days) is responsible and capable enough to develop and market botht he hardware and the OS. They do a rather good job of it for a small premium.

Put another way: If I lose an hour a month because of a hardware vendor who refers me to an OS vendor to resolve a problem, I've lost an hour. I can't get that time back. If my Mac emits smoke and kernel panics at the same time, I know I can get resolution to both problems by calling Apple.

Re:Honesty

If my Mac emits smoke and kernel panics at the same time, I know I can get resolution to both problems by calling Apple.

I imagine that in that case, there is really only one problem to be solved. :)

Re:Honesty

Look at their numbers they are comparing the G5 to the P4 3.0 and Xeon 3.06. What about the new 3.2's?

How dare Apple! But then they only had the G5 2GHz. Maybe they should wait for the 3GHz, then the comparison will be fairer?

The fact that everyone is nitpicking these benchmarks shows how close the performance is. And with such a huge "megahertz" disparity between the Xeon and the G5 shows how much power the G5 has to offer.

Re:Honesty

You've obviously never worked in a data center where cooling and power are a premium. In a room full of hundreds of Intel crap, you start to consider things like power consumption and heat dissipation. This is a fact: The higher the clock the more power and heat.

We started holding our data center manager accountable for his own electric bill. After that, efficiency (lower clocks to do the same work) started to take on a whole new meaning.

This particular criterion also got us to get the MS-weenies to shut up and we started to implement more Linux systems.