How an Automated Mistake by Apple Killed All of a Mac Developer's Apps (9to5mac.com) 41
Long-time Slashdot reader philml writes:
Popular Mac developer Charlie Monroe woke up to find that none of his users could run his software. Instead, Mac OS was giving a message saying that it "will damage your computer".
Monroe described the ensuing hassle in a blog post titled "A day without business." In a later update he added that Apple "has called and apologized for the complications. The issue was caused by my account being erroneously flagged by automated processes."
But 9 to 5 Mac describes how Apple's mistake affected Monroe's apps: Users were unable to open them, and a message flagged them as malware, advising users to delete the apps to avoid damaging their Macs.
Developer Charlie Monroe, creator of the Downie video downloader, among other apps, said that Apple didn't even send him a message saying it had happened, and for several hours he didn't know whether he still had a business or not⦠He said that it took Apple 24 hours to partly fix the problem, removing the flags, though that still left him having to recompile, re-sign, and redistribute everything... Most app users will never know the story behind this, only that they bought an app, Apple told them it was malware, and they deleted it as instructed.
It also seems unlikely to help Apple's antitrust battles, where many are arguing that the company holds too much power over users and developers alike.
Monroe described the ensuing hassle in a blog post titled "A day without business." In a later update he added that Apple "has called and apologized for the complications. The issue was caused by my account being erroneously flagged by automated processes."
But 9 to 5 Mac describes how Apple's mistake affected Monroe's apps: Users were unable to open them, and a message flagged them as malware, advising users to delete the apps to avoid damaging their Macs.
Developer Charlie Monroe, creator of the Downie video downloader, among other apps, said that Apple didn't even send him a message saying it had happened, and for several hours he didn't know whether he still had a business or not⦠He said that it took Apple 24 hours to partly fix the problem, removing the flags, though that still left him having to recompile, re-sign, and redistribute everything... Most app users will never know the story behind this, only that they bought an app, Apple told them it was malware, and they deleted it as instructed.
It also seems unlikely to help Apple's antitrust battles, where many are arguing that the company holds too much power over users and developers alike.
Sounds Like A Lawsuit (Score:5, Insightful)
This type of shit has to stop.
Re: (Score:2, Insightful)
I would say they did almost unrepairable damage to his venture and cost him quite a bit of money. They should have to reimburse him heavily for this. Automated systems should not be a replacement for human verification. If anything the lawsuit would help make the case against Apple. I mean they were so quick to take Microsoft to court over bundling IE4 with Windows....yet Apple is literally able to make or break a developer and nothing? This type of shit has to stop.
While I agree with you, what are his actual damages? He could possibly prove he lost some sales on that day based on his daily sales volume, and some loss of reputation. I would think he could also claim damages for the time spent notifying users of the error on Apple's part; but that's likely an automated email to all registered users so not a lot of time. I am sympathetic to the idea that Apple owes damages, but the costs of pursuing a lawsuit may outweigh what he can collect.
Re: Sounds Like A Lawsuit (Score:5, Insightful)
Re: (Score:3)
I don't think it's fair to call someone a sheep when they get a warning that a program is malware, delivered by the OS, and then delete the program.
If they got that message and didn't delete the program, and it actually WAS malware, you'd call them computer-illiterate morons, wouldn't you?
Re: (Score:2)
If they got that message and didn't delete the program, and it actually WAS malware, you'd call them computer-illiterate morons, wouldn't you?
But we're talking about Apple, so it's okay to Think Different!
Re: Sounds Like A Lawsuit (Score:5, Insightful)
Re: (Score:2)
Wait, if you paid for an app that gets flagged as malware you don't get an automatic refund?
Re: (Score:2)
Who wants to bet that as part of obtaining a license for all the development tools on the Mac, access to Apple's tools for signing and distributing Mac software, there are at least four or five instances of the fine print indemnifying Apple from all damages, no matter how much they fuck up?
Re: (Score:1)
Re: Sounds Like A Lawsuit (Score:1)
Sorry you will find around here apple is never wrong.
Why do you make such ridiculous statements?
Here, I'll start: Apple was wrong. They should have backed-up their automated "malware watch" process with a final "human" approval step before doing all that Notification and Blacklisting shit.
But I'll bet it never happens again.
Re: (Score:1)
I don't know what site you think you're on. But slashdot is by no means a pro-Apple site. In fact, it's quite the opposite. Does: "No wireless. Less space than a nomad. Lame." jog your memory?
Re: Sounds Like A Lawsuit (Score:2, Troll)
This type of shit has to stop.
One false positive in how many App Approvals, and you raise your fist and proclaim âoeThis sit has got to stop?!?!?
Please.
I'd say that is actually a pretty robust algorithm, that just needs a manual "final signoff" added to make it perfect, or at least as perfect as possible.
Apple no doubt added the automated screening to catch stuff the human App Approval process may have missed. Unfortunately, they forgot to properly "close the loop" by letting a human have the final say whether to Blacklist an App.
Re: (Score:2)
This type of shit has to stop.
One false positive in how many App Approvals, and you raise your fist and proclaim âoeThis sit has got to stop?!?!?
You don't actually know that. For all we know, this could be part of a new automated system that they just rolled out yesterday, in which case it could be one false positive in a few dozen app approvals.
On the one hand, I agree with you that this isn't the end of the world, though it's a major mistake that I suspect will cost Apple dearly if the developer decides to lawyer up. On the other hand, can you imagine if this had happened to some major competitor like Amazon or Spotify? That could easily have b
Such an impact should have a human review (Score:5, Insightful)
I doubt this type of kill switch happens that often that it can't be quickly verified; if it is a common occurrence, Apple has other problems than automated kill switches.
Re: (Score:2)
What would the human do? Run the script that checks a program for naughtiness?
Re: (Score:2)
They could apply judgement and common sense, two things which an automated algorithm can not do.
If the developer has been distributing their applications without complaint for several years, then maybe it's a false positive.
If the checks are too sensitive, then a manual check could identify cases where something was flagged up incorrectly, and the algorithm can be adjusted to improve its behaviour for the future.
Blindly destroying a developer's livelihood all for the sake of a manual sanity check isn't good
Re: (Score:2)
If the developer has been distributing their applications without complaint for several years, then maybe it's a false positive.
A robot could do that, too.
Re: (Score:2)
If it was correctly programmed to cater for all special cases, yes it could. This isn't the world we live in though. A human check is required which the decision has such significant financial and legal, not to mention ethical, considerations. Would you want your company to be destroyed because some other company ran an automated check and it flagged you up erroneously, with zero liability for their "mistake"?
Re: (Score:2)
What would the human do? Run the script that checks a program for naughtiness?
No, check the evil bit.
Re: Such an impact should have a human review (Score:1)
I doubt this type of kill switch happens that often that it can't be quickly verified; if it is a common occurrence, Apple has other problems than automated kill switches.
I agree wholeheartedly with your entire post. I think maybe the Automated system could block purchasing of the App and put up an "Under Reviewâ page on the App's listing-page; giving Apple time to conduct a thorough manual review, contact the Developer, etc. before a human makes the decision to Notify Users, Blacklist the App, and so forth.
As you said, this doesn't happen often enough to negate the feasibility of human "final say".
A step that I bet will be added to this process, pronto!
Horsecrap (Score:2)
Most app users will never know the story behind this, only that they bought an app, Apple told them it was malware, and they deleted it as instructed
No. This isn't App Store, to get his software you download it from his website, where he is more than capable of putting a big fucking sign saying "Sorry, Apple fucked up my Developer Certificate so you need to download this update."
Re: Horsecrap (Score:2)
More to come - Stay tuned! (Score:3)
It can't have been that bad (Score:1)
We're talking Apple. That's like what, one thousand people or so?
Mac OS security (Score:5, Interesting)
I have a software bundle that I use to teach Software Defined Radio to Electrical Engineering students. The bundle includes an OpenJDK runtime, Eclipse along with software modules the students can write or modify.
I modify eclipse.ini to point within the directory tree of the bundle to make the whole thing self-contained. When I create a package and then try installing that bundle from the package, I get some kind of "This software is corrupted, it could be malware" message. I localized the problem to not using the Eclipse image as-is with eclipse.ini unchanged.
Our ECE department computer support dude is Mac-centric, but between the two of us we could not get this to work with the modified eclipse.ini text file. I suggested it may have something to do with a checksum in Eclipse and some manner of software signing protocol.
I am not making any money off this apart from being paid to teach at the U, but it seems the Mac Universe is adverse to this kind of thing. Maybe the problem is that Eclipse for Mac OS is signed and I am wanting to redistribute a modified Eclipse that triggers a malware detection?
The work around is to have students with Macs install Java and Eclipse on their own and have them follow several pages of written instructions on how to configure the Eclipse project for the course. Actually, the introductory CS course in programming is a prereq for my course, so the already have Java and Eclipse on their Mac laptop. But still, I would like to put a load-and-go bundle on my Web page for use by members of the local amateur radio group.
How hard is it to get some manner of software signing privileges on MacOS for an academic setting?
Re: Mac OS security (Score:2)
How hard is it to get some manner of software signing privileges on MacOS for an academic setting?
Actually, pretty fucking hard, without punching a great big hole in the entire concept of signed software (which is, by and large, a very good thing for Users).
Why not just teach your students the way you install unsigned software on a Mac? It is much simpler than dragging them through a multi-step manual install procedure.
Re: (Score:2)
He should be able to get "signing privileges" for his own software for $99 - the cost of a one-year developer's license. But what he seemingly wants to do is be able to modify someone else's signed software... which, as you point out, would defeat the purpose of signed software in the first place.
I would think he could accomplish his goal, though, by distributing his modified eclipse setup inside some sort of self-contained virtual machine or docker image.
Re: (Score:3)
No, he would want to modify someone else's software and re-sign it with his own key.
Eclipse is Free software - the EPL allows modification, redistribution, etc. but doesn't always require source like the GPL does (so it isn't as strong copyleft I guess). Can you give me a good reason someone shouldn't be able to modify Eclipse (or any other F/OSS) and re-distribute with a new signature?
Re: (Score:2)
That is exactly what I want to do, change eclipse.ini to point to the directory tree inside the bundle and redistribute that with an OpenJDK and the course software in a package or .DMG file.
Our computer support person may have suggested something along the lines of distributing two files -- one is the .DMG containing the bundle and number two is the text file of the modified eclipse.ini.
The installation procedure is to first install the .DMG and next download and copy the replacement eclipse.ini to th
Re: (Score:2)
That is exactly what I want to do, change eclipse.ini to point to the directory tree inside the bundle and redistribute that with an OpenJDK and the course software in a package or .DMG file.
Our computer support person may have suggested something along the lines of distributing two files -- one is the .DMG containing the bundle and number two is the text file of the modified eclipse.ini.
The installation procedure is to first install the .DMG and next download and copy the replacement eclipse.ini to the correct location in the directory tree. I was holding out against that work around, but with the semester starting soon, this is probably the way to go.
Sorry.
The black-hat hackers have made it so we can't have nice things anymore.
But can you see how what you wanted to do pretty much flies in the face of Signed Software; and also how, in the vastly common-case, Signed Software is a huge benefit for the User?
Re: (Score:2)
As much as a benefit as Free/Open Source software?
Where's the balance?
Or is this the problem that the GPLv3 and AGPL etc. were intended to solve (this is basically the Tivo-ization issue right??)
Cute (Score:3)
It's really cute that they call this, an incident by a company that can spend $6B on a new spaceship campus, but obviously nothing on QA or to have someone monitor such an automated system to prevent such problems, a "mistake".
They should call it what this really was: a total fuckup.
Re: (Score:3)
That campus is a total fuckup. Its an open plan office designed by some extrovert asshole. The engineers literally revolted when they were told they have to work open plan. Eventually Apple added some conventional buildings to the campus and stuck the marketing folks in the open plan space. After Covid19 guess who is working from home.
Re:Cute (Score:5, Interesting)
Worse than that, it's probably an irreparable mistake.
Most developers eventually drop support for older operating systems. Blocking the signing key for the app, assuming it affects all versions of the app, would also block the use of the app on older devices. Permanently.
Why is this permanent? Because unless Apple has changed things very recently, the App Store does not provide any way to ship bug fixes (or even security updates) for previous versions of the app. Every version that you push to the store has to have a higher version number than the previous version number, which means that it will get downloaded by every device. (And, lest you think that this is minor content ingestion bug that Apple will surely fix in response to an incident like this, I would point out that people and companies have been complaining to Apple about this design flaw in their ingest system for at least a decade, and they have done nothing. In other words, I'd bet on self-levitating pigs happening sooner.)
What this means is that the only way to "fix" those older devices would be to upload a new build with a new version that A. has support for those older devices, B. still contains all of the fixes for newer devices (e.g. notch support), C. uses the latest version of any data structures so that it won't crap itself when restored from a backup of the latest version of the app, D. does so in a way that won't break on older iOS versions, and E. contains full migration support for every version of data structures that has ever been shipped, so that it won't break when backups of versions from older devices are restored. (You should be doing E. anyway, but you'd be surprised how often this is not the case.)
Even if a company decided that it was worth spending six months developing such a compatible version to try to get those customers back, the multi-month delay would guarantee that few (if any) of those customers would actually come back, so you can safely assume that no one would spend that effort just for the single-digit percentage of users on older iOS versions.
I guess if you were really careful you could go through your prior releases and, for each version that was the last version prior to dropping support for some version of iOS, create a new build that contains an iOS version check that limits the app's operation to the set of iOS versions that were dropped in the following build (so that no device that could have ever run a later build will run it), and shows a "Please wait. We'll upload a version compatible with your device soon." message on later devices. Then you could go through and rebuild all of those previous versions. But even that would require a lot of effort, and it would take at least a couple of weeks to properly integrate that detection code/UI and test it across hundreds of different hardware versions.
And that's after you get to the point where you can even build old versions of your app. Remember that Apple removes SDK support for building apps for older operating systems on an annual basis (again, something that some of us have been complaining about for more than a decade). And you can't always successfully compile code targeted at an older SDK against the latest SDK for various reasons (not the least of which is Apple's use of "built on or after" checks). So before you can even start building and testing your fix, you would have to buy a fleet of old Macs, install a bunch of old macOS versions on them, and then install a bunch of old versions of Xcode.
And if you want to test on actual hardware, rather than the simulator, you would have to find old test devices that never got upgraded to various releases of iOS. Most companies keep those around for as long as they support that version of iOS, but afterwards, probably not. And because it is not possible to downgrade iOS (again because of an arbitrary Apple li
apple is pushing hard for app store only with cene (Score:1)
apple is pushing hard for app store only with censorship and the only thing that will save them is to have an censorship free part of the store if they don't want to give up the lock in.
Don't Deal With Apple (Score:1)
I have an iPad Mini for user testing and that's all. I avoid everything Apple because of their general unfriendliness to developers. OK, mistakes happen, but you rectify it. You find the customers, you tell them it was a mistake by Apple, you make a generous payment to the developer.
Of two minds (Score:1)
All of my garbage apps have this error (Score:1)