Mysterious Avid Issue Knocks Out Mac Pro Workstations Across Hollywood (variety.com) 98
A possible computer virus attack has knocked out Mac Pro workstations for many film and TV editors across Los Angeles. According to Variety, the issue -- which is causing the workstations to refuse to reboot -- is widespread among users of Mac Pro computers running older versions of Apple's operating system as well as Avid's Media Composer software. From the report: Avid said in a statement that it was aware of the issue: "Avid is aware of the reboot issue affecting Apple Mac Pro devices running some Avid products, which arose late yesterday. This issue is top priority for our engineering and support teams, who have been working diligently to determine and resolve the root cause. As we learn more, we will immediately publish information -- directly to our customers and via our community forums and social media platforms -- in order to resolve this issue for all affected customers and prevent any further issues."
"A lot of L.A. post shops and people out on shows having their Macs slowly crash," reported video post-production consultant Matt Penn on Twitter. Freelance film editor Marcus Pun reposted a message from a popular Avid Facebook user group, advising users not to turn off their workstations. Other users reported that multiple computers at their company were affected by the issue, with social media chatter indicating that a number of different companies, and even major shows like "Modern Family," were affected by the issue. UPDATE: The issue appears to be caused by a Google Chrome update gone haywire.
"A lot of L.A. post shops and people out on shows having their Macs slowly crash," reported video post-production consultant Matt Penn on Twitter. Freelance film editor Marcus Pun reposted a message from a popular Avid Facebook user group, advising users not to turn off their workstations. Other users reported that multiple computers at their company were affected by the issue, with social media chatter indicating that a number of different companies, and even major shows like "Modern Family," were affected by the issue. UPDATE: The issue appears to be caused by a Google Chrome update gone haywire.
Apple (Score:4, Funny)
Macs. They just wo-
Re: (Score:2)
Macs. They just wo-
...until the boutique software your run suffers from a supply chain compromise?
Re: (Score:2)
It's been confirmed it's actually google chrome doing it, not Avid
So, hows your botique google chrome...?
and...? (Score:2)
And this is better, how? Still portends a serious problem.
Re: (Score:2)
It's been confirmed it's actually google chrome doing it, not Avid
So, hows your botique google chrome...?
Not boutique? Honestly I'm not sure what you're trying to say. It's a combination of Chrome and third party software that requires people to disable security features, if I read the article correctly. Does this make it an Apple problem again somehow and I missed it?
Re:Apple (Score:5, Funny)
It's a combination of Chrome and third party software that requires people to disable security features, if I read the article correctly.
I see nothing that could possibly go wrong in that scenario.
Re: (Score:2)
What is "botique"? Is that a mispelling of "boutique" or "botox"?
And what does the fact that it is Google Chrome have to do with the boutiqueness of the software?
Re: (Score:2)
What, you mean the same problem Microsoft has been facing for decades and which has always been Microsoft's fault? Regardless if it was called Flash or Quicktime or whatever? Microsoft's fault.
Yeah, well, this time it's Apple's fault. Apple machines stop working, Apple's fault. It's the software that's gumming it up? No, that's not how it works. APPLE'S FAULT.
Re: (Score:3)
I'm curious what magic allows a Mac to "slowly crash". I'm guessing they mean "get less responsive", like in low-memory scenarios?
Anyhow, it's interesting that this is so widespread and apparently synchronized. That may mean this was very fast-spreading wormable malware, or, perhaps more likely, it was a payload introduced supply-side from Avid or some similarly common source. It wouldn't be the first time malware has slipped in as a payload with official software.
It's also somewhat unusual for modern ma
Re:Apple (Score:5, Informative)
No, it's a strange combination of factors. It only happened now because most likely, Google updated Chrome yesterday.
Effectively, there's a bug in Chrome and it destroys the /var symlink. Chrome users don't notice it because macOS comes with System Integrity Protection which prevents Chrome from messing up the system to begin with.
But Avid users DID disable SIP simply because they wanted to get their hardware working, and their hardware did not go through Apple's process to certify and sign drivers. Likely because the accelerator card manufacturer was too cheap and decided to skip it on their expensive card.
Now the Google Chrome bug comes into play, macOS' self-preservation feature was disabled, leading to an unbootable system.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Disabling SIP is pretty much the only way I've found to keep the ReFind bootloader from being overwritten, if you want to dual-boot Linux on an iMac.
Re: Apple (Score:2)
I will log on to my linux desktop run now and try to delete my /var as a regular user without fear of consequence.
Just trying to delete /var is retarded on Google's parts, but presumably this means Chrome requires admin privileges for /something/ and that is unacceptable for a web browser. Is this some crazy thing like WebUSB that Apple is refusing to provide an API for?
At some point, if you take full control of an OS to circumvent the vendor, "you break it you bought it". To avoid that, don't offer featu
Re: (Score:2, Troll)
How on earth can a mere application bugger up the Operating System?
What a total and complete hunk of shit is an operating system that cannot even maintain its own integrity.
This is an Apple problem. How on earth could the Operating System permit this to happen?
Re:Apple (Score:5, Informative)
I'm not certain what you missed in GP's post...
An irresponsible browser update destroyed part of the OS.
The OS has a protection layer that prevents this from happening. No problem for most users.
The irresponsible and ridiculously expensive software used to edit video couldn't be arsed to sign or certify their drivers.
The same OS protection layer prevents said software/drivers from running without certification.
Users turned off the protection layer to run their ridiculously expensive software.
Ergo, the protection that prevented the irresponsible browser from buggering the system in all other cases was turned off by the users because of the irresponsible video software.
IOW, you can't disable the brake and then blame the manufacturer when the car doesn't stop.
protection layer needs to be off to use non apple (Score:2)
protection layer needs to be off to use non apple hardware is the bigger issue. windows does not force that on most hardware.
Re: protection layer needs to be off to use non a (Score:2)
Although Iâ(TM)m wondering if they could have been more selective and disabled SIP for drivers (kexts) and kept it enabled for the file system. Some details of those options here: https://forums.developer.apple... [apple.com]
Re: (Score:2)
Re: (Score:1)
The same OS protection layer prevents said software/drivers from running without certification.
And that's fucked up. Their "certification" is bullshit. The OS is supposed to be isolated from "software/drivers".
*sigh* Some day they'll get real and make the OS truly read only
Re: (Score:2)
The next version of the OS (due to be released next month) uses a read-only system volume alongside the read/write data volume.
Re: (Score:2)
Re: (Score:2)
The users are doing what they need to do in order to get their work done. If they screwed up anywhere, it was counting on Apple.
Re: (Score:2)
To bring in the car analogy, if I'm riding a motorcycle without a helmet in a state where that's legal, and a car hits me from behind, the fault is with driver, even though wearing a helmet might have saved my life.
Re: (Score:1)
What *are* those other reasons?
Re: (Score:2)
What *are* those other reasons [to disable SIP]?
If you need to make any edit to anything under /System/ or other protected directories. startup scripts, changes to the login screen, etc. Note that these are changes for a corporate or school environment, which also matches the Avid-user/Hollywood demographic in terms of homogeneous computing setup. I expect some highschools to have had this issue recently.
Re: (Score:1)
CAR ANALOGY WOOT
Re: Apple (Score:2)
Re: (Score:2)
Ah, interesting. I guess you've heard more than the official story. That makes much more sense than malware. I guess it's also true for software that crashes are caused by simultaneous failures. Just ordinary bugs colliding with bad practices by Avid users.
That's pretty amazing, though, that Chrome would clobber the /var symlink. Hopefully as part of Chrome's automated test suite, SIP is turned OFF on macOS test machines to help catch future issues like this - along with specific regression tests check
Re: (Score:2)
> it would be unforgivable if it ever happened again
--It's d--n near unforgivable that they allowed it to happen ONCE. Chrome Canary has been very unstable for the last few weeks on OSX, and now mainline Chrome is making the box unbootable? Fix your s--t, Google!
Re: (Score:2)
How much does certification cost? For Windows a basic cert (enough to install the driver with a "do you trust these guys?" request to the user) is about $200.
Re: (Score:2)
The Mac's Mach kernel is tuned for desktop work, similar to some Linux distro's. This allows "foreground" applications to be more responsive while trading throughput and swapping out inactive programs more aggressively as well as pre-empting the desktop applications over background applications in case of resource contention. This leads to a 'slow crash' or the spinning wheel of death where a single application could be hanging but everything "important" is still perfectly chugging along.
In this case it see
Cue Nelson... (Score:2)
"Ha-ha!"
https://youtu.be/kdOPBP9vuZA [youtu.be]
It's not Avid doing it, it's Google Chrome (Score:5, Informative)
Re: (Score:3)
So spyware was sinking its teeth even deeper into places it shouldn't be. Who woulda thunk it.
Re: (Score:1)
Spyware doesn't delete /var because that would draw attention to it.
Re: (Score:2)
You are obviously wrong because in this case that is exactly what the spyware did.
#varsectomy? (Score:2)
https://mrmacintosh.com/google... [mrmacintosh.com]
This does not merit a catchy title. Really, there is only one beautiful and pure vulnerability, perfect in form and lacking nothing, and its name is Row Hammer .
Oh, your vuln exploits a symlink or whatever? Well Row Hammer uses physics to flip targeted bits in memory and that will never stop being impressive.
Re: (Score:2)
This does not merit a catchy title. Really, there is only one beautiful and pure vulnerability, perfect in form and lacking nothing, and its name is Row Hammer .
The sheer ingenuity of it and the fact that it actually works is indeed a thing of beauty.
Re:It's not Avid doing it, it's Google Chrome (Score:4, Insightful)
If a browser issue is causing your workstation to not boot, the problem is with the operating system.
Re:It's not Avid doing it, it's Google Chrome (Score:5, Informative)
If a browser issue is causing your workstation to not boot, the problem is with the operating system.
Per the article, there is a standard security feature that many people running Avid disable in order to get third party drivers to run, and the issue only occurs in these systems. So it's kind of a Google and Avid and Apple problem.
Re: (Score:1)
Not being able to get third party drivers to run without disabling security is also a problem with the operating system.
There's no way around the fact that this problem represents OSX design flaws at more than one level.
Re: (Score:2)
Avid could sign their driver, and it would load without disabling SIP. The moment you load an unsigned kernel extension, isn't really all bets are off? Isn't that true with other operating systems too? Or does Window's have a fine grained controls, such that you can limit what access a driver has access to? I would think such an architecture would come with severe performance penalties.
Re: (Score:3)
So what your saying is Apple should turn off driver signing checks for drivers because people might turn driver signing checks off on account of third party developers not signing their drivers.
Thats a nutty take dude.
Re: (Score:3)
Re:It's not Avid doing it, it's Google Chrome (Score:4)
If the "browser issue" is deleting a system file then the problem is with the browser that's deleting the system file.
If users need to turn of OS-level file-system protections of said system file in order to get some drivers to install/work then the problem is with the makers of the drivers.
Re: (Score:2)
No. The problem is with the user that decided to defeat security. It appears that those users learned WHY they should not do that sort of thing, and hopefully they will go bankrupt and thus not have the opportunity to be so stupid again. Besides it will look good on their resume that the lost their last job because they "bankrupted the company by defeating security policy in order to permit shoddily crap to function" -- I should nope they will be on the dole for the remainder of their lives.
I have to con
defeating security to run nvidia = don't use mac (Score:2)
defeating security to run nvidia = don't use mac or other hardware that does not give you choice.
Re:It's not Avid doing it, it's Google Chrome (Score:4, Interesting)
I hear that Davinci Resolve [blackmagicdesign.com] is quite nice, and with really good licensing (since the company prefers to sell high-priced UI hardware for it).
But if the real problem was turning off SIP to run Nvidia cards (presumably for CUDA), then the real WTF is that Apple has so far refused to sign (yes you read that correctly) Nvidia drivers for 10.14 and that support newer cards like 1060, and without any explanation. This has been a problem for like a year now, so the absolute fucking morons seem to be at Apple. I was not aware that disabling SIP could get around that, but not surprised. I'm also at a loss to explain why a web browser can fuck with the /var symlink.
I'm not sure that anyone knows why Apple is currently being this way, but they've got a good reason to be pissy with Nvidia. Around 2010-2012 Nvidia supplied GPU chips with physical flaws in the chip carrier that among other things caused a large number of otherwise very well designed MacBookPro laptops to become unusable without significant board-level repairs of the type that Apple does not do.
Re: (Score:2)
Usually when Apple refuses to do something it's to enrich themselves. Could they be hoping to sell more AMD graphics cards with their systems, rather than allowing people to buy much cheaper generic Nvidia ones?
The other possibility is to make it harder for people to build a Hackintosh. Nvidia cards are more popular on the Windows side and forcing you to either use dodgy drivers to an AMD card might discourage some people.
Re: (Score:2)
Re: (Score:1)
The problem wasn't Nvidia, it was the EU's mandate for lead-free solder in products sold there. The engineer who invented surface mount technology died before that mandate happened and he never tested for other soldering compounds.
GPUs have very rapid thermal cycles compared to CPUs. What was less well known is that many of the solder balls used in a surface mount have voids inside them. For lead this doesn't matter because it's incredibly pliable. Tin amalgams are a lot less like lead and a lot more like g
Re: (Score:2)
Re: (Score:2)
To be fair, apple blocks users by default*. To apple, "it shouldn't be possible", not in the walled garden. But anyone worth talking to knows better than to hide behind that phrase, than to write code that leans heavily on flow assumptions.
>>the problem is with the operating system
Hmm. To be fair to that, OSX probably shouldn't be going into shit-the-bed unbootable KP loop mode. Though I also have to wonder what the hell deletePathIfSymlink is up to, from a chrome updater.
*people using external video cards or some shit
Re: (Score:2)
If a browser issue is causing your workstation to not boot, the problem is with the operating system.
Apparently the problem is with Google’s Keystone auto-update daemon, which Google tries to foist on everyone but a sane person would never let run on their computer.
Re:It's not Avid doing it, it's Google Chrome (Score:4, Informative)
Since 2014, the macOS ships with a feature called System Integrity Protection [wikipedia.org] turned on by default. When enabled, this feature prevents all users (even root) from modifying areas of the disk that are supposed to be owned by the system. In this case, it prevents the issue [mrmacintosh.com] by refusing to allow Chrome to break the system.
Now, lots of people, especially on /. would be very insistent that SIP be a feature that end users can disable. After all, it's your computer, you should (suitably warned) have the right to tinker with the system as you want, even if that means bricking your machine. Protection for the naive should not impact the power user. And indeed, MacOS obliges and allows the user to disable SIP (to prevent remote disablement, you have to reboot the machine into a separate partition that can set a flag to disable it, so you have to really mean it).
So, is the problem with the OS? From the OS perspective, these users exercised their choice and deliberately went out of their way to disable the thing that would have preserved the integrity of the system, having been warned that it's not an excellent idea to do so. But hey, software freedom right?
My fear here isn't so much that people give a shit about whether it's the OS or Chrome or Avid's fault, it's that these episodes strongly incentivize companies not to allow such power-user functionality because they get bad press when someone clicks "yes" to the "are you absolutely sure you want to do this" button and then they get bad press over the dumb fallout. I'm sure they want to support the power users as best they can within the constraints, but those folks have to take responsibility -- if you turn off system protection and then some bad software breaks your system, well, shit don't do that.
Re: (Score:2, Interesting)
Seems pretty obvious that 'allow unsigned drivers to load' should be a separate option from the rest of the system integrity protection. It's going to be a common requirement for people with obscure hardware.
Re: (Score:3)
Why? What's the difference? An unsigned driver is basically unsigned kernel code, which basically means the two options are the same security wise. In fact, they're really aliases of each other because an unsigned driver pretty much disables system integrity - kernel mode things have a habit of doing th
Re: (Score:2)
Re: (Score:2)
Oh yeah, ton of blame for Chrome.
I was really looking at the angle of how a browser was even able to brick the system, and I found the answer was because people turned off the anti-brick protection feature.
Re: (Score:2)
Re: (Score:3)
And the reason it showed up in Avid users first is that many of them are forced to disable SIP (which would prevent Chrome from modifying the /var symlink) in order to run third party video cards. Most users have no need to do this.
Re:It's not Avid doing it, it's Google Chrome (Score:4, Interesting)
No user should need to do this. SIP doesn't prevent third-party drivers. They just have to be properly signed with an Apple Developer ID certificate.
There are basically no legitimate reasons for an end user to turn off SIP, with the possible exception of hand-wiping enough of the OS to let you downgrade to an earlier version. If Avid is in any way encouraging users to turn off SIP, that's incredibly bad.
Re: (Score:2)
Re: It's not Avid doing it, it's Google Chrome (Score:3)
Appleâ(TM)s refusal to sign nVidia drivers puts a lot of people in this position. Apple has actively blocked nVidia drivers from the MacOS ecosystem.
What do video editors need? The fastest video cards.
Re: (Score:1)
Apple has always had shit third-party driver support. They want to own the whole horizontal.
Re: It's not Avid doing it, it's Google Chrome (Score:4, Interesting)
Unless I'm missing something pretty significant, I'm pretty sure pretty much everything you just said is incorrect. Certifying the driver with a given card is typically the job of the company building the NVIDIA-based card, not NVIDIA, and not Apple. The card maker is responsible for asking Apple to grant kext signing permission on their Developer ID certificate, after which they can sign their own kexts. Apple doesn't sign any kernel extensions except for the ones that they ship as part of the OS, and NVIDIA probably doesn't, either, because each card is likely to want its own kext with a specific version of the driver and an appropriate plist for matching against the card.
Now to be fair, what a lot of folks do is hack one of the stock NVIDIA drivers to point it at a different, potentially unsupported card by changing the matching dictionary. But the right way to do that is with a codeless kext that merely forces the stock driver to match against different hardware, NOT by modifying the built-in Apple driver directly. And by doing it that way, there is no need to turn off SIP.
Unfortunately, Apple, in their infinite wisdom, does not allow even codeless kexts to be unsigned, which is kind of a pain if you're hacking the driver yourself. That said, you don't have to disable SIP just to use unsigned kexts. All you have to do is set "kext-dev-mode=1" as part of your boot args.
Like I said, there is no reason why ANY end user should EVER be disabling SIP. Ever. Ever. Ever.
Re: It's not Avid doing it, it's Google Chrome (Score:4, Informative)
Actually, it looks like I'm a little out of date. Apple, in their infinite wisdom, recently removed the ability to load unsigned kexts, even for development. That's pretty awful.
Either way, the point remains that the board vendors should be signing and shipping codeless kexts. There's nothing inherently preventing them from doing so.
Re: (Score:2)
You should not have to disable major features to install an unsigned driver. I decide what i want to run on my computer. Unsigned drivers should simply require an additional verification step, in which the system signs the driver at install time. That still prevents clandestine installs without forcing the user to bend over for the vendor. This scheme is pure and simple a way to exert undue control over the market for peripherals. And yes, i know Microsoft does the same thing, but "Microsoft was doing it" i
Re: (Score:2)
Verification step authorized by who? The user presumably.
This is really the ultimate problem that SIP and other protections are trying to solve, which is that "ask the user to grant X" is not super meaningful anymore. I get asked to elevate to root/admin routinely across Linux/Mac/Windows machines, and while the prompt is nice (and often declined), I don't actually know what the code t
Re: (Score:2)
Saying "running an unsigned driver in the kernel should just require authorization" means that any of those mundane activities I authorized could actually be signing/loading a kernel deriver behind my back.
So make the authorization process different, and even more explicit. Make the user type "AUTHORIZE" or solve a little sudoku, or whatever makes you feel good about the process. Make it only available from within whatever tool your OS uses to manipulate driver settings. Just don't make the user disable protections that are in there for their benefit. That's ridiculous (which is why they are now being ridiculed.)
Re: (Score:2)
I complained that codeless kexts shouldn't require signing way back when they first proposed this. It's an utterly silly design flaw that inevitably leads to problems like this.
That said, I disagree with you for drivers in general. Unsigned drivers with actual code are a special kind of risky, because they're running in the kernel, which means if they can't be trusted, then neither can your entire computer. I could pretty easily write a kext that would send every keystroke you type to a server in some t
Re: (Score:2)
I guess this is why i run Linux when i want to get work done. Windows and Mac are for people who need someone to hold their dick for them while they piss. If you can't handle a UAC prompt, then maybe you should stick with an iPad. Or maybe just an abacus. The computer is there to serve me, not the other way around, and if i want to install a driver the OS shouldn't be making me do a little dance for its amusement. That you think such is acceptable is a sign of Stockholm syndrome. Beat me harder, ghost of St
Its not a user fault.. (Score:2)
>There are basically no legitimate reasons for an end user to turn off SIP
Except, you know, if they need to use a 3rd party driver for hardware that is not signed by an apple developer id...
Or do you think the end user should just, I dont know, write their own driver? Not use their hardware?
The fault here would lie with the hardware developer not providing a signed driver - however with Apple there can be MANY reasons for that, because Apple will only 'bless' them with the required certificate if Apple d
Re: (Score:2)
Scream at the hardware vendor until they spend the $99 for a developer program membership and sign the kext themselves?
I mean, it's a pretty low bar.
Re: (Score:2)
Re: (Score:3)
Not sure why you think it's a good idea that you would need a manufacturer's permission to run a driver. Having to disable security because you still don't own the product you purchased tells me that there is more wrong here than blaming the user. Security of a product you own in the hands of another party is not security.
Re: (Score:2)
I think you're over-cooking a bit there.
You don't need the manufacturer's (apple) permission to run a driver. What you do need, is to absolve them of supporting you if you choose to do something they don't support. You do indeed own the product, but the moment you want them to help you, or give you a (free) software update or whatever else, then you're obviously still in their pocket to some extent. If you don't want to be any part of Apple after you bought your mac, then turn off SIP and updates and suppor
Re: (Score:2)
> There are basically no legitimate reasons for an end user to turn off SIP
--Really? How unimaginative of you. Some of us like to dual-boot into Linux without having ReFind overwritten at random.
Re: (Score:3)
"Chrome updated to 77.0.3865.90 which appears to be the latest stable version (I was actually surprised it still supports 10.10 TBH)"
Why surprised? I would doubt that there has been any change in the OSX system APIs ever. Wrapper crap, for sure. But any changes of importance to the OS base, kind of doubtful.
Good time to buy a Mac Pro on sale (Score:1)
nvidia and non apple ati card lockout stage 1 (Score:2)
nvidia and non apple ati card lockout stage 1
apple store says buy an new mac pro starting at 5999
It must be those Intel CPUs! (Score:3)
"A lot of L.A. post shops and people out on shows having their Macs slowly crash,"
Back when Apple used Motorola chips, they were able to crash astonishingly fast. We used to remark on how quickly they could shut down before [youtube.com].
user error (Score:2)
NEVER! NEVER! allow automatic updating on production machines. They shouldn't even have Chrome on them. These aren't computers they are video editing workstations that are responsible for creating your companies product. You install your workflow on a base operating system only. Every update is tested in house on a non-production machine before being rolled out.
Re: (Score:1)
It's a workstation and should be reserved to that purpose. If the OS is multiple versions behind and requires kernel level functionality be disabled to let the hardware run then yeah, maybe don't install ANY NEW SOFTWARE on it.