Upgraded to Big Sur and now my codesigning fails on 10.11 and earlier

Awesome. I think this will work for a long time. Unless Apple wants to break a lot of existing .pkg installers. I highly doubt they will fix productbuild on Big Sur or any of underlying libs that broke this. The beauty of the macports install is that the libraries used by xar are independent of what Apple includes (and maybe modifies).

1 Like

I’m in two minds about this. I understand the desire to support those on older operating systems, and I understand some people have legitimate reasons for staying on an older OS. However the fact is that SHA1 is not secure, and it hasn’t been for years, so it is not suitable for signing installers. Distributing securely signed installers is the proper thing to do for us and our customers. Is it really right to try and wriggle out of this one? It is probably for the greater good and Apple has done this wilfully so is clearly happy to shoulder the responsibility.

I strongly disagree. In that case the right thing for them to do would be to patch 10.9 and later to support the more securely signed installers. After all building for 10.9 is officially supported by Xcode 12.2 and introducing a breaking change to installers is a problem if you don’t plan to add support for the legacy systems.

1 Like

If we were talking about a serious vulnerability that placed their customers at risk I would agree about issuing a patch for an OS released in 2013 (which was last maintained 4 years ago I believe), but it’s not really the case here when all it does is potentially lock people out of new toys so I think they’re entitled to keep nudging consumers forwards in this kind of a way. We devs can stay on our Intel Macs running Catalina and XCode 12 for many years to come and still support those customers and even all the ones running M1 Macs if we so desire and I’m sure many will do just that.

The problem is that Apple is NOT taking responsibility for this at all. 100% of your customers will say “Your installer doesn’t work. Fix it.”

Thanks pflugshaupt, I managed to sign using your script, however, the pkg works on 10.9 and 10.14 (haven’t tested Catalina yet), but it’s failing on Big Sur (silicon): ".pkg can’t be installed because its digital signature is not trusted".

I’ve adapted my build script to build the universal binaries on Big Sur (silicon) and create the pkg. Then bring the pkg to an Intel Mac with Mojave, productsign the installer, then notarize it.
This works on 10.9 and 11.0, but requires two Macs and it’s tricky to fully automate.

I will try to install Big Sur on an Intel Mac to see if that “fixes” productsign, but this is also not ideal as it will prevent us from upgrading our build machine in the future.

Another alternative, which I’m not so fond of, is to have separate installers: one up to 10.15, and one for 11.0+.

Sure they will (if nobody explained this to them already), but I’m very happy to explain why the installer doesn’t work, who lit the fire and why they did it. I can have a fairly standard email templated ready for it. Apple acknowledges that the change is the expected behaviour, so to me that’s handing us the green light to indicate they understand the consequences and are taking on the responsibility for making the change.

I want to live in your world, where customers accept that and where they read the FAQ etc.

It’s simply not practical. We would have literally thousands (!) of tickets overnight and 99% of those customers would NOT be satisfied with a “Well, it’s Apple’s fault. Too bad”. Their reply, rightfully, would be “Every other developer solved this problem and only yours doesn’t work. Fix it”.

Anything else is naive.

1 Like

That’s not really the presentation I’d use with a customer. Every now and again devs shift up the minimum system requirements and provide frozen legacy installers that are no longer updated for the benefit of people that use older systems. Most customers holding on to ageing systems generally understand they won’t always get the latest updates forever, and they know eventually they will move on to a newer system in due course and then can hop back on the maintained path. As soon as I give a customer a way to get access to an older installer they’re more often than not pretty content, do you not find that to be the case?

It’s the way it was with transitioning from PPC or dropping 32-bit, it’s the way it will eventually be with Intel binaries when something gets introduced that makes the friction building for those too much to bear, and we have numerous mid-points of lesser significance that we’ve all come through more than once. Apple does this over and over again to force people to move forwards, they ignore the blow-back, and it works pretty much every time because they have the power to force change whether we like it or not. Sometimes it’s done for the customer’s benefit, sometimes it’s done for their benefit, occasionally it’s even done for the developer’s benefit although they probably don’t recognise it as such because the general resistance to change or irritation caused by doing unplanned and seemingly unnecessary work just takes over. I’ve been there many times, we all have.

I don’t think it’s any different this time, do you?

1 Like

No, it’s no different. Apple is extremely selfish. They do this kind of stuff every 2-3 years. What bothers me is that they have zero consideration for their developers. Nothing is ever clear. People have to find out the hard way and then scramble for solutions.

I think it’s ridiculous we have to keep an old machine around (or use a VM) just to sign our installers. Why doesn’t the new signing tool have an option to use the old signature, maybe with a warning, etc.? Instead, they silently change it and leave us with hours of research, forum posts, etc.

In the 20 years reFX has been doing plugins (yes, we started in January 2001), we had to update something Windows-specific exactly 1 time. Transitioning to 64-bit, and that was mainly getting newer VST-SDKs and compiler settings. Nothing Microsoft had anything to do with.

During that same time Apple went from OS9 to OSX. Then from PowerPC to Intel. Then from Intel to Apple Silicon. Every 2-3 OSX versions, old functionality gets deprecated and then stops working a few versions altogether later. It’s a constant struggle just to keep up.

That’s why I was so happy when I discovered JUCE. Now at least the majority of this BS is done for us. Now Apple had to do this without warning us and your reaction is “Oh well, just inform your customers and give them some old version”. Our customers simply don’t accept that. They paid the same amount of money, they want the same plugin to work on their “outdated” system the exact same way it works on the latest system.

You’re conflating a major change, like OS9 to OSX, with a relatively minor OS update.

5 Likes

For me my packages work on everything from 10.7 to Big Sur. But maybe I need to do more testing on actual hardware. I only have one Big Sur machine and it is the DTK. What did you test on? What OS did you use to create the package where you ripped the signature from?

Update: I ran some more checks and pkgutil --check-signature is a handy tool to check all the .pkg files. The packages I built verify just fine. I think the extremely important step to be successful is making sure the donor installer (foo.pkg) installs fine on all systems to support before ripping the signature from it.

I’ve re-tested everything and now it works! So I definitely made some mistakes yesterday. Sorry!

By the way, I don’t have the DTK, I have a MacBook Pro 13" with M1, running Big Sur 11.0.1. If you want me to test on the M1, send me the PKGs via pm.

I didn’t want to download MacPorts too, so I gave building xar from source a shot, and this seems to work:

  • Apply one-liner patch from https://askubuntu.com/a/1195797/674115
  • CPPFLAGS="-I/usr/local/opt/openssl@1.1/include" LDFLAGS="-L/usr/local/opt/openssl@1.1/lib" arch -arch x86_64 ./configure
    • Flags are from brew info openssl
  • arch -arch x86_64 make install

And with it the script works for me!
But also for me, stapling failed (after successful notarization)

Not sure if that is an issue specific to my setup, but while the xarsigned pkg passes notarization with success , I can’t staple the pkg on my dtk mac:

xcrun stapler staple foo.pkg

exits with error code 66, no error message displayed. According to man page, this is:

[EX_NOINPUT]       path cannot be found, is not code-signed, or is not of a 
supported file format, or, if the validate option is passed, the existing ticket is 
missing or invalid.

As soon as I sign the pkg with productsign instead of xarsign, the stapling works.

pkgutil --check-signature foo.pkg is happy with my xarsigned pkg, and spctl --verbose -vv --assess --type install foo.pkg says that the pkg is accepted and notarized.

So I guess I’ll ignore this error for now.

Weird - I’m not getting errors from the same xcrun stapler command.

I think I also found an issue with codesign. I’m currently building my plugins and installers on Big Sur silicon (MBP 13" M1). The strange thing is, the plugin works perfectly fine on OS X 10.9 and Logic 10.2.2, but fails to load on OS X 10.10 and with the same exact version of Logic (and auvaltool). The plugin also loads ok on macOS 10.13 to 11.1. I’ll try to check other versions in the following days.

Runinng auvaltool on 10.10, I get this error:

code signature invalid for '/Library/Audio/Plug-Ins/Components/Wires.component/Contents/MacOS/Wires'
FATAL ERROR: OpenAComponent: result: -50,0xFFFFFFCE

However, the same exact version of auvaltool on 10.9 (!) returns AU VALIDATION SUCCEEDED.

Replacing the signature with codesign on Mojave, fixes the issue.
Anyone else with the same issue?

1 Like

Ok, the issue is exactly the same as with productsign. More info here: https://www.kvraudio.com/forum/viewtopic.php?f=33&t=555648&start=30

codesign on Big Sur (silicon) doesn’t add sha1.

10.9 seems to ignore the signature, so auvaltool validates the plugin
10.10 can’t read sha256, so validation fails
10.11 seems to be ok, but I need to verify it properly

Now, how can we make codesign add sha1 on Big Sur (silicon)? Is there an alternative to codesign that we can use?

1 Like

Regarding the stapling, I too am finding that is failing using the xar technique.

I’ve got through a full notarise of a xar modified package and checked that on a separate computer after an upload/download cycle to be sure to get the quarantine bit set properly, it can run when the machine is online, but when it’s offline Catalina won’t run the package since it can’t go online and check with the notary service (it’s got to do that if there is nothing stapled to it). This all works properly with the exact same installer signed with productsign (with the minimum macOS version limitation, ofc).

The problem is clear if you run xcrun stapler staple -v, verbose output shows it’s in fact not happy about the code signature on a xar modified installer, and the package simply doesn’t get stapled. The spctl notarisation check works if there is the option to go online to complete the operation as it can just pull it from Apple directly.

I question the value of using the xar technique given that limitation. Doing this to support a collection of old operating systems that require the use of an obsolete and broken hashing algorithm when the consequence is not supporting the notarisation service for customers that need to use their systems offline does not really feel like a great trade from a user workflow or security perspective.

I also don’t much like the idea of keeping RSA private keys hanging around in an unencrypted form for xar’s benefit away from the keychain where they are tucked away safely. Maybe a moot point for those that are keen to keep using SHA1 signatures which can be forged anyway.

So it’s looking like ‘last known good’ is Catalina’s productsign in a VM for maximum compatibility, but that’s not going to provide much solace for anybody looking to move over to M1 for builds any time soon. I imagine that’s not much of a concern while many of us have faster Intel dev systems, but it’ll sting a bit when they bring out the beasts.

2 Likes

Thank you for digging deep and finding the root cause of the issues. This also invalidates that xar method I found for me.

For me this means I’ll sign and notarize on my 10.13 machine, which is annoying, but IMHO it is too early to drop <10.12.

While I could continue to develop on my intel machine I think for the future it is important to do the development on the new platform and threat it as the primary target that gets the most time running in a debugger to catch all those smaller issues that keep cropping up.

For future reference… I now ended up upgrading my intel machine to 10.15 to be used as a build machine while I’ll be developing on my m1 mbp. As others have stated before 10.15 with XCode 12 is the only safe way to build UB2 binaries and installers which can be loaded on pre 10.12 systems.
The solution with the custom xar tool I presented earlier and many people seem to have liked is not good for production use because the resulting installers do not load on 10.10 and cannot be used for offline installs on systems requiring notarisation. So please ignore the posts and script!

9 Likes