I have a Mac app mostly built with Xcode that uses a dynamic library (dylib) built with Free Pascal via Lazarus. When I turned on Apple's "Hardened Runtime" feature, the dylib stopped working, until I checked the option "Allow DYLD Environment Variables". This is described as "Allows an application to be impacted by DYLD environment variables, which can be used to inject code into the process." That code injection bit sounds like something I'd like to avoid. Any idea why this happens or what I can do about it?
To answer some questions asked in a comment:
I'm not sure exactly what failed, as I didn't write the dylib. It may have been trying to talk to a server over the Internet. All I know right now is that it returned an unexpected and unhelpful error code.
The app calls functions in the dylib using dlopen and dlsym. The dylib is in the Contents/Frameworks subdirectory of the app. The app does not set any environment variables, either in the Info.plist or in code.
otool -L output for dylib:
@rpath/lib<redacted>.dylib (compatibility version 0.0.0, current version 0.0.0)
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1259.0.0)
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1404.47.0)
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.13.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
otool -L output for main executable:
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.251.4)
/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 492.0.0)
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
@rpath/Quesa.framework/Versions/A/Quesa (compatibility version 1.6.0, current version 2.0.0)
@executable_path/../Frameworks/Ming.framework/Versions/A/Ming (compatibility version 0.0.0, current version 0.0.0)
@rpath/SBEngineLib4.framework/Versions/A/SBEngineLib4 (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 274.22.0)
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 50.1.0)
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 158.0.0)
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
/System/Library/Frameworks/AddressBook.framework/Versions/A/AddressBook (compatibility version 1.0.0, current version 1893.0.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.250.1)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit (compatibility version 1.0.0, current version 607.1.40)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1570.15.0)
/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1671.40.118)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1570.15.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 944.3.0)
/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
Update: I set a breakpoint at dlopen, and found that the dylib in question was trying to open another dylib, libiconv.dylib, by name rather than full path. It turns out that fails in the hardened runtime without the special entitlement.
The Free Pascal dylib apparently does dlopen("libiconv.dylib", ...) with just the file name and no path. This relies on dyld searching in $LD_LIBRARY_PATH, $DYLD_LIBRARY_PATH, current working directory, and $DYLD_FALLBACK_LIBRARY_PATH (or, if that last is undefined, $HOME/lib, /usr/local/lib, and /usr/lib).
However, dyld won't search anywhere for hardened-runtime processes without the Allow DYLD Environment Variables entitlement. Only full paths will work.
The dylib will either have to be changed somehow (if you can), or you'll need to enable Allow DYLD Environment Variables despite the risks (which I don't think are particularly severe).