I have poor understanding in this question. The major step to distribute any application is the code signing, it signs application with dependant dynamic library. As I understand OS will check signed application during installation and subsequent calls. If application or dynamic library was changed then os rejects the launch. Many dylib files are supplied to conform LGPL license and therefore dylib files can be potentially substituted by user later. But it will break the launch of application (because signed dylib was replaced). Are my assumptions correct? Maybe there is comprehensive book/guide which covers this topic? I found apple documentation pretty bad
That's not how this works.
The signature of the library isn't going to matter if the entire library will be replaced. What matters is the signature of the main executable of the process, specifically whether it enforces library validation. But even if you disable library validation, that is likely only going to work for dlopen() scenarios.
The problem is that app bundles are signed as a whole. Even non-executable resource files within the bundle are hashed, and then this list of hashes is hashed again and stored in the code signature of the main binary. While it looks syntactically possible to exclude files from this, I don't know whether Gatekeeper would accept this, and I don't know whether it would work for dylibs in particular.
But even if you found a combination that works, it will likely only be a matter of time before Apple breaks it, because the whole point of codesigning is that only pre-approved binaries are allowed to be executed.
The simple solution is: if you replace the library, you re-sign the entire bundle.