Filesystem Hierarchy Standard: Where should my package install files in Linux?

I'm looking for help understanding the FHS, particularly as it applies to software I'm developing. There seem to be nuances that neither I nor my colleagues are aware of.

I'm especially confused about whether the executable binaries are considered "shareable" files and/or "locally installed software" and/or "add-on application software packages". I wouldn't expect those to be mutually exclusive, but the FHS gives different locations for each.

For background: The software is a partially-privileged system service, part of a suite of security software. It's not considered critical to the boot process. I'll be distributing the software as deb and rpm packages. It's commercial, closed-source software.

/usr: "shareable, read-only data" [FHS 4.1], where "shareable" is "can be stored on one host and used on others" [FHS 2] Seems to fit, since AFAIK the binaries could be installed on one volume that's mounted to several hosts.
/usr/local: "Locally installed software must be placed within /usr/local rather than /usr" Also seems to fit, since the binaries could be installed locally.
/opt: "for the installation of add-on application software packages" [FHS 3.13] Since the software isn't part of a distro, presumably it would count as "add-on", and it's being distributed as a "package".

So which is right, and why? I'm hoping that when I understand that I'll be able to figure out the rest.

Solution

In my opinion:

/usr - Leave it alone, IMO it's for the OS installation and the OS installation only. Per FHS 3.0: "Large software packages must not use a direct subdirectory under the /usr hierarchy." So you're sharing /usr/bin with the OS.
/usr/local - Tends to be used by local admins, often to build software, so isn't really the safest place. I like to think of /usr/local as the "local" playground for the system's admins - IMO it's theirs to put their "locally installed (and likely locally built) software" in.
/opt - Designed for non-OS packages. Avoiding collisions in /opt is easy if you have a distinctive name. Per FHS 3.0: "/opt is reserved for the installation of add-on application software packages." That seems to be exactly what you're delivering. I suspect it wouldn't be hard to get a unique LANANA-registered name, giving you at least some degree of certainty that you won't have a collision. I suspect the reason many organizations don't use /opt is that making your software self-contained - like being able to locate its shared objects - is a bit harder.

Given all that, IMO the best place is /opt. You're less likely to have a name collision, and less likely for a local admin cleaning a botched build out of /usr/local accidentally nuking your installed software.