By using runningApplications of NSWorkspace, it is possible to get a list of running apps on Mac OS as NSRunningApplication objects, and from this get additional information like what application is in the foreground.
It is possible to identify the running application using their name (localizedName), but I'm sure that can be spoofed by rogue applications. Other things like bundleIdentifier seem better, but I believe that too could be spoofed.
I would imagine that pretty much all of the metadata could be spoofed for applications outside of the public app store, but for any apps gotten from the app store things like bundleIdentifier should be safe ways to identify an app, right?
If we include arbitrary apps that someone downloads from the Internet, is there any better way to identify an app as to filter out rogue apps? I realize that there may be no solution that has no drawbacks, but looking for a best-effort attempt.
As you mention, all of these things can be pretty easily spoofed. Having written a product that does exactly what you're describing professionally, the solution is relatively straightforward: fingerprint every version of every popular app into a massive database, and then fingerprint each app you discover on the machine and look them up in your database. When you discover an app you've never seen before, flag it for adding to your database.
Maintaining that database is very large and ongoing endeavor. That's where most of the value of the product is. The agent code is not that complicated. The up-to-date database is what customers pay for. It's a pretty hard space to get into.
You're correct that you can verify signatures to make sure that things downloaded from MAS or part of the OS are what they claim to be. This will get you started, but isn't nearly enough; there's just so much that doesn't come from MAS.
The other headache is that you can see what "apps" are currently running in NSWorkspace, but it's pretty messy what it means. A lot of things that you don't think of as "apps" show up in runningApplications, like MobileDeviceUpdater and nbagent. On the other hand, things like mysqld aren't. Fingerprinting from runningApplications can miss things that aren't in that list, or malicious apps could lie about their bundle path to make themselves look legitimate. You can use tools like lsof to see what files a process really has open, but it gets more and more complicated.
Best of luck; it's a deep rabbit hole with dozens of corner cases, and very little documentation.