I'm a hobbyist programmer who maintains some programs written in C# (game mods in particular). I distribute those by uploading the DLL in a zip file to a website. Whenever I upload a new version, users would get a "suspicious download" warning. From what I heard, this warning goes away after a lot of people have downloaded it, but this has to happen for every new release.
I know that code signing can help in this area and I see three options:
While I can afford option 1, I feel this is still too steep a price for what's just a hobby. Plus I'm fine with sticking to option 3. However I've been looking into option 2 and wondering if it provides any benefits compared to option 3.
All great questions. By way of quick background for other readers, both the browser and the operating system might flag any EXE, DLL, MSI, etc. downloaded, with or without a trusted signature. They all have different algorithms for how they decide to flag, the precise detials of which are not widely publicized (if at all). However the common factors I know of are:
Do browsers/operating systems indeed have some reputation system that causes the "suspicious download" for unsigned binaries to disappear after a while?
Yes they do, but in my anecdotal experience it requires a LOT more reputation than you're likely to accumulate unless your application is extremely popular. For example the installer for 7zip has no digital signature and I've never seen Chrome or Windows flag it. But that's 7Zip.
Would using a self-signed certificate allow that reputation to carry over to new builds?
Again based on my experience, this is true for trusted certificates, although this benefit resets with each renewal. I'm afraid I can't answer with certainty if this would work for untrusted self-signed certs. (If you asked me to wager, though, I would say not, for reasons below).
Are there any other benefits/downsides to using self signed certificates for code signing, compared to not signing the binary at all?
I am speculating, but there is good reason for an algorithm to be more suspicious of an untrusted self-signature than no signature at all, interpreting it as a possible attempt to spoof a reputable author. Otherwise you could self-sign a certificate as "Microsoft Corporation", for example, and enough users might be fooled by that into ignoring and clicking past the warning for the algorithm to eventually start passing it. While there are legitimate cases for self-signing (corporate environments with a single trusted root, development machines, etc.), the root would be in the trusted root store in those cases. For distribution to the general public, it could easily be viewed as sketchy.
Is there perhaps a fourth option that I'm not aware of?
There is, but it's probably not an option for you specifically as a hobbyist - extended validation (EV) certificates give you the highest level of trust possible right out of the gate, potentially even with no prior download history.
They are much more expensive than a regular cert, and they're only available if you're a legally incorporated entity. Even then there are numerous hoops to go through. But as a distributor of fairly esoteric B2B software with frequnt updates and professional customers with little patience for technical difficulties, we had no choice but to get one or deal with constant recurring issues, both browser and OS. Ever since, we have never to my knowledge had a download blocked by the browser or the OS (Z-Scalar and systems like that are a different matter, but you usually have no option there except to beg your customer to beg their IT department to whitelist you).