Is it possible to make a Flash embedded in a browser access a DLL, by making it signed or whatnot? I know that it's possible with Java Applets.
No. By far your best bet is going to be exposing whatever methods you need in your Class Library as web methods and letting Flash call them that way.