Is it possible to have something like AttributeUsage to restrict the use of an attribute to a specific class (not just AttributeTargets.Class - that would be any class)?
No. There is nothing in the framework that would do this.
However, the code that uses the attribute in question could always check to make sure that the class's type is the specific class (or one of its subclasses).
Attributes, by themselves, do nothing - so this should have the same effect.