I need an OID to use to add a custom extension (already ASN1.encoded) to an X.509v3/PKCS#7 digital signature. I don't care if it's interoperable; this is a private proprietary application (actually, a research project). Right now I actually don't even care if it's an OID that's in use in some proprietary application, although obviously I couldn't use that OID for long.
Currently I've been using 1.2.3.4.5, which is in the test code for the crypto library I'm using (cryptlib). However, signature creation is crashing and I don't know if this might be why. I do know that the library rejects at least some invalid OID's when checking inputs.
I suspect that the problem is that you are not correctly encoding the OID you are using as an ASN.1 OBJECT ID object. I know of Cryptlib but haven't used it so I can't be sure why it's crashing, but I suspect that the reason will be invalid coding rather than just choosing an inappropriate OID value. I doubt that Cryptlib attaches any significance to the OID value you supply (unless, possibly, you supply the OID for a known certificate extenssion such as "basic constraints" with an unexpected value).
I notice from http://www.imc.org/ietf-pkix/pkix-oid.asn that PKIX defines an OID range for testing, and I expect that you will confuse no-one if you (ab)use an oid in that range for your own (internal, unofficial) testing. The testing OIDs are anything starting with 1.3.6.1.5.5.7.13.
I'm a bit concerned by your talk of "an X.509v3/PKCS#7 digital signature" ... X.509 and PKCS#7 are really quite different. I gather from your talk of a custom "Extension" that it's an X.509v3 certificate you're trying to create, not a PKCS#7 signature ... is that correct?