Does OpenSSL have any support for operations in the quadratic extension field (or, alternatively, operations with complex numbers) using the BN API?
If not, are there any other open source libraries that do?
You may want to look at PBC. This is a library for computing pairings on some elliptic curves, which implies using field extensions of various degrees, including quadratic extensions for some fields.