My .NET application is 32-bit because I use TWAIN inside it. But I need to write 64-bit library or app and call its methods from my program. I must create a 64-bit library to interact with WMI SMO, and its functions don't work properly if it compiled as 32-bit dll.
How can I resolve this issue? Is it possible to use a 64-bit library from a 32-bit application?
You cannot reference a 32-bit library from a 64-bit executable and viceversa. This is a technical limitation.
However, depending on your other constraints, you may decide to employ a service-based architecture to avoid issues like yours and be more agile in the future.
In fact, you could write "micro" services specialized to solve one single purpose (scanning, interacting with WMI, etc...) and then call them from your application. This way, you are completely free to choose whichever technology/architecture you require to achieve every single task your application requires. Isolating your "features" in services, allows you to have a different evolutionary path per service, as each service is loosely coupled with each others.
This solution requires writing more code than using a library. First you need to create a windows service (msdn) to wrap your code, then you need to implement the communication channel. Have a look at WCF and WebApi2 to have an idea of what you can achieve.