I am trying to install a PKCS#11 module into Mozilla Firefox via an MSI setup application. After doing some search through the internet, I bumped into the modutil tool which has a wide variety of responsibilities especially in my case for installing / uninstalling the PKCS#11 module to NSS security databases. I downloaded the latest source code of modutil and built it using MozillaBuild.
Also, I writing a C code to add module to database using secmod_addnewmodule, which returns "-1" as a error.
Following is my code to add module where configdir is the path to secmoddb database.
int rv = NSS_Initialize(configdir, "", "", SECMOD_DB, NSS_INIT_FORCEOPEN );
hr = SECMOD_AddNewModule("test","D:\\Pkcs\\PKCS11_VSC.dll",internalMechanismFlags,internalEnableFlags);
Does anybody know how I can overcome this issue, and is there any unseen aspect to work with this utility. Also, is there a solution to install/uninstall pkcs#11 module for all user profiles with one common process. Finally, is there a better solution to do the task of installation using my setup. I am using WiX Toolset to create MSI setup
if NSS_Initialize fails to open secmod database we cannot add the module to database. the nss library automatically takes the default flags required for adding database. secmod _deletemodule deletes the module which is added to the database.