Unable to read latest changes to the ProgramData

I have a program that executes an executable using Process.Start(). The executable I invoke is a third party program that updates a folder in ProgramData. Once the folder in the ProgramData gets updated the next set of lines in my program try to read the latest changes.

I have noticed that the latest changes cannot be read even after the executable gets executed, however when I run my program again from the beginning I can see the changes being read correctly. I am assuming this has something to do with AppDomain not able to see the changes during execution.

Is there anyway I can get things working here?

In the code below inside method HSMTransactionHandler if an exception with message HSM_ENCRYPTION_KEY_NOT_FOUND occurs, then I execute an exe by calling method UpdateFromRFS and then invoke HSMTransactionHandler recursively. The execution of the exe gets the required resource but the code doesn't read it. If I run another program during the execution of the current program, the second program reads the resource without any problem. Which makes me think if a process or app-domain can see the changes that occurs to the ProgramData folder after it has started?

Just so that everyone know I am using PKCS11Interop library which is a managed .net wrapper built around a native dll. I am also not sure if using a native dll can cause this.

Any help will be highly appreciated.

Below is the code:

public sealed class KeyStoreOperations
    {
        private KeyStoreContext m_keyStoreContext;

        private static Pkcs11 m_Pkcs11;
        private static readonly object _syncLockPkcs11 = new object();
        private static readonly object _syncLockHSMLogin = new object();

        public KeyStoreOperations(KeyStoreContext keyStoreContext)
        {
            m_keyStoreContext = keyStoreContext;
            InitializePkcs11Object();
        }

        public string Encrypt(string keyName, string message)
        {
            ValidateInputs(message, "Message");
            var encryptedMessage = string.Empty;
            HSMTransactionHandler((Session session) =>
            {
                Mechanism mechanism = new Mechanism(CKM.CKM_RSA_PKCS);
                var publicKey = GetPublicKey(keyName, session);
                if (publicKey == null)
                    throw new HSMException(ErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND);
                var originalKeyBytes = EncryptionHelper.Decode(message);
                var encryptedKeyBytes = session.Encrypt(mechanism, publicKey, originalKeyBytes);
                encryptedMessage = EncryptionHelper.Encode(encryptedKeyBytes);
            });
            return encryptedMessage;
        }

        public string Decrypt(string keyName, string cipher)
        {
            ValidateInputs(cipher, "Cipher");
            var decryptedMessage = string.Empty;
            HSMTransactionHandler((Session session) =>
            {
                Mechanism mechanism = new Mechanism(CKM.CKM_RSA_PKCS);
                var privateKey = GetPrivateKey(keyName, session);
                if (privateKey == null)
                    throw new HSMException(ErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND);
                var encryptedSymmetricKeyBytes = EncryptionHelper.Decode(cipher);
                var decryptedSymmetricKeyBytes = session.Decrypt(mechanism, privateKey, encryptedSymmetricKeyBytes);
                decryptedMessage = EncryptionHelper.Encode(decryptedSymmetricKeyBytes);
            });
            return decryptedMessage;
        }

        #region Private methods  

        #region Validations

        private void ValidateInputs(string input, string name)
        {
            if (string.IsNullOrEmpty(input))
                throw new ArgumentNullException(name);
        }

        #endregion Validations

        private void HSMTransactionHandler(Action<Session> action, bool commit = false, int retrialAttempt = 5)
        {
            Slot hsmSlot = null;
            Session hsmSession = null;
            bool logggedIn = false;
            try
            {
                hsmSlot = GetSlot(m_NCipherKeyStoreContext.ModuleToken);
                hsmSession = hsmSlot.OpenSession(false);
                lock (_syncLockHSMLogin)
                {
                    hsmSession.Login(CKU.CKU_USER, m_NCipherKeyStoreContext.SecurityPin);
                    logggedIn = true;
                    action(hsmSession);
                    hsmSession.Logout();
                    logggedIn = false;
                }
                if (commit)
                    CommitToRFS();
            }
            catch (Pkcs11Exception ex)
            {
                HandleHSMErrors(ex);
            }
            catch (HSMException ex)
            {
                if (ex.Message == EncryptionKeyStoreErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND && retrialAttempt > 0)
                {
                    if (logggedIn)
                    {
                        hsmSession.Logout();
                        logggedIn = false;
                    }
                    if (!(hsmSession == null))
                        hsmSession.CloseSession();
                    UpdateFromRFS();
                    Thread.Sleep(1000);
                    HSMTransactionHandler(action, retrialAttempt: retrialAttempt - 1);
                }
                else
                    throw ex;
            }
            finally
            {
                if (logggedIn)
                    hsmSession.Logout();
                if (!(hsmSession == null))
                    hsmSession.CloseSession();
            }
        }

        private void UpdateFromRFS()
        {
            using (var rfsSyncProcess = GetRfsSyncProcess("--update"))
            {
                ExecuteRFSSyncProcess(rfsSyncProcess);
            }
        }

        private Process GetRfsSyncProcess(string args)
        {
            Process rfsSyncProcess = new Process();
            rfsSyncProcess.StartInfo.FileName = "C:\\Program Files (x86)\\nCipher\\nfast\\bin\\rfs-sync.exe";
            rfsSyncProcess.StartInfo.Arguments = args;
            return rfsSyncProcess;
        }

        private void ExecuteRFSSyncProcess(Process rfsSyncProcess)
        {
            rfsSyncProcess.Start();
            rfsSyncProcess.WaitForExit();
        }

        private ObjectHandle GetPrivateKey(string keyName, Session session)
        {
            ObjectHandle privateKey = null;
            List<ObjectHandle> foundObjects = null;
            List<ObjectAttribute> objectAttributes = new List<ObjectAttribute>();
            objectAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
            objectAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, true));

            foundObjects = session.FindAllObjects(objectAttributes);
            if (foundObjects != null && foundObjects.Count > 0)
            {
                privateKey = foundObjects[0];
            }
            return privateKey;
        }

        private ObjectHandle GetPublicKey(string keyName, Session session)
        {
            ObjectHandle publicKey = null;
            List<ObjectHandle> foundObjects = null;
            List<ObjectAttribute> objectAttributes = new List<ObjectAttribute>();
            objectAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
            objectAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, false));

            foundObjects = session.FindAllObjects(objectAttributes);
            if (foundObjects != null && foundObjects.Count > 0)
            {
                publicKey = foundObjects[0];
            }
            return publicKey;
        }

        private List<ObjectAttribute> CreatePublicKeyTemplate(string keyName, byte[] ckaId)
        {
            List<ObjectAttribute> publicKeyAttributes = new List<ObjectAttribute>();
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_TOKEN, true));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, false));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ID, ckaId));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ENCRYPT, true));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY, true));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY_RECOVER, true));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_WRAP, true));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_MODULUS_BITS, Convert.ToUInt64(m_keyStoreContext.KeySize)));
            publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PUBLIC_EXPONENT, new byte[] { 0x01, 0x00, 0x01 }));

            return publicKeyAttributes;
        }

        private List<ObjectAttribute> CreatePrivateKeyTemplate(string keyName, byte[] ckaId)
        {
            List<ObjectAttribute> privateKeyAttributes = new List<ObjectAttribute>();
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_TOKEN, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ID, ckaId));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SENSITIVE, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_DECRYPT, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SIGN, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SIGN_RECOVER, true));
            privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_UNWRAP, true));

            return privateKeyAttributes;
        }

        private Slot GetSlot(string tokenLabel)
        {
            Slot matchingSlot = null;
            List<Slot> slots = m_Pkcs11.GetSlotList(true);
            matchingSlot = slots[0];
            if (tokenLabel != null)
            {
                matchingSlot = null;
                foreach (Slot slot in slots)
                {
                    TokenInfo tokenInfo = null;
                    try
                    {
                        tokenInfo = slot.GetTokenInfo();
                    }
                    catch (Pkcs11Exception ex)
                    {
                        if (ex.RV != CKR.CKR_TOKEN_NOT_RECOGNIZED && ex.RV != CKR.CKR_TOKEN_NOT_PRESENT)
                            throw;
                    }

                    if (tokenInfo == null)
                        continue;

                    if (!string.IsNullOrEmpty(m_keyStoreContext.ModuleToken))
                        if (0 != string.Compare(m_keyStoreContext.ModuleToken, tokenInfo.Label, StringComparison.Ordinal))
                            continue;

                    matchingSlot = slot;
                    break;
                }

                if (matchingSlot == null)
                    throw new HSMException(string.Format(ErrorConstant.HSM_CONFIGURATION_ERROR_INCORRECT_SLOT, tokenLabel));
            }
            return matchingSlot;
        }

        private void InitializePkcs11Object()
        {
            if (m_Pkcs11 == null)
            {
                lock (_syncLockPkcs11)
                {
                    m_Pkcs11 = new Pkcs11(m_keyStoreContext.PKCS11LibraryPath, true);
                }
            }
        }

        private void HandleHSMErrors(Pkcs11Exception ex)
        {
            if (ex.RV == CKR.CKR_PIN_INCORRECT)
            {
                throw new HSMException(ErrorConstant.HSM_CONFIGURATION_ERROR_PIN_INCORRECT, ex);
            }
            else
            {
                throw new HSMException(ErrorConstant.HSM_CONFIGURATION_ERROR_GENERIC, ex);
            }
        }

        #endregion
    }

Edit 1: Here is the modified code that worked for me and please note that the most important thing here is to set variable CKNFAST_ASSUME_SINGLE_PROCESS to 0 in cknfastrc file

public sealed class KeyStoreOperations
        {
            private KeyStoreContext m_keyStoreContext;

            private static Pkcs11 m_Pkcs11;
            private static readonly object _syncLockPkcs11 = new object();
            private static readonly object _syncLockHSMLogin = new object();

            public KeyStoreOperations(KeyStoreContext keyStoreContext)
            {
                m_keyStoreContext = keyStoreContext;
                InitializePkcs11Object();
            }

            public string Encrypt(string keyName, string message)
            {
                ValidateInputs(message, "Message");
                var encryptedMessage = string.Empty;
                HSMTransactionHandler((Session session) =>
                {
                    Mechanism mechanism = new Mechanism(CKM.CKM_RSA_PKCS);
                    var publicKey = GetPublicKey(keyName, session);
                    if (publicKey == null)
                        throw new HSMException(ErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND);
                    var originalKeyBytes = EncryptionHelper.Decode(message);
                    var encryptedKeyBytes = session.Encrypt(mechanism, publicKey, originalKeyBytes);
                    encryptedMessage = EncryptionHelper.Encode(encryptedKeyBytes);
                });
                return encryptedMessage;
            }

            public string Decrypt(string keyName, string cipher)
            {
                ValidateInputs(cipher, "Cipher");
                var decryptedMessage = string.Empty;
                HSMTransactionHandler((Session session) =>
                {
                    Mechanism mechanism = new Mechanism(CKM.CKM_RSA_PKCS);
                    var privateKey = GetPrivateKey(keyName, session);
                    if (privateKey == null)
                        throw new HSMException(ErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND);
                    var encryptedSymmetricKeyBytes = EncryptionHelper.Decode(cipher);
                    var decryptedSymmetricKeyBytes = session.Decrypt(mechanism, privateKey, encryptedSymmetricKeyBytes);
                    decryptedMessage = EncryptionHelper.Encode(decryptedSymmetricKeyBytes);
                });
                return decryptedMessage;
            }

            #region Private methods  

            #region Validations

            private void ValidateInputs(string input, string name)
            {
                if (string.IsNullOrEmpty(input))
                    throw new ArgumentNullException(name);
            }

            #endregion Validations

            private void HSMTransactionHandler(Action<Session> action, bool commit = false, int retrialAttempt = 5)
            {
                Slot hsmSlot = null;
                Session hsmSession = null;
                bool logggedIn = false;
                try
                {
                    hsmSlot = GetSlot(m_NCipherKeyStoreContext.ModuleToken);
                    hsmSession = hsmSlot.OpenSession(false);
                    lock (_syncLockHSMLogin)
                    {
                        hsmSession.Login(CKU.CKU_USER, m_NCipherKeyStoreContext.SecurityPin);
                        logggedIn = true;
                        action(hsmSession);
                        hsmSession.Logout();
                        logggedIn = false;
                    }
                    if (commit)
                        CommitToRFS();
                }
                catch (Pkcs11Exception ex)
                {
                    HandleHSMErrors(ex);
                }
                catch (HSMException ex)
                {
                    if (ex.Message == EncryptionKeyStoreErrorConstant.HSM_ENCRYPTION_KEY_NOT_FOUND && retrialAttempt > 0)
                    {
                        if (logggedIn)
                        {
                            hsmSession.Logout();
                            logggedIn = false;
                        }
                        if (!(hsmSession == null))
                        {
                            hsmSession.CloseSession();
                            hsmSession = null;
                        }
                        UpdateFromRFS();
                        Thread.Sleep(1000);
                        if (!m_Pkcs11.Disposed)
                        {
                            m_Pkcs11.Dispose();
                            m_Pkcs11 = null;
                        }
                        HSMTransactionHandler(action, retrialAttempt: retrialAttempt - 1);
                    }
                    else
                        throw ex;
                }
                finally
                {
                    if (logggedIn)
                        hsmSession.Logout();
                    if (!(hsmSession == null))
                        hsmSession.CloseSession();
                }
            }

            private void UpdateFromRFS()
            {
                using (var rfsSyncProcess = GetRfsSyncProcess("--update"))
                {
                    ExecuteRFSSyncProcess(rfsSyncProcess);
                }
            }

            private Process GetRfsSyncProcess(string args)
            {
                Process rfsSyncProcess = new Process();
                rfsSyncProcess.StartInfo.FileName = "C:\\Program Files (x86)\\nCipher\\nfast\\bin\\rfs-sync.exe";
                rfsSyncProcess.StartInfo.Arguments = args;
                return rfsSyncProcess;
            }

            private void ExecuteRFSSyncProcess(Process rfsSyncProcess)
            {
                rfsSyncProcess.Start();
                rfsSyncProcess.WaitForExit();
            }

            private ObjectHandle GetPrivateKey(string keyName, Session session)
            {
                ObjectHandle privateKey = null;
                List<ObjectHandle> foundObjects = null;
                List<ObjectAttribute> objectAttributes = new List<ObjectAttribute>();
                objectAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
                objectAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, true));

                foundObjects = session.FindAllObjects(objectAttributes);
                if (foundObjects != null && foundObjects.Count > 0)
                {
                    privateKey = foundObjects[0];
                }
                return privateKey;
            }

            private ObjectHandle GetPublicKey(string keyName, Session session)
            {
                ObjectHandle publicKey = null;
                List<ObjectHandle> foundObjects = null;
                List<ObjectAttribute> objectAttributes = new List<ObjectAttribute>();
                objectAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
                objectAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, false));

                foundObjects = session.FindAllObjects(objectAttributes);
                if (foundObjects != null && foundObjects.Count > 0)
                {
                    publicKey = foundObjects[0];
                }
                return publicKey;
            }

            private List<ObjectAttribute> CreatePublicKeyTemplate(string keyName, byte[] ckaId)
            {
                List<ObjectAttribute> publicKeyAttributes = new List<ObjectAttribute>();
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_TOKEN, true));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, false));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ID, ckaId));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ENCRYPT, true));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY, true));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_VERIFY_RECOVER, true));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_WRAP, true));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_MODULUS_BITS, Convert.ToUInt64(m_keyStoreContext.KeySize)));
                publicKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PUBLIC_EXPONENT, new byte[] { 0x01, 0x00, 0x01 }));

                return publicKeyAttributes;
            }

            private List<ObjectAttribute> CreatePrivateKeyTemplate(string keyName, byte[] ckaId)
            {
                List<ObjectAttribute> privateKeyAttributes = new List<ObjectAttribute>();
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_TOKEN, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_PRIVATE, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_LABEL, keyName));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_ID, ckaId));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SENSITIVE, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_DECRYPT, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SIGN, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_SIGN_RECOVER, true));
                privateKeyAttributes.Add(new ObjectAttribute(CKA.CKA_UNWRAP, true));

                return privateKeyAttributes;
            }

            private Slot GetSlot(string tokenLabel)
            {
                Slot matchingSlot = null;
                List<Slot> slots = m_Pkcs11.GetSlotList(true);
                matchingSlot = slots[0];
                if (tokenLabel != null)
                {
                    matchingSlot = null;
                    foreach (Slot slot in slots)
                    {
                        TokenInfo tokenInfo = null;
                        try
                        {
                            tokenInfo = slot.GetTokenInfo();
                        }
                        catch (Pkcs11Exception ex)
                        {
                            if (ex.RV != CKR.CKR_TOKEN_NOT_RECOGNIZED && ex.RV != CKR.CKR_TOKEN_NOT_PRESENT)
                                throw;
                        }

                        if (tokenInfo == null)
                            continue;

                        if (!string.IsNullOrEmpty(m_keyStoreContext.ModuleToken))
                            if (0 != string.Compare(m_keyStoreContext.ModuleToken, tokenInfo.Label, StringComparison.Ordinal))
                                continue;

                        matchingSlot = slot;
                        break;
                    }

                    if (matchingSlot == null)
                        throw new HSMException(string.Format(ErrorConstant.HSM_CONFIGURATION_ERROR_INCORRECT_SLOT, tokenLabel));
                }
                return matchingSlot;
            }

            private void InitializePkcs11Object()
            {
                if (m_Pkcs11 == null)
                {
                    lock (_syncLockPkcs11)
                    {
                        m_Pkcs11 = new Pkcs11(m_keyStoreContext.PKCS11LibraryPath, true);
                    }
                }
            }

            private void HandleHSMErrors(Pkcs11Exception ex)
            {
                if (ex.RV == CKR.CKR_PIN_INCORRECT)
                {
                    throw new HSMException(ErrorConstant.HSM_CONFIGURATION_ERROR_PIN_INCORRECT, ex);
                }
                else
                {
                    throw new HSMException(ErrorConstant.HSM_CONFIGURATION_ERROR_GENERIC, ex);
                }
            }

            #endregion
        }

Edit 2: I checked and found that its working without even setting CKNFAST_ASSUME_SINGLE_PROCESS to 0, so may all that is needed is to dispose the pkcs11 object and re-initialize it

Solution

Based on your previous questions #1, #2 and #3 I am guessing (because you did not write it) that you are executing rfs-sync.exe and your PKCS#11 library still does not see freshly synced keys. If that is the case, then you need to consult HSM user guide and find variable (similar to CKNFAST_FAKE_ACCELERATOR_LOGIN) which makes your PKCS#11 library reread local FS each time you perform searching operation. Without that variable PKCS#11 library just caches the contents of local FS during its initialization.