Problem with Steam RSA Login implementation in Swift 5.0

Question

So when logging in to steam, I need my password encrypted with RSA public key they provided. But when I encrypt with SecKeyEncryptedData and sent it for authentication, it says my password is incorrect. I think maybe it's a problem with the format of encryption but I cannot figure it out. Please help me with that.

static func encrypt(string: String, mod: String, exp: String) -> String? {
        
        let keyString = self.rsaPublicKeyder(mod: mod, exp: exp)
        guard let data = Data(base64Encoded: keyString) else { return nil }
        
        var attributes: CFDictionary {
            return [kSecAttrKeyType         : kSecAttrKeyTypeRSA,
                    kSecAttrKeyClass        : kSecAttrKeyClassPublic,
                    kSecAttrKeySizeInBits   : 2048,
                    kSecReturnPersistentRef : kCFBooleanTrue!] as CFDictionary
        }
        
        var error: Unmanaged<CFError>? = nil
        guard let secKey = SecKeyCreateWithData(data as CFData, attributes, &error) else {
            print(error.debugDescription)
            return nil
        }
        guard let result = SecKeyCreateEncryptedData(secKey, SecKeyAlgorithm.rsaEncryptionPKCS1, string.data(using: .utf8)! as CFData, &error) else{
            print(error.debugDescription)
            return nil
        }
        return (result as Data).base64EncodedString()
    }
}

The self.rsaPublicKeyder function's role is to convert the mod and exp format to PKCS#8 DER format and it can import with SecKeyCreateWithData. And I tried the result Der in cyberchef which seems no problem.

I tried using BigInt library to encrypt myself, but still failed. How????

class RSA{
    static func encrypt(string: String, mod: String, exp: String) -> String
    {
        let secret = pkcs1pad2(data: string.data(using: .utf8)!, keySize: mod.count / 2)!
        return secret.power(BigUInt(exp, radix: 16)!, modulus: BigUInt(mod, radix: 16)!).serialize().base64EncodedString()
    }
    
    static func pkcs1pad2(data: Data, keySize: Int) -> BigUInt?{
        if (keySize < data.count + 11){
            return nil;
        }
        var rndData: [UInt8] = [UInt8](repeating: 0, count: keySize - 3 - data.count)
        let status = SecRandomCopyBytes(kSecRandomDefault, rndData.count, &rndData)
        for i in 0..<rndData.count{
            if rndData[i] == 0{
                rndData[i] = UInt8(i+1)
            }
        }
        guard status == errSecSuccess else{
            return nil
        }
        
        return BigUInt(Data([0x00, 0x02]) + Data(rndData) + Data([0x00]) + data)
    }
}

I have cost a bunch of time on this problem and I still don't know which part of my code is doing wrong. I have put all codes on Github. If you can help me with it I'll be so grateful. https://github.com/MTAwsl/iAuth/tree/dev

Answer 1

I tried to debug myself using Fiddler and I found the answer. Firstly, when a base64 string is transferred through HTTP GET method, it needs to be encoded by "%" encoding. So, String.addingPercentEncoding should be called to encode the string with proper encoding. But in CharacterSet.urlHostAllowed set, it did not include the "+" character so when the server decoding the data from base64, it treats "+" as space, which is definitely not what we wanted. I add the extension to String module and it solved the problem. Besides, yeah, both the BigInt and SecKey method works. There's nothing about the RSA encryption.

extension String{
    
    var encoded: String? {
        var urlB64Encoded: CharacterSet = .urlHostAllowed
        urlB64Encoded.remove(charactersIn: "+")
        return self.addingPercentEncoding(withAllowedCharacters: urlB64Encoded)
    }
}