How do i find the license key for this program?

I'm trying to solve this exercise for university. We have to "crack" a program, which is missing a license file that is required to start the program. We are only working with a Linux shell.

So what I've already done is creating this missing license file. It is an .ini that includes a license-holder and a license key. The only thing i'm looking for now is the correct license key. The task says we should use "strace" and "ltrace" to solve this problem. This is what i get as an output:

fopen("license.ini", "r")                                   =0x55c088307380

fgets("LicenseHolder=annabell.krause@ex"..., 4096, 0x55c088307380)  = 0x7ffe72450860

strncmp("LicenseKey=", "LicenseHolder=annabell.krause@ex"..., 11)   = 3

strncmp("LicenseHolder=", "LicenseHolder=annabell.krause@ex"..., 14)    = 0

sscanf(0x7ffe72450860, 0x55c08753c16b, 0x7ffe72450800, 0xffffc000)  = 1

fgets("LicenseKey=aoeklycf", 4096, 0x55c088307380) = 0x7ffe72450860

strncmp("LicenseKey=", "LicenseKey=aoeklycf", 11)           = 0

sscanf(0x7ffe72450860, 0x55c08753c121, 0x7ffe72450840, 0xfffff800)  = 1

fgets("LicenseKey=aoeklycf", 4096, 0x55c088307380)          = 0

memfrob(0x7ffe72450840, 8, 0, 0xfbad2498)                   = 0x7ffe72450840

strncmp("KEOAFSIL", "aoeklycf", 8)                          = -22

fwrite("ERROR: License key is invalid.\n", 1, 31, 0x7faeabe60680
ERROR: License key is invalid.
)                                   = 31

+++ exited (status 1) +++

So I guess the answer lies somewhere within the memfrob and strncmp function at the end. But i don't know what's the next step.

Solution

Let's look at the library trace call by call. The important part is in step 5.

Analysis

Open the file

fopen("license.ini", "r")                                   =0x55c088307380

Opens the license file.

Parse the license holder
```
fgets("LicenseHolder=annabell.krause@ex"..., 4096, 0x55c088307380)  = 0x7ffe72450860
```
Reads a line from the file: LicenseHolder=annabell.krause@ex….
```
strncmp("LicenseKey=", "LicenseHolder=annabell.krause@ex"..., 11)   = 3
```
Does the line start with LicenseKey=? The return value of 3 means no, it does not.
```
strncmp("LicenseHolder=", "LicenseHolder=annabell.krause@ex"..., 14)    = 0
```
Does the line start with LicenseHolder=? Yes, it does.
```
sscanf(0x7ffe72450860, 0x55c08753c16b, 0x7ffe72450800, 0xffffc000)  = 1
```
Unfortunately, ltrace has not dereferenced any of the addresses to show us the contents. We know that 0x7ffe72450860 is the current line, so it's presumably pulling out the e-mail address from the current line.
Parse the license key
```
fgets("LicenseKey=aoeklycf", 4096, 0x55c088307380) = 0x7ffe72450860
```
It reads another line: LicenseKey=aoeklycf.
```
strncmp("LicenseKey=", "LicenseKey=aoeklycf", 11)           = 0
```
Does the line start with LicenseKey=? Yes, it does.
```
sscanf(0x7ffe72450860, 0x55c08753c121, 0x7ffe72450840, 0xfffff800)  = 1
```
It's parsing the current line. Presumably, it's extracting the license key you entered, aoeklycf, and saving it in a variable for later comparison against the expected license key. Something like sscanf(line, "LicenseKey=%s", licenseKey);.
End-of-file
```
fgets("LicenseKey=aoeklycf", 4096, 0x55c088307380)          = 0
```
It tries to read another line and hits EOF. Ignore the first argument, it's just showing what was left in the buffer from the last call.
License key comparison
```
memfrob(0x7ffe72450840, 8, 0, 0xfbad2498)                   = 0x7ffe72450840
```
"Encrypts" 8 bytes of some memory area by XORing each byte with 42. This can be reversed by running memfrob() again. I put "encrypts" in air quotes because this can barely be called encryption. It's just a little bit of obfuscation.

Notice that 0x7ffe72450840 is the address from the sscanf() above. It's frobbing the variable I called licenseKey above, the LicenseKey= string it extracted from the input file.
```
strncmp("KEOAFSIL", "aoeklycf", 8)                          = -22
```
This is the money line. It compares actual and expected values and fails.

Error message

fwrite("ERROR: License key is invalid.\n", 1, 31, 0x7faeabe60680)                                   = 31

An error is printed.

Synthesis

But the author doesn't want you to be able to run a simple string search like strings ./program to pull the license key out of the executable. To prevent that you have to enter the frobbed version of the license key in license.ini, not the raw string strings finds.

The code might look something like:

char *expected = "aoeklycf";
char actual[BUFSIZE];

sscanf(line, "LicenseKey=%s", actual);
memfrob(actual);
if (strncmp(actual, expected, strlen(expected)) != 0) {
    error("ERROR: License key is invalid.\n");
}

Did you extract aoeklycf from the program? If so, you missed the memfrob() step. license.ini needs to list the "encrypted" version of the license key: KEOAFSIL.