It is stated here:
A terminator canary contains NULL(0x00), CR (0x0d), LF (0x0a) and EOF (0xff) -- four characters that should terminate most string operations, rendering the overflow attempt harmless.
I know the null (0x00) can help prevent strcpy, strncpy, stpcpy, and strcat. Also LF (0x0A) can work for gets and fgets.
What are 0x0D and 0xFF good for stopping?
The purpose of the "canary" is to detect when a buffer has been overrun. It's placed just before the return address of the stack, after any buffers allocated in the current stack frame. If its value changes then the stack checking code know buffer has been overrun and aborts the program before it can do any damage.
The problem with this is that if the attacker overwrites the canary with the same value as it had before then buffer overrun won't be detected. To make this more difficult either a random number is used as the canary, so the attacker can't predict it, or the special "terminator canary" value you're asking about is used. The byte values that make up the terminator canary were chosen because they will terminate various copy operation used by programs. If the these values are in the string ("shellcode") the attacker uses to try overwrite the return value with then most code that would copy the string will stop before overwriting the return value.
Here are examples of memory copying operations that would be terminated before over writing the return value if the terminate canary appears in the source input:
char buf[10];
strcpy(buf, src);
The example above of the most common case. Any operation copying a C string with stop a the first NUL (0) byte in the source string.
char buf[10];
gets(buf);
Using gets is common beginner mistake and doesn't normally appear in production code, but it's not hard to write more sophisticated code that reads a line but doesn't take care to not overflow the buffer. What marks the end-of-line depends on the convention. The usual Unix convention is to use a single line feed character (LF, 0x0A), but Windows uses the carriage return and line feed (CR LF, 0x0D 0x0A) sequence. Since the terminator canary contains both the CR and LF byte values, both EOL terminators are present in the canary. Any operation copying a single line will stop before overwriting the return value.
char buf[10];
char *dest = buf;
while(1) {
char c = getchar();
if (c == EOF) {
break;
}
*dest++ = c;
}
How the EOF terminator works here is harder to see and explain than in other two examples. In addition to the buffer overflow bug, this code contains another bug that leads to 0xFF being interpreted as EOF. Like using gets, this is also a rookie mistake, but one that's more common in production code. The bug is caused by using char c instead int c.
The value returned by getchar is actually an int rather than char. This makes it possible to distinguish valid byte values from special EOF return value, which on most systems is -1. Byte value read from the file are returned as unsigned char values cast to int. So the byte value '\xFF' in the file is returned as the int value 255. When this is assigned to char variable c its truncated from (on most systems these days) a 32-bit signed integer value to an 8-bit signed integer value. This turns 32-bit signed integer value 255 into the 8-bit signed integer value -1. This conversion also turns the 32-bit signed integer value -1 (EOF) into the 8-bit signed integer value -1.
Since both '\xFF' and EOF end up being converted to -1, they both end up comparing as equal to EOF. This means that in the example code above when either value is returned by getchar the loop will terminate. Any code that makes this sort of mistake will stop copying at the first '\xFF' byte in the source input. However code that uses getchar, getc, fgetc or a similar function correctly, assigning the return value to an int, will continue to copy past any '\xFF' bytes.
The terminator canary makes it so an attacker can't exploit a given buffer overrun in code, if the overrunning code uses a NUL (0), EOL (0x0D and/or 0x0A) or a broken EOF (0xFF) comparison to terminate the copy. My guess is that most case of buffer overruns are made unexploitable by NUL being in the terminator canary. Many of the remainder would be protected by the EOL characters, while the broken EOF byte probably doesn't have much applicability at all.