Is it possible to create an alternative calling convention without modifying the compiler itself? This hypothetical calling convention could , for instance, be used for optimization
gcc / clang have their knowledge of calling conventions compiled in to the compiler executable, not defined by a config file you could override without recompiling the compiler.
So yes you'd need to modify the compiler, but it might not be hard for the open-source compilers. The gcc-internals documentation has a whole chapter on stack layout and calling conventions, documenting how calling conventions are described to the compiler. It looks like this is done in C source code, not in a special language the way machine-definition files describe the instruction-set to the compiler.
I didn't check how clang / LLVM defines calling conventions.
But there is some limited flexibility in some cases with GNU C (gcc/clang/ICC). It's not great, but it's not totally fixed.
On x86, the compilers know about multiple existing calling conventions. e.g. x86-64 System V and Windows have different calling conventions, and you can use __attribute__((ms_abi)) to declare a function that uses the Windows convention, even when compiled for a non-Windows target platform. When compiling calls to functions, gcc of course generates code to pass args and set up the stack according to the calling convention declared for the function.
For 32-bit x86, there are many conventions. See the gcc docs for function attributes.
e.g. you can use __attribute__((fastcall)) void foo(int a, int b) { ... } to declare a function that expects it args in ECX and EDX, instead of on the stack.
There's even support (on 32-bit x86) for setting the max number of register arguments for a calling convention, with __attribute((regparm(3))) to declare a function that takes up to 3 integer args in registers instead of the stack. But the sequence of registers is not configurable, and it only uses EAX, ECX, and EDX (the registers that are call-clobbered in most conventions).
You can also compile with -mregparm=2 to set this globally.
But there's no support for declaring functions to be called with shadow space (like x64 Windows), or with caller-pops (except for some specific caller-pops conventions), or with boolean results returned in FLAGS instead of an integer register, or any of the other stuff you can do with hand-written assembly.
MSVC similarly supports some __declspec options for vectorcall vs. regular.