Windows and Unix have their own calling-conventions for x86-64. But, if a language requires / benefits from it, can it have its own calling conventions for internal use? For eg, Swift / Python(compiled) may get benefitted from having multiple registers for returning multiple results from a function (since those languages support that). So reserving, say 3 registers (rax, rcx, rdx), for return values in their language-calling-convention, does it break anything? causing any unnoticed bugs, or causing undefined behavior in any case?
Also, if it is not necessary to follow that convention, then why do they have defined calling-conventions for user space?
Of course, they can. The residual problem will be the interaction with the API/the OS where you have to abide the OS way of doing things.
Overall, the main point will probably be the cost/benefit relation.
But for special purposes, this is possible and may even be superior (why else would you even want to do it?).
Also, take into account possible side effects on OS-specific things like the red-zone.