atomic_compare_exchange_strong_explicit(mem, old, new, <mem_order>, <mem_order>);
ftruncate(fd, <size>);
All I want is that these two lines of code always occur without any interference (WITHOUT USING LOCKS). Immediately after that CAS, ftruncate(2) should be called. I read a small description about memory orders, although I don’t understand them much. But they seemed to make this possible. Is there any way around?
Your title asks for the things to occur in order. That's easy, and C basically does that automatically with mo_seq_cst; all visible side-effects of CAS will appear before any from ftruncate.
(Not strictly required by the ISO C standard, but in practice real implementations implement seq-cst with a full barrier, except AArch64 where STLR doesn't stall to drain the store buffer unless/until there's a LDAR while the seq-cst store is still in the store buffer. But a system call is definitely going to also include a full barrier.)
Within the thread doing the operation, the atomic is Sequenced Before the system call.
What kind of interference are you worried about? Some other thread changing the size of the file? You can't prevent that race condition.
There's no way to combine some operation on memory + a system call into a single atomic transaction. You would need to use a hypothetical system call that atomically does what you want. (Presumably it would have to do locking inside the kernel to make a file operation and a memory modification appear as one atomic transaction.) e.g. the Linux futex system call atomically does a couple things, but of course there's nothing like this for any other operations.
Or you need locking. (Or to suspend all other threads of your process somehow.)