From C++ Concurrency in Action:
difference between std::atomic and std::atomic_flag is that std::atomic may not be lock-free; the implementation may have to acquire a mutex internally in order to ensure the atomicity of the operations
I wonder why. If atomic_flag is guaranteed to be lock-free, why isn't it guaranteed for atomic<bool> as well?
Is this because of the member function compare_exchange_weak? I know that some machines lack a single compare-and-exchange instruction, is that the reason?
First of all, you are perfectly allowed to have something like std::atomic<very_nontrivial_large_structure>, so std::atomic as such cannot generally be guaranteed to be lock-free (although most specializations for trivial types like bool or int probably could, on most systems). But that is somewhat unrelated.
The exact reasoning why atomic_flag and nothing else must be lock-free is given in the Note in N2427/29.3:
Hence the operations must be address-free. No other type requires lock-free operations, and hence the atomic_flag type is the minimum hardware-implemented type needed to conform to this standard. The remaining types can be emulated with atomic_flag, though with less than ideal properties.
In other words, it's the minimum thing that must be guaranteed on every platform, so it's possible to implement the standard correctly.