I wonder how are anti-cheats checking if the client actually runs the anti-cheat? I mean, like why can't just someone reverse engineer csgo, and change the code of vac so that it doesn't detect hacks? What does vac send to server to make sure it's running? Like if it sends tcp messages to the server, can't someone just create fake vac that sends these tcp messages? How can I make sure that client uses my anti-cheat?
Yes, the client and the server are in constant communication. If something on the server is expecting something from VAC on the client and it doesn't receive it, that would indicate that VAC on the client has been compromised.
Yes, people can just disable the anticheat and then emulate the network traffic that is required.
If you want this functionality you would need to implement it. It's typically referred to as a "heartbeat" packet in it's simplest form. The verification of the heartbeat packets needs to be done server side and client side and it all needs to be heavily obfuscated and embedded within the functionality of the anticheat itself. It can't only be called once, there needs to be constant verification across many parts of the anticheat, in order that it will not be easy to emulate or remove.
If you're working on making an anticheat, you need to be smarter than 99% of cheat developers or else you will not have much benefit.