AFAIK, there are roughly two kinds of code objects in V8. One is javascript bytecode interpreted by Ignition, and the other is machine code which is compiled & optimized by Turbofan. According to the execution/frames.h, V8 constructs a different stackframe for each kind of code object. It means V8 should know the kind of the callee before it executes.
When unoptimized code(JS Bytecode) calls the optimized one, I guess that Ignition could handle the case properly to build a new stackframe for an optimized one. However, when the optimized code(machine code) calls the unoptimized one, I'm curious about how V8 determines whether callee is unoptimized or not. If the machine code is run directly on the processor, nothing can help V8 determine the kind of callee.
Also, in my understanding, V8 should detect whether some code dependency is compromised to mark or deoptimize the invalidated code objects. It also seems infeasible if V8 doesn't monitor the execution of machine code.
So my question is:
Does V8 monitor the execution of (optimized) machine code? If so, how does it happen?
If 1 is false, then how does V8 check the invalidation of code dependency or detect whether the callee is compiled or not?
(V8 developer here.)
V8 constructs a different stackframe for each kind of code object. It means V8 should know the kind of the callee before it executes.
Stack frames are set up by whoever needs them, so callers don't have to inspect callees. (They could, if they had to.)
If the machine code is run directly on the processor, nothing can help V8 determine the kind of callee.
V8 could emit machine code that checks the kind of callee. But to repeat myself: knowing the optimization status of a callee isn't necessary.
Does V8 monitor the execution of (optimized) machine code?
No, it doesn't.
how does V8 check the invalidation of code dependency
Dependencies are installed on something that can change (e.g. maps, protectors). When that change happens, registered code dependencies will be deoptimized (i.e. marked for "lazy deoptimization" when they have activations on the stack, or simply thrown away otherwise).
[how does V8] detect whether the callee is compiled or not?
It doesn't need to. When a function isn't compiled yet, its "code" will be a stub that triggers compilation.