Suppose we have 3 methods: method 2 is called from method 1, method 3 is called from method 2. Methods 2 and 3 are of size 30 bytecodes each. Also, suppose for definiteness method 2 is always called from method 1 exactly once, and method 3 is always called from method 2 exaclty once.
If method 2 is inlined first, method 3 will be called from the body of method 1 directly, and could be inlined in its turn. If method 3 is inlined into method 2 first, the size of the latter will become about 60 bytecodes, and it couldn't be inlined, because default MaxInlineSize threshold is 35 bytecodes.
In which order HotSpot JIT inlines methods: top-down or down-top?
The MaxInlineSize affects compilations of methods executed at least one time but less than MinInliningThreshold times only. For methods executed more than MinInliningThreshold there is a different setting -XX:FreqInlineSize=… having a far bigger (platform dependent) default value. Hotspots are still inlined regardless of the MaxInlineSize. You may test it by running an application with -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining -XX:MaxInlineSize=0. It will still report inlining of hot spots (these with the comment “(hot)”). Only methods formerly reported as inlined with the comment “executed < MinInliningThreshold times” might then get the comment to “too big”. If you set down the FreqInlineSize you might receive comments like “hot method too big”. I never saw them with the default setting.