After digging into the code a bit I see that invalidate() on a control will increase a counter which seems to mark the control as invalidated.
This seems to lead to a rerender.
So if you have a control that you want rerendered, is it better practice to use invalidate() or rerender()?
How does a rerender actually get triggered? (other than by explicitly invoking it of course)
Both are marked as "protected", meaning you should not call any of them unless you are really deep into developing custom controls.
A control gets invalidated when (for example) a property is changed. In that case you usually want to re-render the control, that's why setters call invalidate by default.
When you overwrite your invalidate method in your custom control, you can analyze the source of the invalidation and then decide whether you really want to rerender etc. See for example the unified.Shell which decides what to do on invalidate based on the source: https://sapui5.netweaver.ondemand.com/sdk/resources/sap/ui/unified/Shell-dbg.js line 1539ff
Again: Protected means that it should not be called from the outside (and that it should not be necessary, except for debugging).