The layoutopt.exe in the SDK/tools folder for Android.
Where can I find the rules it works with etc?
The source code location would help.
The source.android.com website is a good place to start.
There is a section for the SDK - https://android.googlesource.com/platform/sdk
And you can then drill down into the layoutopt folder: