I cannot find the MSI format specification. It says that its an open standard, but I don't see any documentation - just bits and pieces of information here and there.
I want to modify some .msi installer, but I want to understand first what it is doing.
Sounds like you got a misunderstanding of Microsoft's promise.
Open Specification Promise
Published: September 12, 2006. Revised: February 15, 2007
Microsoft irrevocably promises not to assert any Microsoft Necessary Claims against you for making, using, selling, offering for sale, importing or distributing any implementation to the extent it conforms to a Covered Specification (“Covered Implementation”), subject to the following. This is a personal promise directly from Microsoft to you, and you acknowledge as a condition of benefiting from it that no Microsoft rights are received from suppliers, distributors, or otherwise in connection with this promise. If you file, maintain or voluntarily participate in a patent infringement lawsuit against a Microsoft implementation of such Covered Specification, then this personal promise does not apply with respect to any Covered Implementation of the same Covered Specification made or used by you. To clarify, “Microsoft Necessary Claims” are those claims of Microsoft-owned or Microsoft-controlled patents that are necessary to implement only the required portions of the Covered Specification that are described in detail and not merely referenced in such Specification. “Covered Specifications” are listed below.
This promise is not an assurance either (i) that any of Microsoft issued patent claims covers a Covered Implementation or are enforceable or (ii) that a Covered Implementation would not infringe patents or other intellectual property rights of any third party. No other rights except those expressly stated in this promise shall be deemed granted, waived or received by implication, exhaustion, estoppel, or otherwise.
The full list of the open specifications can be found at
http://www.microsoft.com/openspecifications/en/us/programs/osp/default.aspx
Since MSI is not listed within, you can safely assume that it is a private standard and won't be shared to general public.