I have program that is reading and writing mat files using the Matlab API. Therefore my only dependency ist on libmat.dll
Can i ship libmat.dll only (instead of requiring ~600mb MCR) or is that prohibited by a license?
As long as you write MAT files out your code, you can...
Here are the terms from Matlab's license.txt (located in Matlab installation folder) under which you can deploy and share your libmat.dll linked application.
Excerpt from "Deployment Addendum" license section:
OTHER SOURCE CODE AND SHARED OBJECT CODE LIBRARIES.
6.1. Programs may include selected Source Code and shared Object Code library files that implement various documented application programming interface capabilities of the Programs for which the Source Code or shared library file is part.
6.1.1. Licensed User may use and modify the selected Source Code files solely for creation of Licensee's own Applications. Licensee may copy and distribute Object Code compiled from this Source Code, but only as either standalone Object Code file (regardless whether a Derivative Form) or Object Code Linked to the Application, and only for use with and deployment of Licensee's own Application.
6.1.2. Licensee may use, copy and distribute shared Object Code library files (regardless whether a Derivative Form) for deployment of Licensee's own Application, but only if a header file exists in the Program for the shared library file. Licensee may not copy or distribute header files themselves.
6.1.3. Licensee may not transfer Source Code, development rights, or development capabilities for any Source Code or Object Code to any Third Party.
6.2. MAT-files. Licensed User may create and distribute Applications that read MAT-files using the MAT-file API, however, if such Applications are distributed to Third Parties, they must also implement MAT-file write capability.
Be aware that libmat.dll has its own further MATLAB library dependencies - so you'll need to ship also other Matlab DLLs in order to use libmat.dll.