I'm using a JDBCRealm with JAAS Context=jdbcDigestRealm in Payara to do http digest auth. If I set it up to use clear text passwords in my user database, everything works as expected. But I would like to store the passwords as MD5(username:realm:password) in the database, much like Apache httpd does. Unfortunately cannot find settings in the JDBCRealm to handle that. Storing clear text passwords is of course not desirable.
How should I configure the settings of the JDBCRealm in Payara to allow using passwords stored as MD5(username:realm:password) ?
This is my working setup with clear text passwords:
<auth-realm classname="com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm" name="digestrealm">
<property name="jaas-context" value="jdbcDigestRealm"></property>
<property name="datasource-jndi" value="jdbc/userrepo"></property>
<property name="user-table" value="usertable"></property>
<property name="user-name-column" value="username"></property>
<property name="password-column" value="password"></property>
<property name="group-table" value="grouptable"></property>
<property name="group-name-column" value="groupname"></property>
<property name="charset" value="UTF-8"></property>
<property name="digest-algorithm" value="None"></property>
</auth-realm>
I found the answer myself after browsing through the Payara source code a while. By setting the property "Encoding" to "Hashed" on the JDBC realm, passwords in the database are assumed to be in format MD5(username:realm:password).
This feature seems to be entirely undocumented.
<auth-realm classname="com.sun.enterprise.security.auth.realm.jdbc.JDBCRealm" name="digestrealm">
<property name="jaas-context" value="jdbcDigestRealm"></property>
<property name="datasource-jndi" value="jdbc/userrepo"></property>
<property name="user-table" value="usertable"></property>
<property name="user-name-column" value="username"></property>
<property name="password-column" value="password"></property>
<property name="group-table" value="grouptable"></property>
<property name="group-name-column" value="groupname"></property>
<property name="charset" value="UTF-8"></property>
<property name="digest-algorithm" value="None"></property>
<property name="encoding" value="HASHED"></property>
</auth-realm>