I have a requirement where I intend to build a tool to scan the email contents including the attachments. The email servers is either going to be SendMail or z/OS Communication Server, both support SMTP. The sever is not Miscrosoft implementation so MAPI or Outlook API is not there into picture. The tool would be Java based code and basically need to look for contents that are not-permitted based on some rules. What are my options here? There is the possibility of using a proxy server but we are looking for a more direct approach.
The z/OS Communication Server SMTP implementation has a built-in "exit" capability - see http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/F1A1B4B0/30.3?DT=20110609204120#HDRWQ1299.
The exit is called for just about any SMTP activity and it can examine, change or reject just about anything based on the rules you establish. It is generally written in IBM Assembler Language, but there's no reason you couldn't have a thin assembler layer that passes data to a Java app using whatever protocol you like (say, a pipe or a socket).
There are many little details to handle, such as character encoding (EBCDIC vs. ASCII or UTF-8, for example) plus weeding out attachments from email content. But using the exit preserves all the z/OS specific features of IBM's SMPT server without trying to recreate any of that yourself.
Good luck!