If there any way to know if a client expects server to enable SSL?
I am building a small SMTP server and have implemented SSL on 465 but some clients do not expect SSL so obviously connection fails.
Is it possible to tell this in any way?
There is no clean way for a server to detect if a client expects to use SSL/TLS at the start of the connection. In fact, if the server is expected to send data first (as is the case with SMTP: the server sends a banner before the client sends any data), there is no way at all to do that.
This is the reasons why SSL/TLS is generally used in one of these two ways:
STARTTLS is a bit less efficient because of the non-SSL/TLS conversation between the server and client that happens first (uses several network round trips) and it is not available for use with all protocols (HTTP doesn't support it), but it's generally preferred if available because it makes it easier for things like automatic configuration of email settings (no need to probe a bunch of possible ports and pick the best one).
Port 465 is an example of the first solution: pick a new port and run SSL/TLS on it. That means servers and clients are both supposed to use SSL/TLS right away for communications on that port.
If you are seeing clients trying to talk plaintext SMTP on port 465, those clients are BROKEN. There really isn't anything you can do to work around them. The clients have serious bugs which should be fixed...
Moreover, for SMTP, you really need to be using STARTTLS, not SMTP over SSL/TLS on port 465.