I can deliver normal UDP messages between two computers using a direct ethernet connection using my software. But I couldn't do the same for multicasting messages, I've tried other softwares that can send and receive multicast UDP messages and they didn't work as well. So I thought and wanted to ask if a direct Ethernet connection is a proper pyhsical setup or not. And if not what should I do?
Yes, a direct ethernet supports multicast. As opposed to an NBMA network like frame relay (rare these days) which does not support multicast for example.
Make sure you are not actually using a switch (which is a bridge) that has IGMP snooping enabled. IGMP snooping will not propagate multicast to nodes that have not sent an IGMP join (assuming your software does not do the join).