Is there any way to use the Google Smart Home actions platform for custom devices that are not supported like let's say a motion sensor? If not is there any way we can create custom actions and make it work together with Smart Home platform?
The "types" of devices are somewhat trivial to support or not support. They're really useful semantically, when the user is referring to a set of devices (like lights). What is more important are the traits, which represent the capabilities of devices.
For a motion sensor, you can register it as any device type (such as a Light). While that may not make a lot of sense, it'd work. You would then want to take a look at the traits for what capabilities make sense.
Consulting the list, you may or may not find one that fits. If you need custom capabilities, you should take a look at building a conversational Action which provides greater flexibility.