Is there a way to have a subtype of a datatype in SML, such that the subtype would cover only some of the datatype's constructors? Something like this:
(* This works as expected *)
datatype Animal = Cat | Dog | Elephant | Zebra;
(* This overrides the Animal constructors instead of co-opting them *)
datatype Pet = Cat | Dog;
(* This causes a syntax error *)
type Pet = Cat | Dog;
I'll add to @molbdnilo's answer: What you mean to say is, can datatype value constructors (Cat, Dog, etc.) be overloaded for multiple types? And the answer is no. This happens to be an uncommon feature in functional languages. Some functional languages feature some degree of overloading of values.
You could say that your Pet type is a subtype of Animal, as in values being a subset, even though SML does not operate with subtypes in this way. Your third attempt of type Pet = Cat | Dog is not possible because type is used for aliases which cannot have their own unique constructors, only refer to existing types rather than values such as Cat and Dog; datatype value constructors live in the domain of values, whereas types Pet and Animal live in the domain of types.
So one thing you can do instead is use parameterised value constructors:
type lives = int
datatype mammal = Cat of lives | Dog
datatype reptile = Turtle | Lizard
datatype fictive = Unicorn
datatype animal = Mammal of mammal
| Reptile of reptile
| Fictive of fictive
Now you have values Mammal (Cat 9), Reptile Turtle, Fictive Unicorn that all belong to the same type. I've used uppercase for value constructors and lowercase for type names just to make it clear what's what, even though this convention is not strictly enforced in SML.