I have a custom entity and I want to create a N:N relationship to a customizable entity (ServiceAppointment).
So I go in the custom entity > N:N Relationships > New Many-to-Many Relationship. Now in the "Other Entity" section where we can choose the other entity name, there is a list of available entities for the N:N, but the ServiceAppointment is not in the list. Also I can't create a N:N relation from the ServiceAppointment side.
Is it a limitation of CRM 4.0? And is there a way to "unlock" the entity so we can create N:N relationships with this entity?
Yes, this is a limitation of CRM. Several entities are not available for customization or relationships.
I don't have a 4.0 environment handy, but if that entity allows 1:N relationships, I'd probably just create a custom entity with lookups to ServiceAppointment and your other entity that would serve the same purpose as a native N:N relationship.
It might be possible to update the database or customizations to allow this type of relationship, however, those types of changes are considered unsupported by Microsoft, and you will have trouble getting support if anything goes wrong.