We are setting API Gateway to be accessible only inside VPC or VPC endpoints. In AWS API Gateway you can create a custom domain with Edge or Regional configuration. Is there any way somehow point map DNS name from Route53 to API Gateway "ugly" DNS name for Private type or to VPC Endpoint DNS name but with setting the header parameter automatically(it's also possible to send request VPC Endpoint but with specifying header: <APIGW DNS>)?
This is not possible.
Custom domain names are not supported for private APIs.
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html
If you really wanted this functionality, it could be accomplished with a proxy server (e.g. HAProxy or Nginx) running inside the VPC that accepts requests for the custom domain and forwards requests to the API Gateway private endpoint using the correct Host header and the correct TLS SNI... but this increases complexity and creates an additional dependency in your stack that seems unjustifiable just for the purpose of having a non-ugly domain name for an API that is only consumable internally.