I am building an application, for that I am using serverless and netlify. so for the api which is using serverless I want to use sub-domain api.example.com and for dashboard I want to use example.com via netlify,
I purchase the domain via Route53, so that I can use the serverless, but to use netlify I update the domain name server to netlify name server , and I lost the sub-domain in serverless.
Could it be possible to have two different NS for domain and sub-domain ?
It is possible to to have different name servers for domain and subdomain. You have got to understand the resolution process though.
DNS query goes like this .com -> example.com -> api.example.com
As I understood how you have currently configured your DNS:
example.com -> Delegates to Netlify name servers. Any further subdomains (like api.example.com) will then further be looked into Netlify NS records. If you want to go back to your AWS account again, for that you need to have further delagation for "api.example.com" in your Netlify NS's pointing back again to your hosted zone in AWS.