Custom DNS server on AWS

I am trying to create a custom DNS server for my domain on an AWS EC2 Instance. The instance is running Ubuntu 20.04 LTS and the DNS server is configured using bind 9. When I restart the bind server using

sudo service bind9 restart

The /var/log/syslog shows a successful start with no errors.

Now if I run a dig on the server itself like:

dig @localhost mydomain.com

I get the right response. Something like:

; <<>> DiG 9.16.1-Ubuntu <<>> @localhost mydomain.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15819
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 8abafb45d049d921010000006212f87a85a91753aaa86a24 (good)
;; QUESTION SECTION:
;cirtructor.com.                        IN      A

;; ANSWER SECTION:
mydomain.com.         604800  IN      A       99.99.999.999

;; Query time: 4 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 21 02:27:06 UTC 2022
;; MSG SIZE  rcvd: 87

But now when I run the dig command from another machine like:

dig @888.888.888.888 mydomain.com

Where I use my DNS server IP address in place of 888.888.888.888 I get:

; <<>> DiG 9.16.1-Ubuntu <<>> @888.888.888.888 mydomain.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

I also tried doing dig from an instance on the same VPC. Something like:

dig @10.0.0.999 mydomain.com

Still was not able to make it work.

To check whether the DNS server is connected to the internet I opened port 88 in the security group of AWS and ran a http server on port 88. I could get to the http server easily by accessing http://888.888.888.888:88

So the server is connected but somehow the DNS queries do not reach it.

IP Tables policy is ACCEPT and no rules are added. Port 53 is open in the Security Group already.

What steps can I take to figure out what is going on? Is there a way to trace where the DNS lookup fails? Any help or suggestions would be extremely useful.

Solution

This reminds me of two scenarios, I had in the past, have you checked these?

The route back to the querying system was not there/not working at all, so the DNS answers never made it back to the origin. Hence only the local lookup works, but inside the VPC and outside of it resolution doesn't work.
There's some other service listening on UDP/TCP 53. I don't have any exact scenario in mind, where the bind service could respond with success but still can't listen on DNS ports, but I also can imagine that.

Tools I would use are tcpdump and traceroute.