I used data-fancybox="" to open fancybox2 iframe (without any extra script). When I scroll the page by dragging the scrollbar, the modal is closed on mouse release (It acts like clicking outside of modal). However scrolling using the mouse wheel works fine. I used fancybox like this:
<a href="email.php" data-fancybox="">Send Mail</a>
The problem was solved by moving from fancybox2 to fancybox3 however data-type is mandatory in fancybox3 and is not iframe by default.