I built an one-dir app with pyinstaller but I found that there are some folder with some module files like License, Metadata, Installer etc. The folders are named as module-version.dist-info
will there be any issue if I remove those folders from the one-dir?
You should not remove any of them.
While your program may run and function fine without a few of them, you are likely violating some of the OSS licenses you have agreed to if you remove any of them. Also all of the files serve some purpose otherwise the package maintainers would not have included them in their package.
Removing them is unlikely to make much of an impact on size, and every file you remove increases your chances of breaking something.