I saw many web application firewall like mod_security with OWASP extention
If I use that in my server, can I be sure by 99% that no one can hack my site with PHP codes ? likes XSS ...
There is no 99% sure, the only way to be 99% sure is to "unplug the Ethernet cable" (figuratively speaking, I realize it's all virtual), and even then it's not 100%.
I would recommend running PCI Compliance scans, it is relatively inexpensive and it will give you an idea what known vulnerabilities your server/app is vulnerable to. You can also do penetration testing, there are many services for that, it will provide additional insights.