How can I implement my own validation function for a grid?
Documentation has just requireLimitOneResponsePerColumn.
I would encourage you to file a feature request with this template if you want this functionality:
https://issuetracker.google.com/issues/new?component=191640&template=823905
Be sure to add in some good justification, use cases, context, examples, what it might look like in code. As much information as possible.
Since this feature request will probably not get implemented very soon. If you absolutely need some specific grid validation, and you want to keep it within Google Apps Script, your best option would be to make a web app, and basically build out your entire form in HTML/CSS/JavaScript. The great advantage of this method is that you have complete control and you can even use JS libraries.
Be sure to check out the Client-to-server Communication article, for more information about how to pass information from your Apps Script "Back End" to the web app "front end".