If I were to want to create a new programming language, is there a specific file format or extension used to define the grammar of this new language? Or should I just use a plain text file?
What you are looking for might be a metasyntax notation, sometimes also called "formal grammar".
A common notation to describe at least certain types of formal languages is Extended Backus-Naur Form. It defines the grammar by means of "production rules" that gradually expand a "start symbol" into any valid expressions in the described language.
This format is, however, mainly a notation (i.e. it describes how you write down the language description, even when you're writing on paper). It is not so much a "file format" in the sense that there would be a standard extension for it.