Maybe it's more of a paradigmatic/philosophical question, but I'm wondering for quite some time "What do you install if you install PHP?" Is it the interpreter/compiler or the language constructs (functions, syntax elements etc.) or something completely else?
I'm not an expert at php and have no computer science so I find it hard to truly understand...
You are installing a PHP interpreter (it is an interpreted language on its base), along with its built-in functions (APIs provided as a library along with the language for mostly common used functionality). Also a SAPI (Server API) is installed, which allows the PHP runtime environment to be accessed through a web server.
UPDATE: In a sense, each programming language is considered being defined by the commands recognized by its interpreter/compiler. So we could say that the compiler "defines" the language, as it defines its syntax, semantics and particular behavior. I suggest that you read more about the compilers.