If Java allowed "instanceof" as a name for variables (and fields, type names, package names), it appears, at a first glance, that the language would still remain unambiguous.
In most or all of the productions in Java where an Identifier can appear, there are contextual cues that would prevent confusion with a binary operator.
Regarding the basic production:
RelationalExpression:
...
RelationalExpression instanceof ReferenceType
There are no expressions of the form RelationalExpression Identifier ReferenceType, since appending a single Identifier to any Expression is never valid, and no ReferenceType can be extended by adding an Identifier on the front.
The only other reason I can think of why instanceof must be a keyword would be if there were some other production containing an Identifier which can be broken up into an instanceof expression. That is, there may be productions which are ambiguous if we allow instanceof as an Identifier. However, I can't seem to find any, since an Identifier is almost always separated from its surrounding tokens by a dot (or is identifiable as a MethodName by a following lparen).
Is instanceof a keyword simply out of tradition, rather than necessity? Could new relational operators be introduced in future Java versions, with tokens that collide with identifiers? (For example, could a hypothetical "relatedto" operator be introduced without making it a keyword, which would break existing code?)
That question is different, it's asking why "instanceof" isn't a method, I'm asking whether there are reasons syntactically why
You have a point in that it could have been a method on Object or we have
if (myClass.class.isInstance(obj))
This is more cumbersome, however I would say that chains of instanceof are not considered best practice and making it a little harder might not have been a bad idea.
It is worth noting that earlier version of Java didn't use intrinsics as much as they do now and using a method would have been far less efficient than a native keyword, though I don't believe that would have to be true today.
Is instanceof a keyword simply out of tradition, rather than necessity?
IMHO keywords were/are considered good practice to make words with special meaning stand out as having and only having a special purpose.
Could new relational operators be introduced in future Java versions, with tokens that collide with identifiers?
Yes, One of the proposals for adding val and var is that they be special types, rather than keywords to avoid conflicting with code which have used them for variable names.
Given a chose, a new language would make these keywords and it is only for backward compatibility that they might be other wise. Alternatively it has been considered to use final rather than val and transient rather than var.
Personally, I think they should add them how other languages do it for consistency otherwise you are going to have every new Java developer asking basic questions like How do I compare strings in Java? What they did made sense but it confuses just about every new developer.
By comparison, they banned making _ a lambda variable to avoid confusion with other languages where this has a special meaning and they have a warning about using _ as a variable that it might be removed in future versions.