This question pertains to the inconsistent behaviour of Mediawiki syntax and its templates and parser functions.
When placed directly in a wiki page, both these snippets work as expected:
{{#ifeq:{{{1|}}}|{{{2}}}|{{{1}}}}}
and
{{#ifeq:{{{1|{{FULLPAGENAME}}}}}|{{NAMESPACE:{{{1|{{FULLPAGENAME}}}}}}}:{{ROOTPAGENAME:{{{1|{{FULLPAGENAME}}}}}}}|{{{1|{{FULLPAGENAME}}}}}}}
The first snippet is simple. The second outputs the full name of the page, but only if it is not a sub-page.
However, when each snippet is placed in a template, let's say the first in Template:IsEqual, and the second in Template:IsRootPage, both of which are then included in some page, such as the output of each is the subject of an #if test, the first continues to behave as expected, while the second always yields the string not empty section of the #if statement.
So:
{{#if:{{IsEqual|A|B}}|equal|unequal}} will output "unequal"
while:
{{#if:{{IsRootPage}}|root|non-root}} will always output "root", even if placed in a subpage or if the name of a non-top-page is passed as a parameter to Template:IsRootPage, driving the template to return nothing.
According to the documentation of #if:
This function evaluates a test string and determines whether or not it is empty. A test string containing only white space is considered to be empty.
Moreover, the inconsistent behaviour persists for Template:IsRootPage even when it is amended to return an empty string when the #ifeq test fails, by introducing a fourth empty parameter to the #ifeq in the template, i.e by adding a third | followed by nothing/space.
It seems that something happens in between the output of the template and the #if parser function.
Is there a known explanation for this discrepancy in behaviour?
Is there some syntax hack to make it work as expected?
Or am I missing something that's otherwise obvious?
Edit: This behaviour is witnessed in Mediawiki 1.35.5
Your sample template does not return an empty string when the page has sub pages. It returns <nowiki/> for which {{#if:<nowiki/>|yes|no}} properly returns yes.
If you remove the <nowiki/>, you'll get the behavior you want.