Can it be assumed that a browser that indicates Mozilla/5.0 (compatible...) is HTML5 capable?
No. You should use something like Modernizr to detect features and fallback where necessary.
HTML5 is a collection of different features, so you should check for the browser features you want. The User-Agent field shouldn't be used for determining compatibility, since the client program can lie about that. User-Agent sniffing is not a recommended way to determine compatibility -- it's too error-prone, and there are multiple browsers that have ways of changing the User-Agent field to appear as other browsers.
The reason that the Mozilla/5.0 or Mozilla/4.0 shows up in User-Agent for so many browsers is to show that they have backward compatibility with some version of Netscape (4.0 = Netscape 4.0 -- 5.0 is compatible with the original suite (aka SeaMonkey)). Basically every browser out there will claim this compatibility, so it's not a very useful indicator.