Usually, running ./configure --host=xxx is enough. However, sometimes it is source of very subtle bugs - ./configure doesn't always complain (and fail) if something is not 100% right.
For instance, it has happened to me more than once that some feature has been just guessed and as you can imagine, sometimes that guess wasn't right and the bug (odd behaviour) was spotted much later.
In such cases I solve it by looking into configure, writing down the incorrectly guessed variable, checking its correct value on real hardware, putting it into config.cache file and restarting the whole process with --config-cache.
My question is, how safe is to actually produce the whole config.cache on real hardware and copy it over to the build machine?
Obviously, the installed libraries must be the same but what about other possible problems -- for instance what if some tool (grep, python etc) is installed on some other place? Or it is missing altogether? Can ./configure handle this (update its invalid value to a valid/checked one) ?
So far it seems that the only thing needed to update in config.cache is ac_cv_env_host_alias_set and setac_cv_env_host_alias_value, then ./configure happily continues.
My question is, how safe is to actually produce the whole config.cache on real hardware and copy it over to the build machine?
Not very safe. Although some of the things configure determines and caches are target-system properties, I would expect many to be build-system properties, such as details of the cross-compiling toolchain and cross libraries you are using. You cannot determine such things reliably by running configure on the intended host system. I'm surprised that your efforts in this direction have not run into more trouble than you describe. That's probably a sign of a cross-development environment that is well-built and -configured for its target.
Obviously, the installed libraries must be the same
That's probably among the least of the problems with your proposal.
but what about other possible problems -- for instance what if some tool (grep, python etc) is installed on some other place? Or it is missing altogether?
Yes, if the build requires those or similar tools, then those details are some of the build-system properties that I was talking about.
Can
./configurehandle this (update its invalid value to a valid/checked one) ?
No. Where configure relies on a cache file, the whole point is that it does not recheck cached results. You could very likely set that up that kind of thing manually for a project you maintain yourself, but that's not very feasible for third-party code, especially if you're building multiple packages.
I think the bottom line is that cross-compiling is challenging. Native compiling is to be preferred where possible, even if you need to do it in a virtual environment instead of a physical one. And if you can successfully run a project's configure script in the target environment, then you probably have a native development environment there sufficient for just such a native build.