I'm working on website that is quirks mode and can't use <!DOCTYPE html>.
In my project CSS3 translate image doesn't work in IE10 and 11 but it does work in IE9 for the reasons mentioned above.
Is there a solution to this problem except add <!DOCTYPE html>?
Just to be clear: The whole point of quirks mode is to emulate the way IE5 worked.
This includes disabling most of the HTML, CSS and JavaScript features that have been added to the browser since then.
If you are in quirks mode then you do not have those features. There is no work around or solution to this; that is how it is, and that's how quirks mode is intended to work.
If you have to use quirks mode then you also have to accept that your browser's capabilities will be from the 1990's.
Your only options are to work out how to upgrade your software so you can use it in a more modern browser environment, or else ditch the idea of using modern CSS and use whatever techniques where available back in the '90s. Probably a custom ActiveX control or a Flash animation. Pretty ugly, whichever option you go for.
I will also add the obvious point you really need to start thinking about how you're going to upgrade away from quirks mode. The default browser (Edge) that comes with Windows 10 doesn't support quirks mode at all. There will come a point in time when you simply can't avoid upgrading your desktop computers, and at that point, you may find that you simply won't be able to run your quirks mode site. It will be a lot easier if you start working toward dealing with that now than if you leave it and find it hits you in the face later on.