It seems that Docbook Maven Plugin is using its own XSLT transformation. Is it possible to force my own transformation to it?
The customizations docbkx provides is same as docbook. I guess you just want to have your own customization layer that imports the docbook xsls.
This is of course possible. See http://blog.kasunbg.org/2010/12/how-to-use-docbkx-tools-as-maven.html
It provides a detailed guide on how to use docbkx-maven-plugin with your own customization layer.
Here's the configuration
<configuration>
<foCustomization>src/docbkx/xsl/fo.xsl</foCustomization>
<xhtmlCustomization>src/docbkx/xsl/xhtml.xsl </xhtmlCustomization>
</configuration>