What I want is to have @ModifiedBy, @LastModifiedDate, @CreatedBy and CreatedDate to be saved to all audit tables. The thing that makes it difficult is that I don't want to have those fields in my @Entity POJO. How can this be done if it even is possible?
It is, but the implementation is what differs slightly to what you said.
The problem with introducing those things in the actual audit-row itself is you run the risk of potentially having a lot of denormalized data scattered across the audit schema, especially when you think about the fact that one audit revision can encompass multiple entities.
The Envers-way to accomplish what you describe is to tackle the pieces separately.
In order to capture who performs an audit-operation, the best way to do this is to extend or provide your own implementation of the revision-entity. In this entity you'd include a column meant to store the user name or whatever user identifying value you want.
In order to populate the user name or identifying value in that entity, you'll also need to write a custom RevisionListener and specify that on the revision-entity's @RevisionEntity annotation. You can find examples in the user documentation here.
In order to get the timestamp of the revision, you'd need to fetch not only the entity but the revision-entity for that audit row too. In doing so you're able to obtain not only the timestamp the revision occured, the custom field for who made the change, but also the type of revision (ADD,MOD,DEL) so you can then branch on whether the values you're reading are the Creation vs Modification roles.