If you used bit masks to store read and write lock in a single AtomicInteger, could you realize a fast ReadWriteLock class?
How would it be different from a regular ReentrantReadWriteLock?
TL;DR - It won't work.
As @Radiodef points out, you would not be able to implement the ReadWriteLock API. Methods such as getOwner, getQueuedThreads and so on are unimplementable if the state of the lock is just a single AtomicInteger.
Full reentrancy would be unimplementable. Reentrancy typically requires you to encode the identities of the threads currently holding the lock, and the reentry count for each one. For the readers we could use a single count (and no identity) but the single writer needs both an identity and a count. Shoe-horning a count and a thread identity into 32 bit integer probably won't work. (A Thread does offer a numeric id attribute that is unique and unchanging for the thread's lifetime ... but the id is a long.)
If you use just a single AtomicInteger as the state of the lock, you cannot park a thread waiting on a contended lock. (For parking to work, the thread that releases a lock needs to know which thread to unpark. But you can't represent that.) This means that you would need to use spinlocking1 which is expensive and not scaleable.
In summary, you can't implement the ReadWriteLock API, or full reentrancy semantics. If you removed those requirements, you could possibly implement simple read-write locks (reentrant for readers, nonreentrant for writers) but you would need to do spinlocking.
1 - With spinlocking, a thread waiting for a contended lock "spins" executing a busy loop until the lock is available. This is OK for locks when contention is unlikely and short-lived ... or when there is nothing else that the core could do. But it is too inefficient for normal uses.