I want to invalidate refresh jwt token without maintaining a blacklist of used refresh tokens with rotations, for this I had the idea of including a ValidationCode in the payload of the RT that the server generates and store whenever 2 refresh tokens are detected being in use with different rotation number (As an example RT2 that the normal user got from his last request and RT3 that the malicious user generated after sending a malicious request using the old RT2).
Once the server finds that a RT2 is in use while the latest one is RT3. The server should "Invalidate" the previous tokens, and issue a new RT when the user reconnects using his password etc. The process of invalidating token is simply changing the ValidationCode in the newly generated token, and accepts any request in which the token is valid + the validation code in the payload matches the one stored in the server for that user.
If Using this approach, if the malicious user try to use RT3 again, even if the jwt token is valid, the ValidationCode now changed and it will not match the one in the server however the newly generated tokens will.
Is this approach secure and good enough to replace blacklisting old tokens? which I think defies the purpose of using jwt at first + wasting time and memory storing the list and querying in the database
What you're describing here is a solution where you can just keep the latest RT used by the user in the database and allow only refresh requests with the RT saved in the DB. This is a valid approach but it has one drawback: you can have only one active pair of AT/RT for the user. If that is OK for you then you can go with this solution.
wasting time and memory storing the list and querying in the database
Either way you will have to query the database, so that doesn't change much. What you gain is a bit of storage space.