I know that Dijkstra's algorithm in reality is implemented using a Fibonacci heap. But can it also be implemented using a red black tree and still have a worst-case running time of O(m log n)?
For starters, it's rare to actually see Dijkstra's algorithm implemented with a Fibonacci heap. Although the Fibonacci heap gives great asymptotic performance (O(m + n log n)), in practice it has such high constant factors that other types of heaps are more efficient.
As to your question - yes, you could use a red-black tree as a priority queue to get O(m log n) performance. This works because you can find the minimum element in a red-black tree in O(log n) time and simulate a decrease-key operation on the tree in time O(log n) by doing a deletion followed by an insertion. However, this is probably not as efficient as using a standard binary heap, since the red-black tree has worse locality of reference and more memory overhead. More generally, you always can use a balanced binary search tree whenever you need a priority queue, though usually doing so is overkill.
Hope this helps!