gnuplot: Plotting matrix with the colors I want

I want to plot a matrix with colors with gnuplot, and I'm using the following code:

set terminal png
set output "Matrix.png"
set title "Matrix"
set autoscale xfix
set autoscale yfix
plot "Lattice.txt" matrix with image

Lattice.txt can be filled with 0, 1, 2, 3, 4 and 5.

The code as it works for the most part, but if in lattice.txt there are only 0 and 1, it will assign the black color to 0. If I have 1, 2 and 3 in Lattice.txt, it will assign the black color to 1.

How can I choose a color for each number so it always outputs, for example, black when there is a 0, red when there is a 1, blue when there is a 2?

Thanks for your time.

Solution

set palette define (0 "black", 0 "black", 1 "red", 1 "red", 2 "blue", 2 "blue", 3 "orange", 3 "orange", 4 "gold", 4 "gold", 5 "green", 5 "green")
set cbrange [0:5]

[modified to show that for this case a simpler palette definition is OK]

set palette define (0 "black", 1 "red", 2 "blue", 3 "orange", 4 "gold", 5 "green")
set cbrange [0:5]