Suppose I have two register computational blocks in the target machine: I and X. One may apply only integer operations to I-registers and both integer and float operations to X-registers. There're two types of instructions as well:
def ADDIi32 : MyInstruction< ..., (outs I:$Rs), (ins I:$Rm, I:$Rn), [(set i32:$Rs, (add i32:$Rm, i32:$Rn)]>;
...
def ADDXi32 : MyInstruction< ..., (outs X:$Rs), (ins X:$Rm, X:$Rn), [(set i32:$Rs, (add i32:$Rm, i32:$Rn)]>;
def ADDXf32 : MyInstruction< ..., (outs X:$Rs), (ins X:$Rm, X:$Rn), [(set f32:$Rs, (fadd f32:$Rm, f32:$Rn)]>;
...
They are differently encoded and have different asmstrings. So llvm can map
int a, b;
a = a + b;
To ADDIi32 or ADDXi32 but
float a, b;
a = a + b;
gets mapped to ADDXf32 only.
I would like LLVM to use ADDIi32 when possible but unfortunately I found no way to tell it that one instruction (or register) "costs" more than another. CostPerUse in Register class seems to be a candidate but it defines cost among other registers in group, not among all registers. I saw some threads claiming that AddedComplexity (field in Instruction class with ambiguous description) controls choosing patterns but does nothing as well.
It seems that LLVM chooses the first matching instruction. When I define ADDXf32 and ADDXi32 before ADDIi32 it uses only X-functions for any type of data. When I define ADDIi32 before ADDXf32 and ADDXi32 it uses I-instruction for integer data but matches nothing to float data (weird). I suppose I could insert ADDIi32 between ADDXf32 and ADDXi32 but they are currently in different .td files and it doesn't look like long-term solution.
Any ideas?
Note that order of declaration does not affect the instruction selection: LLVM only sorts by matching more complex patterns first (which is what AddedComplexity affects -- note increasing AddedComplexity increases the chances of matching the pattern), tiebreaking by matching instructions with smaller CodeSize, finally tiebreaking nondeterministically. Hence you cannot rely on order of declaration.
EDIT: the below part of my original answer doesn't actually apply to your question -- see http://lists.llvm.org/pipermail/llvm-dev/2007-September/010833.html. In particular:
ISel patterns are matched against DAGs before register allocation. So ISel patterns can't take into account register classes. The register classes in the patterns are a selection constraint for the register allocator if that instruction is chosen, not a constraint on choosing that pattern.
In addition to I and X register classes, you should have a "IntRegsRegClass" or similar, in which you add both I and X registers. (This is the reg class that you call addRegisterClass on.)
The order in which you add the registers in this register class determines the preferred allocation order.