I'm creating an instruction set with only 16 instructions (4 bits opcode, 4 bits immediate), and as such am wondering if I could replace having both ADC (Add-with-carry) and SBC (Subtract-with-carry) with RWC (Repeat-with-carry).
RWC would function as ADC if the last add/subtract/rwc instruction was an add, using its carry.
RWC would function as SBC if the last add/subtract/rwc instruction was a subtract, using its carry (aka borrow).
Simply put, would there ever be a need to use a carry as a borrow or a borrow as a carry (Ex: "add a,b,c; sbc d,e,f;" or "sub a,b,c; adc d,e,f;")? Would there be other potential limitations for this design?
It happens every once in a while. For example, a typical use case of sbc following an add is to materialise the carry into a register:
add r0, r1, r2 # r0 = r1 + r2
sbc r3, r3, r3 # r3 = -1 if addition produced carry, 0 otherwise
Another use case is using adc as a left-rotate-through-carry, accumulating bits into a register. The bits to be accumulated might come from arbitrary sources, not all of which cause the instruction to be preceded by addition.
Another issue you should consider is that when an interrupt occurs, you need to remember if an addition or subtraction was just happening so you can execute the next rwc instruction correctly. So there must be an extra status bit in the flags register keeping track of that.
My advice: do not implement this operation. Instead implement subtraction as a reverse subtraction (i.e. dest = src - dest instead of dest = dest - src) that does not alter flags. Then you can implement subtraction with carry by first subtracting from 1 (to compute the complement) and then adding with carry.
Another option is to implement a skip instruction that checks the condition flags for a condition and skips the next instruction if the condition is met. Then you can implement both adc and sbc by conditionally skipping an instruction to add or subtract 1 (respectively) before performing the addition or subtraction. But note that this will cause the carry to differ for when the input is 0, so you might need to tweak the design. Another side effect is that you don't need conditional jumps anymore; these become skip if not condition + unconditional jump.
A third option keeping closer to your previous idea is to instead build an “apply carry in” instruction that modifies the previous instruction's result according to the old carry in. This instruction would require two additional status flags (one for add/sub, one for carry in) but seems to be more flexible. It could also be expanded into a general modifier instruction, e.g. distinguishing between ror and rcr and others.
If you are looking for inspiration for such a restricted instruction set, consider looking at the PDP-8, a famous computer with just 8 instructions that nevertheless elegantly provides all required operations.