Could anyone point me to the relevant section of the FIX spec pertaining to rejected order modification?
Please consider the following scenario:
A limit order (NewOrderSingle: ClOrdID='blah.0') is placed and
confirmed as submitted by the broker
Modification request of the order (OrderCancelReplaceRequest:
ClOrdID='blah.1'; OrigClOrdID='blah.0') gets rejected due to, say,
limit violation
What happens to the original order (ClOrdID='blah.0')? Is it still considered valid and can be filled? Does the FIX specification define the expected behavior for such scenarios and the expected state of the original order?
TL;DR
You should consult your counterparty's FIX specification document(s) for the exact behavior to expect from that specific counterparty when an attempt to replace a working order is rejected.
Long answer
Assuming nothing has happened to the original order 11=blah.0 between the time it was placed and the OrderCancelReplaceRequest with 11=blah.1|41=blah.0 was sent and rejected (e.g., fill, partial fill(s), external cancel), the original order 11=blah.0 should still be working, and can be filled.
There is nothing in the FIX specification that states the exact expected outcome when an attempt to replace a working order is rejected. Since most exchanges/brokers use some flavor of FIX 4.2, I'll point to the documentation for that version:
Order Cancel Reject - The order cancel reject message is issued by the broker upon receipt of a cancel request or cancel/replace request message which cannot be honored. Requests to change price or decrease quantity are executed only when an outstanding quantity exists. Filled orders cannot be changed (i.e quantity reduced or price change. However, the broker/sellside may support increasing the order quantity on a currently filled order).
In the message specification it has:
Tag | Field Name | Req'd | Comments
39 | OrdStatus | Y | OrdStatus value after this cancel reject is applied.
Whatever the counterparty provides for OrdStatus in the OrderCancelReject message is the state of the original order. I have never run into any counterparty that cancels the original order when a replace request is rejected, but I suppose it's possible. If a counterparty does handle the situation this way, any documentation provided by the counterparty should clearly state so.