For a high performance trading engine, it matches/handles one order at a time, should it save the order and other critical info to DB before it moves on to the next one?
or can I use a separate thread to save to DB asynchronously?
if asynchronously, I can use zeromq to send a msg containing the critical info to DB server, however, if this crashes, order info are lost.
instead if I use a rabbitmq, which persist message to disk, is it safe enough?
Your question opens a whole Pandorra Box full of very important issues and questions.
ZeroMQ is used in this industry right because it is a smart, devillish fast & has both a minimalistic foot-print & low overheads, nevertheless the solution you ask about is not based on a single layer of technology or on a trivial Yes/No answer.
Whatever promises the IT industry gives -- be it for SSD, advanced SSD, 3D-SSD and similar innovations, the core responsibility is on your end-to-end system architecture feasibility, not on a technology element. Suppose any element may and will fail. There is no "Ehm, Houston, we have a problem" call in the Markets, the less in HFT.
If you take just the last 4 decades of the use of the high performance computation in the Finance Markets, there have been met the principal cornerstones for achieving both the high-performance & fault-resillience. Along this path, there were seen also mis-concepts and devastating effects of poor design and lack of error-isolation practices -- just take the Knights HFT crash ( well, rather hours of HFT still-injecting faulty XTO-instructions ) to have a view of the $ize$ an "impact crater" in thi$ indu$try may have, if the cra$h happen$ to materiali$e.
Let's assume you state for your system both the scope and performance envelopes:
[SCOPE]------------------------------------------------------------
Forex Majors instruments { EURUSD, GBPUSD, <list-all-that-applicable> }
Forex Exotic instruments { <list-all-that-applicable> }
Precious Metals { XAUUSD, XAGUSD, <list-all-that-applicable> }
Commodities { <list-all-that-applicable> }
US-stocks { AAPL, GOOG, XOM, <list-all-that-applicable> }
Euro-stocks { <list-all-that-applicable> }
*-stocks { <list-all-that-applicable> }
US-indices { <list-all-that-applicable> }
Euro-indices { FTSE, CAC, DAX, AMS, <list-all-that-applicable> }
*-indices { <list-all-that-applicable> }
Futures { <list-all-that-applicable> }
Options { <list-all-that-applicable> }
Binary Options { <list-all-that-applicable> }
Options on Futures { <list-all-that-applicable> }
Options on Indices { <list-all-that-applicable> }
BT*-based instruments { <list-all-that-applicable> }
In a similar structure, state for design evaluations the real-time operations performance constraints, with both sustained & peak values in [msec]
[PERFORMANCE]------------------------------------------------------
System requirement for a number of Market Events per msec per segment
System requirement for a number of PriceFeed Quotes per msec per segment
System requirement for a number of OrderProcessing Events per msec per segment
This gives you the first, elementary, metrics for constraining your system Real-Time-Operations design. I.e. what overhead / latency limits you have to design against, and thus what tools and what measures you have time in [usec]-s to incorporate in any stage of the pipeline so as to still guarantee the system real-time stability and robustness thresholds.
This is not a question. This is a given externality.
[Order] matching is typically given by the Market Regulatory Body ( if not designing for a Dark Pool & other non-public quasi-market(s) ) so as to meet a Fair Market principle. Order handling is sub-ordinated to your regulatory & audit-supervision duties. Both of these are not a field for a free will of the system design.
The [Order] and other critical info to DB before it moves on to the next one or can I use a separate thread to save to DB asynchronously? .. is answered more by the system requirements from audit/supervisory bodies than from the opinion of the technical committee.
If your system architecture design permits, it can crash. Then you need measures for post-crash re-instating
If your system architecture design requires to develop system that does not crash, this will become an empty issue. Application online-mirroring + hot-standby shading with N+1, (N+1)+1 or (N+M)+1 redundancy models are achievable, so that individual crashes are properly isolated.
Depends on your Business Disaster Recovery Planning & Business Continuity Measures. Testing is inevitable.
Remember the famous Nobelist Dick Feynman's finger pointing into the middle of the Nuke Plant blueprints ...