I have two debian boxes connected by a CX4 cable going between two 10 GbE cards. One is going to be generating data very quickly (between 4Gbits/s and 16Gbits/s), and the other needs to be able to grab all of that and store it in RAM for later parsing. I'm new to this kind of low-level coding, and would happily accept any ideas about what broad approach to use (do I need DMA? RDMA?), or tips and tricks that might apply. Thanks!
If you want to constantly process 1 GB of traffic a second you need a very wide buss and a very fast processing rate, and my experience comes from NIDS. You need specialized hardware to consistantly perform NIDS processing 100MB (1 Gig ethernet) of data (10 Gb is another universe). Ram will not help you because you can fill a GB in 5-10 seconds and 1 GB holds a lot of requests.
If you are trying to do any form of business or web processing with 10 gig, you probably need to put a load distributer that can keep up with 10GB of traffic at the front.
p.s., I must clarify that NIDS is 1:1 traffic processed on the machine that sees the traffic -- i.e, worst case scenario you process every byte on the same machine; whereas business/web processing is 1:many: many machines and an order of magnitude many bytes to process.
-- edit --
Now that you have mentioned that there is a gap between data delivery (no standard 10Gb nic can keep up with 10Gb anyway), we need to know what the content of the processing is before we can make suggestion.
-- edit 2 --
Berkeley DB (a database with a simple data model) behaves like a enterprise database (in terms of transaction rate) when you use multiple threads. If you want to write to disk at high rates you should probably explore this solution. You probably want a raid setup to boost throughput -- raid 0+1 is best in terms of IO throughput and protection.