When my HTTP proxy was a simple multi-threaded one, GET and CONNECT requests went fine: now I want to fit, in the middle of the TCP dialogue between clients and servers, a queueing mechanism where bytes coming from client and servers are pushed in priority queues, and only data with highest priority are popped from queues and sent to destination.
The effective data I got from recv()ing from clients and servers is stored in a struct infoPkt together with some metadata:
typedef struct infoPacket {
int clientFd;
int serverFd;
int request;
int direction;
int priority;
char payload[MAX_PAYLOAD_SIZE];
int payloadSize;
in_addr_t serveraddr;
char host[256];
std::chrono::time_point<std::chrono::system_clock> start;
bool first;
bool last;
} infoPkt;
I can't post all the code here, it is too much. I'll try to explain the flow of a packet infoPkt in my program:
I have the following classes:
ClientManager: looping with accept(), getting HTTP request from client, crafting the first infoPkt of this connection with the request inside it and pushing it in prio queues DequeManager::getInstance().insert(newElem);, start accept()ing again;
HTTPManager: this class, which methods are all static, deals with actual data exchange between client and server; HTTPManager::dealPkt(infoPkt p) is called from DequeManager, associating it to queueingThread thread (see DequeManager below), when first packet of connection is removed; after sending 200 OK to client, I loop withselect() client and server fds to check for incoming data; what I recv(), I insert in prio queues in DequeManager; if I recv() 0 bytes from either client or server, I craft a infoPkt with no payload but with infoPkt.last set to true and break the select() loop;
DequeManager: two threads at work here, one is looping to check for starvation (if a packet has spent too much time in queue, it will be moved in a queue with higher priority) and another is always removing packets from queue with higher priority first, then from queues with lower prio when the higher one is empty; when the thread removes a infoPkt, we have two scenarios:
serverFd in infoPkt is -1: this means this is the HTTP request packet, no connection has still been established because I have no server file descriptor associated; remover thread does socket(), bind() and connect() to remote server, set the new server fd to infoPkt.serverFd, calls std::thread queueingThread(HTTPManager::dealPkt, packet); queuingThread.detach(); and goes on removing again;
serverFd in infoPkt is > 0: this means this is a infoPkt belonging to a connection already existing, pushed in queue by client or server; first I check infoPkt.last flag, if it is true this is the last packet of the connection, recv()ing from client or server returned 0 so I crafted this infoPkt with no payload but setting infoPkt.last to true, so I perform close(infoPkt.clientFd); and close(infoPkt.serverFd);; otherwise, if infoPkt.serverFd is > 0 and infoPkt.last is not true, this packet will be sent to destination checking infoPkt.direction.
The "flow manipulation" I wrote in the title is referring to data exchange in select(), in a CONNECT HTTP connection: the data I get from client is not sent to server right away, but pushed in prio queues and sent later, only when remover thread will remove that data because, according to priority mechanisms, conditions are met to tell it is its turn. While my previous proxy, with immediate data forwarding to destination when data were received went fine: I'm trying to fit the priority mechanism in between, delaying data exchange, but the programs does not work as I expected; connection closes itself after sending one or two packets, and I can't figure out why.
Here's the output of my program when I try to connect to Netflix homepage.
Created ClientManager
Start PROXY, thread id 67426496
Number of interfaces (or their IP addr) has changed, mmap() again
Found new interfaces: wlan0[192.168.1.88]
priority 1 to hostname www.netflix.com
priority 1 to hostname nflxvideo.net
priority 2 to hostname www.youtube.com
priority 2 to hostname googlevideo.com
priority 3 to hostname www.facebook.com
remover thread with method DequeManager::removeAll() detached
controller thread with method DequeManager::aging() detached
67426496 THREAD MAIN
Proxy listening to port 8000, here we go!
67426496 ClientManager::getRequestFromClient received from client 215 byte
67426496 ClientManager::manageClient() inserted C 4, S -1, CONNECT, UPLOAD, prio 1, size 215, host www.netflix.com, first 1, last 0 --->
CONNECT www.netflix.com:443 HTTP/1.1
Host: www.netflix.com:443
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
67426496 ClientManager::getRequestFromClient received from client 215 byte
67426496 ClientManager::manageClient() inserted C 5, S -1, CONNECT, UPLOAD, prio 1, size 215, host www.netflix.com, first 1, last 0 --->
CONNECT www.netflix.com:443 HTTP/1.1
Host: www.netflix.com:443
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
111642368 DequeManager::removeAndDealPkt created pair (4 7)
137008896 HTTPManager::dealPkt() thread 137008896 started
137008896 HTTPManager::dealPkt() dealing with socket fd pair (4 7)
137008896 HTTPManager::dealPkt [CONNECT] sent 200 OK to client
137008896 HTTPManager::dealPkt [CONNECT] max fd+1 between client 4 and server 7: 8
137008896 HTTPManager::dealPkt [CONNECT] queued up pkt from client, 209 bytes
111642368 DequeManager::removeAndDealPkt created pair (5 8)
111642368 DequeManager::removeAndDealPkt send() C 4, S 7, CONNECT, UPLOAD, prio 1, size 209, host www.netflix.com, first 0, last 0
145401600 HTTPManager::dealPkt() thread 145401600 started
145401600 HTTPManager::dealPkt() dealing with socket fd pair (5 8)
145401600 HTTPManager::dealPkt [CONNECT] sent 200 OK to client
145401600 HTTPManager::dealPkt [CONNECT] max fd+1 between client 5 and server 8: 9
111642368 DequeManager::removeAndDealPkt sent_bytes 209
145401600 HTTPManager::dealPkt [CONNECT] queued up pkt from client, 209 bytes
111642368 DequeManager::removeAndDealPkt send() C 5, S 8, CONNECT, UPLOAD, prio 1, size 209, host www.netflix.com, first 0, last 0
111642368 DequeManager::removeAndDealPkt sent_bytes 209
137008896 HTTPManager::dealPkt [CONNECT] recv() 0 from client, crafting last packet 4 7
137008896 HTTPManager::dealPkt dealPkt() thread terminated
145401600 HTTPManager::dealPkt [CONNECT] recv() 0 from client, crafting last packet 5 8
145401600 HTTPManager::dealPkt dealPkt() thread terminated
111642368 DequeManager::removeAndDealPkt get LAST PACKET, closing fds 4 and 7
111642368 DequeManager::removeAndDealPkt get LAST PACKET, closing fds 5 and 8
67426496 ClientManager::getRequestFromClient received from client 215 byte
67426496 ClientManager::manageClient() inserted C 6, S -1, CONNECT, UPLOAD, prio 1, size 215, host www.netflix.com, first 1, last 0 --->
CONNECT www.netflix.com:443 HTTP/1.1
Host: www.netflix.com:443
Proxy-Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
111642368 DequeManager::removeAndDealPkt created pair (6 5)
145401600 HTTPManager::dealPkt() thread 145401600 started
145401600 HTTPManager::dealPkt() dealing with socket fd pair (6 5)
145401600 HTTPManager::dealPkt [CONNECT] sent 200 OK to client
145401600 HTTPManager::dealPkt [CONNECT] max fd+1 between client 6 and server 5: 7
145401600 HTTPManager::dealPkt [CONNECT] queued up pkt from client, 215 bytes
111642368 DequeManager::removeAndDealPkt send() C 6, S 5, CONNECT, UPLOAD, prio 1, size 215, host www.netflix.com, first 0, last 0
111642368 DequeManager::removeAndDealPkt sent_bytes 215
145401600 HTTPManager::dealPkt [CONNECT] recv() 0 from client, crafting last packet 6 5
145401600 HTTPManager::dealPkt dealPkt() thread terminated
111642368 DequeManager::removeAndDealPkt get LAST PACKET, closing fds 6 and 5
^CCaught signal 2
If someone will say "this is wrong because data exchange in TCP connection is not expected to be manually delayed from within with a priority mechanism of yours", I'll throw away my code and put my mind at rest, finding another way to achieve the traffic control in a TCP connection, packet per packet; sincerely, I'd prefer someone saying to me "you moron, just check this and everything will work", so I know I will not have wasted months on this amateur project of mine :D
Edit: I'm providing only the significative code from my project, if you have any question feel free to ask. Here's my Gist
The problem is that TCP has an explicit mechanism to fix reordering problems. This may involve NACK's which force the sender to resend packets.
Your "priority" reordering of bytes in a TCP stream will at best create a lot of resending of data, so your "priority" degrades everything. But it's quite well possible that either side just gives up when the TCP stream is too disrupted.