I’ve been working on a project to try and improve WAN throughput by sending data over parallel TCP connections. The underlying problem is that even if you have a “large” pipe between two geographic locations (e.g. New York and San Francisco) you won’t be able to get anywhere near the bandwidth offered by the pipe because of latency, window sizes, buffers, overhead, etc. For a good resource on this issue, see the enabling high performance data transfers page over at the Pittsburgh Supercomputing Center.

For the past few weeks, I’ve been struggling with getting the solution to work consistently on a saturated link. The failure rate was pretty low, on the order of 1-3%. Our application can’t tolerate any failures, so it is important that we get to the bottom of it. And the failures were not graceful– the application would hang in an apparent deadlock with the receiving side blocking on a read(). I tried experimenting with several different ideas (timeout on sysread/syswrite, non-blocking io, simplified multiplexing), but I wasn’t able to get to source of the problem.

I had a breakthrough yesterday. I added some debugging so that I could identify each chunk of data sent from the source to the destination. To this identification, I also added the destination port the sender thought it was talking to and the receiving port the destination received the chunk on. This was tremendously helpful in revealing how the parallel TCP connections were setup. It also led me to the discovery that you can’t assume that the order in which accept() receives connections will be the same order that the connections were initiated from the other end. This appears to have been the source of the problem. I changed the code so that it identifies the connection by the initiating process’ unique high port. I have yet to see a deadlock condition after a few hundred transfers. Sweet.

A related application I found is bbcp. bbcp works great. They have done a lot of analysis on improving throughput across the WAN. See their bulk file transfer measurements.

Related links from my del.icio.us.

Technorati Tags: networking, performance, file, transfer, slac, bbcp