Zmq stalls under high load
Description
Environment
linux debian stable; Linux 2.6.32-5-amd64
Attachments
1
- 17 Dec 2012, 03:57 PM
Activity
Show:
Vincent Auclair December 18, 2012 at 1:29 PM
Not a bug in zmq.
Vincent Auclair December 18, 2012 at 1:28 PM
Hello, sorry for the noise, I have found the issue.
The problem is that the kernel does not recycle the connections fast enough.
Setting the following values stopped the problem :
sysctl net.ipv4.tcp_tw_reuse=1
sysctl net.ipv4.tcp_tw_recycle=1
sysctl net.ipv4.tcp_fin_timeout=1
As hinted by (second comment) :
Under high load of req/rep connecting and disconnecting, zmq seems to stall inside it's event loop.
The test case is a client with one threads that does :
connect to the server ;
send a number ;
read response.
All of this in a loop.
The main thread does :
bind to address ;
poll incoming ;
read a empty query ;
respond with empty response.
All of this in a loop.
The server part does :
bind to address ;
poll incoming ;
read a query and print the number contained in it ;
respond an empty response ;
create a socket ;
connect to the client ;
send data ;
read response ;
The server part is a
http://pastebin.com/S9iZM8Av server
http://pastebin.com/5y3c5SaA client
The problem does not occur on freebsd 9-rc3 but does on other linux tested.
It does not occur with "ipc://" instead of "tcp://"
Attached is a picture of what the intel profilers says.