Zmq stalls under high load

Description

Under high load of req/rep connecting and disconnecting, zmq seems to stall inside it's event loop.

The test case is a client with one threads that does :

  • connect to the server ;

  • send a number ;

  • read response.
    All of this in a loop.
    The main thread does :

  • bind to address ;

  • poll incoming ;

  • read a empty query ;

  • respond with empty response.
    All of this in a loop.

The server part does :

  • bind to address ;

  • poll incoming ;

  • read a query and print the number contained in it ;

  • respond an empty response ;

  • create a socket ;

  • connect to the client ;

  • send data ;

  • read response ;

The server part is a

http://pastebin.com/S9iZM8Av server
http://pastebin.com/5y3c5SaA client

The problem does not occur on freebsd 9-rc3 but does on other linux tested.
It does not occur with "ipc://" instead of "tcp://"

Attached is a picture of what the intel profilers says.

Environment

linux debian stable; Linux 2.6.32-5-amd64

Attachments

1
  • 17 Dec 2012, 03:57 PM

Activity

Show:

Vincent Auclair December 18, 2012 at 1:29 PM

Not a bug in zmq.

Vincent Auclair December 18, 2012 at 1:28 PM

Hello, sorry for the noise, I have found the issue.
The problem is that the kernel does not recycle the connections fast enough.
Setting the following values stopped the problem :

sysctl net.ipv4.tcp_tw_reuse=1
sysctl net.ipv4.tcp_tw_recycle=1
sysctl net.ipv4.tcp_fin_timeout=1

As hinted by (second comment) :

http://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux

Won't Fix

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

Created December 17, 2012 at 3:57 PM
Updated December 18, 2012 at 1:29 PM
Resolved December 18, 2012 at 1:29 PM