Search code examples
haskellhttp-conduit

When I run a lot of http requests in parallel, I get "no such protocol name: tcp"

The code

mapConcurrently httpLBS requests

throws

ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp))

after about 1k requests.

Solution

  • The requests are all run in parallel and make the process run out of file descriptors. Use

    import ClassyPrelude

traverseThrottled :: (MonadMask m, MonadBaseControl IO m, Forall (Pure m), Traversable t) => Int -> (a -> m b) -> t a -> m (t b)
traverseThrottled concLevel action taskContainer = do
    sem <- newQSem concLevel
    let throttledAction = bracket_ (waitQSem sem) (signalQSem sem) . action
    mapConcurrently throttledAction taskContainer

    (adjust imports if you don't use ClassyPrelude)

    Make sure your equivalent of httpLBS consumes the request if you use a streaming alternative. And that your b is strict in all fields.