Search code examples
haskellhaskell-pipes

Pipes.Binary.decode - what is the StateT for?

I'm trying to write a basic network server using pipes and the assorted libraries that build on it. The intended flow would be:

get bytestring from socket -> decode using binary -> server logic goes here -> send response to socket

Which I figured would be something like:

fromSocket s 4096 >-> decode >-> serverLogic >-> toSocket s

pipes-binary has a decode and a decodeMany, but I am not sure I understand the difference, and I don't know how to use decode. Why does decodeMany take the upstream pipe as an argument instead of being chained off of it with >->? And how do you use decode, what is the StateT for and what should my pipe chain end up looking like?

Solution

  • The StateT (Producer a m r) m x idiom comes from pipes-parse's "Low-level Parsers". It typically means that the library is using draw and unDraw to pull values off a Producer and return them if they're unused. It's an essential component of parsing where failure might occur. It also requires the StateT layer to indicate that a pipe is being selectively drained and refilled in a stateful manner.

    -- | Draw one element from the underlying Producer, 
-- returning Left if the Producer is empty
draw :: Monad m => StateT (Producer a m r) m (Either r a)

-- | Push back an element onto the underlying Producer
unDraw :: Monad m => a -> StateT (Producer a m r) m ()

    So what does that mean for decode and decodeMany? If we look at some simplified types of those functions

    -- for (Monad m, Binary b)

decode     :: StateT (Producer ByteString m r) m (Maybe b)
decodeMany :: Producer ByteString m r 
           -> Producer' b m (Either (Producer ByteString m r) r)

    We first see that decode is drawing off enough ByteString chunks from a Producer ByteString statefully so as to try to parse a b. Since the chunk boundary on the ByteStrings may not align with a parse boundary it's important to do this in StateT so that the leftover chunks can be unDraw-ed back into the Producer.

    decodeMany builds atop decode and attempts to repeatedly decode bs off the input Producer returning a "continuation" Producer of leftover ByteStrings on failure.

    Long story short, due to a need to unDraw leftover ByteString chunks, we can just compose these things together into a chain with (>->). If you want to do that, you can use something like decodeMany to transform a producer and then chain the result, but you'll want to handle error cases carefully.