How to stream from an input file into a conduit that has a state

Question

I want to use conduit for reading from a file and then stream it into a tokenizer. The culprit is, that even the same sequence of chars might produce different tokens depending on previous input.

So how can I handle State in a single Conduit inbetween a bunch of IO ones?

Just writing Conduit Char (State TokenizerStateType) Token does yield several errors of wrong types (IO wanted, State got type of errors).

Since I am not at my own computer I can't provide an not-working-minimal-example, but I will add one as soon as I have ghc available.

Answer 1

I'd recommend using a Conduit Char (StateT TokenizerStateType IO), and using liftIO as necessary to perform IO actions.