I have a recursive function which index a textile and if I use it for huge textfiles, i'll get a stack space overflow. I thought because I put the recursive part in the let part I could avoid this stack space overflow, but I'm still getting it. What would be the best way to avoid a stack space overflow with this function?
--lines to Map
parseLinesToWordEntryMap :: Int -> [String] -> M.Map Word [TextLocation] -> (M.Map Word [TextLocation])
parseLinesToWordEntryMap lineNumber [] wordEntryMap = wordEntryMap
parseLinesToWordEntryMap lineNumber (x:xs) wordEntryMap =
let
lineNumber' = lineNumber-1
wordEntryMapRec = parseLinesToWordEntryMap lineNumber' xs wordEntryMap
in
parseLineToWordEntryMap lineNumber x wordEntryMapRec
What you have is essentially a right fold,
parseLinesToWordEntryMap lineNumber xs wordEntryMap
= foldr update wordEntryMap (zip [lineNumber, lineNumber - 1 .. ] xs)
where
update (num,x) wordMap = parseLineToWordEntryMap num x wordMap
thus if parseLineToWordEntryMap
is strict in its Map
argument (fairly typical for Map
arguments), nothing can be done before the end of the list is reached, and then the result is built going back to the beginning of the list.
If at all possible, you should consume the input the other way round, with a left fold, and make sure that the folded function has the right strictness, so that no large thunks are built.
For more specific suggestions, we'd need to see more of the code.