Search code examples
elixirtaskagent

Elixir: what is right way for fetching external API in separate processes

I want to fetch data from external API: it's a similar request when change only one param (page). I would to get it parallel processed but gather data in one place. What is the right way for done it in Elixir? I've made some investigation and have only one solution for now: use Agent for save data from API calls and Task for a call API.

Solution

  • Task.async_stream might fit your needs.

    (1..10) 
|> Task.async_stream(fn n -> get_page(n) end, max_concurrency: 10, timeout: 30_000, ordered: true)
|> Stream.map(fn {:ok, data} -> data end)
|> Enum.to_list()