How to split a large variable?

Question

I'm working with large variables and it can be very slow "looping" through them with while read line, I found out that the smaller the variable the faster it works.

How can I split large variable into smaller variables and then read them one by one?

for example, What I would like to achieve:

bigVar=$(echo "$bigVar" | split_var)

for var in "${bigVar[@]}"; do
  while read line; do
    ...
  done <<< "${var}"
done

or may be split to bigVar1, bigVar2, bigVar3 etc.. and than read them one by one.

Answer 1

Instead of doing

bigVar=$(someCommand)
while read line
do
   ...
done <<< "$bigVar"

Use

while read line
do
   ...
done <   <(someCommand)

This way, you avoid the problem with big variables entirely, and someCommand can output gigabyte after gigabyte with no problem.

If the reason you put it in a variable was to do work in multiple steps on it, rewrite it as a pipeline.