How to replace line in file with huge file content using shell

I need to replace line in file with content of another huge file. Is that really possible?

I tried

DATA=$(cat $logName)
gawk -i inplace -v r="$DATA" '{gsub("'"$line_to_replace"'",r)}1' file.txt

That was okay for relatively small files (up to 1581 lines), but the error Argument list too long occured for the file with ~7000 lines

Which command should I use instead?

p.s. I also tried sed "s/$line_to_replace/ $(<$DATA)/", but it didn't work at all. I got unterminated s' command

Solution

Reading the huge file into a Bash variable is brittle and a waste of memory. sed already knows how to read another file.

sed -e "/$line_to_replace/!b" -e "r $logName" -e d

If $line_to_replace is not a regular expression, you will probably want to escape any regex special characters in the string. Also, if it contains a slash, use a different regex delimiter, or escape the slash.

If the intent is not to completely replace the entire line, you will need a slightly more complex script.

In very brief,

/.../!b says to skip the rest of the script if the current line doesn't match the regex.
r $logName says to read the contents of the file pointed to by the variable $logName
d removes the line we read as input. (If you leave the line, it will be left before the contents of the file.)

sed "s/foo/$bar/" breaks when $bar contains newlines, or literal slashes. You could escape all newlines and slashes in the data, too; but again, there should be no need in this scenario.

Interpolating shell variables with arbitrary contents into a sed script (or an Awk script, or a Python script, for that matter) is inherently brittle, but it can be done once you understand how this works.