Tail a log file and send rows to curl in 100 line batches

Question

I have a bash script that looks like this:

tail -f -n +1 my.log | \
  awk -f influx.awk | \
  xargs \
  -I '{}' \
  curl \
  -XPOST 'http://influxdb/write?db=foo' \
  --data-binary '{}'

What can I change so that instead of creating a curl request for each row, it would batch them up into say 100 rows (see influx curl docs)?

The problem I'm having is that each InfluxDB "point" needs to be separated by a new line, which is also the delimiter for xargs e.g. adding -L 100 to xargs doesn't work.

Bonus: how would I also make this terminate if no new lines has been added to the file after say 10s?

Answer 1

Rather than xargs, you want to use split, with its --filter option. For example, the following batches lines into groups of two:

$ seq 5 | split -l 2 --filter='echo begin; cat; echo end'
begin
1
2
end
begin
3
4
end
begin
5
end

In your case, you could try something like

tail -f -n +1 my.log | \
  awk -f influx.awk | \
  split -l 100 --filter='\
    curl \
      -XPOST "http://influxdb/write?db=foo" \
      --data-binary @-'

The @- makes curl read data from standard input.