Remove 0xff from the end of a binary flash image using bash script

I have a tool that generates a 32MB binary image to be written to a flash memory. But only the first 2MB contains valuable data, the rest is just 0xff. So I would like to remove the 0xff bytes from the end of the file using a nice bash / makefile script. I could use head:

head -c 2M test.bin > out.bin

But I don't know the actual length, so I would like to find the first occurence of 0xff from the end of the file as input to head or similar.

I'm running my tool from a Makefile on cygwin, so it would be good if it could be executed using standard bash / makefile tools.

Solution

Counting the number of 0xFF bytes at the end of your file can be done with a combination of hexdump (or xxd, od...) to convert your binary file in a stream of ASCII hexadecimal values and a text processor like awk to do the counting. Example:

hexdump -v -e '/1 "%02X\n"' test.bin | \
  awk '/FF/ {n += 1} !/FF/ {n = 0} END {print n}'

Then, removing that number of bytes from the end of the file can be done using e.g. dd or head. Example:

head -c -$d test.bin > results/test.bin

All in all, your Makefile could resemble:

OUTDIR  := results
OLDBINS := $(wildcard *.bin)
NEWBINS := $(addprefix $(OUTDIR)/,$(OLDBINS))

.PHONY: all

all: $(NEWBINS)

$(OUTDIR)/%: % | $(OUTDIR)
    n=$$(hexdump -v -e '/1 "%02X\n"' $< | \
      awk '/FF/ {n += 1} !/FF/ {n = 0} END {print n}'); \
    head -c -$$n $< > $@

$(OUTDIR):
    mkdir -p $@

There are a few make subtleties like phony targets ( all), automatic variables ($<, $@), order-only prerequisites (| $(OUTDIR)), make expansion escaping in recipes (the $$ signs), make functions (wildcard, addprefix), a one-line recipe that uses line continuation (the \ at the end of lines)... But nothing very complicated.

EDIT: try to find a faster solution:

Another option may be more efficient (about 20 times faster on 32MB files in my super-simple tests) if the base64 utility is available and the size of your files is exactly 32MB:

base64-encode without wrapping:
```
 base64 -w0
```
Use sed to remove all trailing //// strings, followed by a last //8= at the end of the file (see the base64 RFC 4648) to understand why these strange trailing texts:
```
 sed -E 's#(////)*//8=$##'
```
base64-decode:
```
 base64 -d
```

One-liner:

base64 -w0 test.bin | sed -E 's#(////)*//8=$##' | base64 -d > results/test.bin

Note that this may leave one or two 0xFF characters at the end on the file, depending on the size of the input file and on the number of trailing 0xFF characters. The new make recipe would be:

$(OUTDIR)/%: % | $(OUTDIR)
    base64 -w0 $< | sed -E 's#(////)*//8=$$##' | base64 -d > $@