Most of the ASCII codes under \x20 appear to be entirely obsolete. Are they used at all today? Can they be considered "up for grabs", or is it best to avoid them?
I need a delimiter for grouping "lines" together and it would sure be nice to co-opt one of these for that purpose.
From man ascii:
Oct Dec Hex Char
----------------------------------------------
000 0 00 NUL '\0'
001 1 01 SOH (start of heading)
002 2 02 STX (start of text)
003 3 03 ETX (end of text)
004 4 04 EOT (end of transmission)
005 5 05 ENQ (enquiry)
006 6 06 ACK (acknowledge)
007 7 07 BEL '\a' (bell)
010 8 08 BS '\b' (backspace)
011 9 09 HT '\t' (horizontal tab)
012 10 0A LF '\n' (new line)
013 11 0B VT '\v' (vertical tab)
014 12 0C FF '\f' (form feed)
015 13 0D CR '\r' (carriage ret)
016 14 0E SO (shift out)
017 15 0F SI (shift in)
020 16 10 DLE (data link escape)
021 17 11 DC1 (device control 1)
022 18 12 DC2 (device control 2)
023 19 13 DC3 (device control 3)
024 20 14 DC4 (device control 4)
025 21 15 NAK (negative ack.)
026 22 16 SYN (synchronous idle)
027 23 17 ETB (end of trans. blk)
030 24 18 CAN (cancel)
031 25 19 EM (end of medium)
032 26 1A SUB (substitute)
033 27 1B ESC (escape)
034 28 1C FS (file separator)
035 29 1D GS (group separator)
036 30 1E RS (record separator)
037 31 1F US (unit separator)
040 32 20 SPACE
First the easy part: There are no network transmission concerns in most modern systems. Current protocols handle almost any data - whether 7-bit ASCII, 8-bit ASCII, Unicode characters, image data or compiled programs - as binary data. That has not always been the case. Many older systems had issues transferring control codes and other "unprintable" characters and especially problems with 8-bit data. But those days are, fortunately, behind us. The one big exception is if you want to be able to copy/paste data via an HTML form - for that you want to leave out all control codes and other funny stuff.
You can, of course, make the format anything you like. However, some characters are still used pretty frequently:
000 0 00 NUL '\0' - does "nothing" but is hard for some text editors to handle
003 3 03 ETX (end of text) - Control-C - "break" in a lot of systems
007 7 07 BEL '\a' (bell) - Still makes a bell sound.
011 9 09 HT '\t' (horizontal tab) - A lot of text editors and file formats use this to set a fixed number of spaces
012 10 0A LF '\n' (new line) - like it says
015 13 0D CR '\r' (carriage ret) - used instead of, or together with \n on many systems
021 17 11 DC1 (device control 1) - Control-Q - Resume transmission - XON
023 19 13 DC3 (device control 3) - Control-S - Pause transmission - XOFF
033 27 1B ESC (escape) - Used for PCL and other printer control codes and plenty of other things
Everything else is pretty much up for grabs. I would especially avoid NUL and XON/XOFF - they are sometimes hard to enter into a file - and BEL because typing a file with BEL can be noisy.
If you have a truly binary format then you can do anything you want. But if you want to have a mostly-human-readable format then limiting the control codes is a good idea.