ASCII_8BITEncoding::ASCII_8BIT is a special encoding that is usually used for a byte string, not a character string. But as the name insists, its characters in the range of ASCII are considered as ASCII characters. This is useful when you use ASCII-8BIT characters with other ASCII compatible characters.
Source: ruby-doc.org/core-2.6.4
I want to use ASCII_8BIT because I need to encode all characters between 0x00 (0d00) and 0xff (0d255), so ASCII (0-127) plus extended ASCII (128-255). ASCII (the encoding, US-ASCII) is a 7 bits encoding that recognizes only ASCII (the charset) characters (0-127). As the name states I was expecting that ASCII_8BIT will extends it to 8 bits to add support for 128-255.
When I use chr the encoding is automatically set to ASCII_8BIT but when I put I put a char between 128-255 (0x80-0xff) directly in a string and then ask what is the encoding I got UTF-8 instead and if I try to convert it to ASCII_8BIT is get an error.
irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):16
1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
Is there a bug in ruby core? I need to be able to encode everything between 8
The other name of ASCII 8BIT is BINARY because as the previous quote stated it should be able to encode any byte.
irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
Please telling me to use another encoding is not the answer to the question unless it is an encoding that really map all 255 extended ASCII characters.
One notable way in which ISO character sets differ from code pages is that the character positions 128 to 159, corresponding to ASCII control characters with the high-order bit set, are specifically unused and undefined in the ISO standards, though they had often been used for printable characters in proprietary code pages, a breaking of ISO standards that was almost universal. Ref. Extended ASCII- ISO 8859 and proprietary adaptations
Available encodings in ruby:
irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
For comparison python encodings https://docs.python.org/3/library/codecs.html#standard-encodings
By reading Extended ASCII - Multi-byte character encodings it seems that the only true extended ASCII encoding is UTF-8 but is Multi-byte . It seems that no true extended ASCII single byte encoding exists either.
In a byte point of view I could use any 8bits (single byte) encoding as said here Extended ASCII - Usage in computer-readable languages
all ASCII bytes (0x00 to 0x7F) have the same meaning in all variants of extended ASCII,
But the problem is that implementations like ISO-8859-1 specifically undefined some char ranges and so will end in errors.
irb(main):009:0> (0..255).map { |c| c.chr}.join.encode(Encoding::ISO_8859_1)
Traceback (most recent call last):
6: from /usr/bin/irb:23:in `<main>'
5: from /usr/bin/irb:23:in `load'
4: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
3: from (irb):9
2: from (irb):9:in `rescue in irb_binding'
1: from (irb):9:in `encode'
Encoding::UndefinedConversionError ("\x80" to UTF-8 in conversion from ASCII-8BIT to UTF-8 to ISO-8859-1)
I found the string method force_encoding.
irb(main)> a = "\x8f"
=> "\x8F"
irb(main)> a.encoding
=> #<Encoding:UTF-8>
irb(main)> a.encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):42
1: from (irb):42:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main)> a.force_encoding(Encoding::ASCII_8BIT)
=> "\x8F"
irb(main):040:0> a.encoding
=> #<Encoding:ASCII-8BIT>
What is the danger of using force_encoding rather than encode? Is it just that if I'm passing a multi-byte char accidentally it will be converted to multiple single byte chars? So not dangerous if one is assured that all characters passed to the application are in the extended ASCII range (single byte) but unsafe and will cause silent conversion issue if some UTF-8 chars are passed to the application for example.
irb(main):044:0> "\ud087".force_encoding(Encoding::ASCII_8BIT)
=> "\xED\x82\x87"
irb(main):045:0> "\ud087".bytes
=> [237, 130, 135]
What @mu-is-too-short 's answer and @ForeverZer0 comment are suggesting is that I should rather use pack and unpack to deal with raw bytes.
So rather than using an encoding and workarounding with it
pattern = 'A' * 2606 + "\x8F\x35\x4A\x5F" + 'C' * 390
pattern.force_encoding(Encoding::ASCII_8BIT)
I should use bytes directly
pattern = ['A'.ord] * 2606 + [0x8F, 0x35, 0x4A, 0x5F] + ['C'.ord] * 390
pattern = pattern.pack('C*')
Or this easier to read syntax
pattern = 'A'.bytes * 2606 + "\x8F\x35\x4A\x5F".bytes + 'C'.bytes * 390
pattern = pattern.pack('C*')
String literals are (usually) UTF-8 encoded regardless of whether or not the bytes are valid UTF-8. Hence this:
"\x8f".encoding
saying UTF-8 even though the string isn't valid UTF-8. You should be safe using String#force_encoding but if you really want to work with raw bytes, you might be better of working with arrays of integers and using Array#pack to mash them into strings:
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*')
# "\x8F\x11\x06#\xFF\x00"
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').encoding
# #<Encoding:ASCII-8BIT>
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').bytes
# [143, 17, 6, 35, 255, 0]
The results should be the same but, IMO, this is explicitly working with binary data (i.e. raw bytes), makes your intent clear, and should avoid any encoding problems.
There's also String#unpack if there is a known structure to the bytes you're reading and you want to crack it open.