I am currently using Python imaplib to process email text.
I use fetch command to fetch the raw data email from GMail server. However, I found one thing really tricky - the equal sign '='. It is not a normal equal sign but a special symbol.
For example:
'=' sometimes acts as the hyphenation mark at the end of text line:
Depending upon your module selections, course lecturers may also contact yo=
u with preparatory work over the next few weeks. It would be wise to start =
reviewing the preparatory reading lists provided on the module syllabi now =
Sometimes, it acts as a escape mark similar to '%', for example:
a=20b is actually a<SPACE>b
=46rom here is actually From here
I am totally confused about such weird notation. I think there must be a guidance to handle this because GMail can handle such thing correctly in their apps.
I see that this is related to HTML encoding, just like '%' will be encoded. But the problem is, all I get from the IMAP response is a string that contain this '=' symbol. How should I handle this? Using regular expression?
In a nutshell, an equal sign at the end of a line indicates a soft line break. An equal sign followed by two hexadecimal characters (0-9, A-F) encodes a single octet (byte).
This encoding scheme is called "quoted printable" and is defined in section 6.7 of RFC 2045. See items (1) and (5), in particular.
6.7. Quoted-Printable Content-Transfer-Encoding
The Quoted-Printable encoding is intended to represent data that
largely consists of octets that correspond to printable characters in
the US-ASCII character set. It encodes the data in such a way that
the resulting octets are unlikely to be modified by mail transport.
If the data being encoded are mostly US-ASCII text, the encoded form
of the data remains largely recognizable by humans. A body which is
entirely US-ASCII may also be encoded in Quoted-Printable to ensure
the integrity of the data should the message pass through a
character-translating, and/or line-wrapping gateway.
In this encoding, octets are to be represented as determined by the
following rules:
(1) (General 8bit representation) Any octet, except a CR or
LF that is part of a CRLF line break of the canonical
(standard) form of the data being encoded, may be
represented by an "=" followed by a two digit
hexadecimal representation of the octet's value. The
digits of the hexadecimal alphabet, for this purpose,
are "0123456789ABCDEF". Uppercase letters must be
used; lowercase letters are not allowed. Thus, for
example, the decimal value 12 (US-ASCII form feed) can
be represented by "=0C", and the decimal value 61 (US-
ASCII EQUAL SIGN) can be represented by "=3D". This
rule must be followed except when the following rules
allow an alternative encoding.
(2) (Literal representation) Octets with decimal values of
33 through 60 inclusive, and 62 through 126, inclusive,
MAY be represented as the US-ASCII characters which
correspond to those octets (EXCLAMATION POINT through
LESS THAN, and GREATER THAN through TILDE,
respectively).
(3) (White Space) Octets with values of 9 and 32 MAY be
represented as US-ASCII TAB (HT) and SPACE characters,
respectively, but MUST NOT be so represented at the end
of an encoded line. Any TAB (HT) or SPACE characters
on an encoded line MUST thus be followed on that line
by a printable character. In particular, an "=" at the
end of an encoded line, indicating a soft line break
(see rule #5) may follow one or more TAB (HT) or SPACE
characters. It follows that an octet with decimal
value 9 or 32 appearing at the end of an encoded line
must be represented according to Rule #1. This rule is
necessary because some MTAs (Message Transport Agents,
programs which transport messages from one user to
another, or perform a portion of such transfers) are
known to pad lines of text with SPACEs, and others are
known to remove "white space" characters from the end
of a line. Therefore, when decoding a Quoted-Printable
body, any trailing white space on a line must be
deleted, as it will necessarily have been added by
intermediate transport agents.
(4) (Line Breaks) A line break in a text body, represented
as a CRLF sequence in the text canonical form, must be
represented by a (RFC 822) line break, which is also a
CRLF sequence, in the Quoted-Printable encoding. Since
the canonical representation of media types other than
text do not generally include the representation of
line breaks as CRLF sequences, no hard line breaks
(i.e. line breaks that are intended to be meaningful
and to be displayed to the user) can occur in the
quoted-printable encoding of such types. Sequences
like "=0D", "=0A", "=0A=0D" and "=0D=0A" will routinely
appear in non-text data represented in quoted-
printable, of course.
Note that many implementations may elect to encode the
local representation of various content types directly
rather than converting to canonical form first,
encoding, and then converting back to local
representation. In particular, this may apply to plain
text material on systems that use newline conventions
other than a CRLF terminator sequence. Such an
implementation optimization is permissible, but only
when the combined canonicalization-encoding step is
equivalent to performing the three steps separately.
(5) (Soft Line Breaks) The Quoted-Printable encoding
REQUIRES that encoded lines be no more than 76
characters long. If longer lines are to be encoded
with the Quoted-Printable encoding, "soft" line breaks
must be used. An equal sign as the last character on a
encoded line indicates such a non-significant ("soft")
line break in the encoded text.
The email has to be decoded to remove the special chars. You should use the quopri module to deal with it.