winmail.dat attachment gets corrupted using ActionMailer in Rails app

Question

I am using ActionMailer in a Ruby on Rails app to read emails (ruby 1.9.3, rails 3.2.13). I have an email that has a winmail.dat file attached to it (ms-tnef) and I am using the tnef gem to extract its contents.

The problem is that when I read the attachment from the mail, it gets corrupted and tnef can not extract files from it.

$ tnef winmail.dat
ERROR: invalid checksum, input file may be corrupted

Extracting the winmail.dat attachment using any mail app, the extracted winmail.dat works fine with tnef and I got it's content.

Comparing the two files I noticed that: - original file is bigger (76k against 72k) - they differ on line breaks: Orginal file has the windows format (0D 0A) and the file saved by rails has the linux format (0A)

I wrote this test:

it 'should extract winmail.dat from email and extract its contents' do
    file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
    message = Mail::Message.new(File.read(file_path))
    anexo = message.attachments[0]
    files = []
    Tnef.unpack(anexo) do |file|
      files << File.basename(file)
    end
    puts files.inspect
    files.size.should == 2
end

That fails with these messages:

WARNING: invalid checksum, input file may be corrupted
Invalid RTF CRC, input file may be corrupted
WARNING: invalid checksum, input file may be corrupted

Assertion failed: ((attr->lvl_type == LVL_MESSAGE) || (attr->lvl_type == LVL_ATTACHMENT)), function attr_read, file attr.c, line 240.

Errno::EPIPE: Broken pipe


anexo = message.attachments[0]
 => #<Mail::Part:2159872060, Multipart: false, Headers: <Content-Type: application/ms-tnef; name="winmail.dat">, <Content-Transfer-Encoding: quoted-printable>, <Content-Disposition: attachment; filename="winmail.dat">>

I tried to save it to disk as bynary, and read it again, but I got the same result

it 'should extract winmail.dat from email and extract its contents' do
    file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
    message = Mail::Message.new(File.read(file_path))
    anexo = message.attachments[0]

    tmpfile_name = "#{::Rails.root}/tmp/#{anexo.filename}"
    File.open(tmpfile_name, 'w+b', 0644) { |f| f.write anexo.body.decoded }
    anexo = File.open(tmpfile_name)

    files = []
    Tnef.unpack(anexo) do |file|
      files << File.basename(file)
    end
    puts files.inspect
    files.size.should == 2
end

How should I read the attachment?

Answer 1

The method anexo.body.decoded calls the decode method of the best suited encoding (Mail::Encodings) for the attachment, in your case quoted_printable.

Some of these encodings (7bit, 8bit and quoted_printable), perform a conversion, changing different types of line breaks to the platform specific line break. the *quoted_printable" call .to_lf that corrupt the winmail.dat file

  # Decode the string from Quoted-Printable. Cope with hard line breaks
  # that were incorrectly encoded as hex instead of literal CRLF.
  def self.decode(str)
    str.gsub(/(?:=0D=0A|=0D|=0A)\r\n/, "\r\n").unpack("M*").first.to_lf
  end

mail/core_extensions/string.rb:

def to_lf
  to_str.gsub(/\n|\r\n|\r/) { "\n" }
end

To solve it you have perform the same encoding without the last .to_lf. To do that you can create a new encoding that does not corrupt your file and use it to encode you attachment.

create the file: lib/encodings/tnef_encoding.rb

require 'mail/encodings/7bit'

module Mail
  module Encodings

    # Encoding to handle Microsoft TNEF format
    # It's pretty similar to quoted_printable, except for the 'to_lf' (decode) and 'to_crlf' (encode)
    class TnefEncoding < SevenBit
      NAME='tnef'

      PRIORITY = 2

      def self.can_encode?(str)
        EightBit.can_encode? str
      end

      def self.decode(str)
        # **difference here** removed '.to_lf'
        str.gsub(/(?:=0D=0A|=0D|=0A)\r\n/, "\r\n").unpack("M*").first
      end

      def self.encode(str)
        # **difference here** removed '.to_crlf'
        [str.to_lf].pack("M")
      end

      def self.cost(str)
        # These bytes probably do not need encoding
        c = str.count("\x9\xA\xD\x20-\x3C\x3E-\x7E")
        # Everything else turns into =XX where XX is a
        # two digit hex number (taking 3 bytes)
        total = (str.bytesize - c)*3 + c
        total.to_f/str.bytesize
      end

      private

      Encodings.register(NAME, self)
    end
  end
end

To use your custom encoding you must, first, register it:

Mail::Encodings.register('tnef', Mail::Encodings::TnefEncoding)

And then, set it as your preferred encoding for the attachment:

anexo.body.encoding('tnef')

Your test would, then, become:

it 'should extract winmail.dat from email and extract its contents' do
    file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
    message = Mail::Message.new(File.read(file_path))
    anexo = message.attachments[0]

    tmpfile_name = "#{::Rails.root}/tmp/#{anexo.filename}"
    Mail::Encodings.register('tnef', Mail::Encodings::TnefEncoding)
    anexo.body.encoding('tnef')
    File.open(tmpfile_name, 'w+b', 0644) { |f| f.write anexo.body.decoded }
    anexo = File.open(tmpfile_name)

    files = []
    Tnef.unpack(anexo) do |file|
        files << File.basename(file)
    end
    puts files.inspect
    files.size.should == 2
end

Hope it helps!