Using ruby SAX parsers for GB2312 encoded xml

Question

Good day,

I have a lot of big xml files that i need to parse, but problems is they have 'gb2312' encoding. I would normaly use SAX parser for this.

So here is in example of xml:

<?xml version="1.0" encoding="gb2312"?>
<Root>
<ValueList Count="112290" FieldCount="11">
  <Item1 Value1="23743" Value2="Дипломатия � Пустой кувшин" Value3="1" Value4="" Value5="6"   Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
  <Item2 Value1="6611" Value2="ДЛ � 018 омела � золотой кинжал" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
  <Item3 Value1="6608" Value2="Наука (ДЛ)�круг фей 021�тяпка" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
  <Item4 Value1="6612" Value2="Знаки ДЛ � 003руны � разрушение" Value3="1" Value4="" Value5="6" Value6="0" Value7="0" Value8="0" Value9="0" Value10="0" Value11="0"/>
....
</Root>

I'm trying to use Nokogiri SAX (also tried libxml-ruby with same result) parser:

require 'nokogiri'

class SchemaParser < Nokogiri::XML::SAX::Document
  def initialize
    @cnt = 0
  end
  def start_element name, attrs =[]
    if name == "Item1"
      @cnt+= 1
      puts @cnt
    end
  end
end

parser = Nokogiri::XML::SAX::Parser.new(SchemaParser.new)
parser.parse_io(File.open('2_4_EQUIPMENT_ESSENCE.xml'), 'gb2312')

But this gives error "`check_encoding': 'GB2312' is not a valid encoding (ArgumentError)". If I remove encoding declaration and let Nokogiri detect encoding himself, I will receive this error:

encoding error : input conversion failed due to input error, bytes 0xA8 0x43 0x20 0xA7
encoding error : input conversion failed due to input error, bytes 0xA8 0x43 0x20 0xA7
I/O error : encoder error  

I also tried to open File with proper encoding, but that didn't help SAX parser:

[3] pry(main)> f = File.open('2_4_EQUIPMENT_ESSENCE.xml', "r:gb2312")
=> #<File:2_4_EQUIPMENT_ESSENCE.xml>
[4] pry(main)> f.external_encoding.name
=> "GB2312"

Did anyone use 'gb2312' encoding with SAX parsers in ruby? Any recommendations how to proceed?

Answer 1

So, here is my workaround.

Problems:

1. Some of characters presented in xml are not 'gb2312' encoding, I have found that 'GB18030' would be a better choice with full Chinese characters.
2. I converted all xml's to utf8, so i can use SAX parser.

I ended up with this rake task:

desc "convert chinese xml files to utf-8"
task :convert do
  rm_rf 'data/utf8'
  mkdir 'data/utf8'
  Dir.foreach('data') {|f|
    if f.end_with?('.xml')
      puts "converted:: data/utf8/#{f}" if system("iconv -f GB18030 -t UTF-8 data/#{f} > data/utf8/#{f}")
    end
  }
  #replace encodings for xml files
  system("bundle exec ruby -pi -e \"gsub(/gb2312/, 'UTF-8')\" data/utf8/*.xml")
end