Saving text from website using Firefox extension, wrong characters saved

Question

Sorry about the vague title but I'm a bit lost so it's hard to be specific. I've started playing around with Firefox extensions using the add-on SDK. What I'm trying to to is to watch a page for changes, a Twitch.tv chat window in this case, and save those changes to a file.

I've gotten this to work, every time something changes on the page it gets saved. But, "unusual" characters like for example something in Korean doesn't get saved properly. I think this has to do with encoding of the file/string? I tried saving the same characters by copy-pasting them into notepad, it asked me to save in Unicode and when I did everything worked fine. So I figured, ok, I'll change the encoding of the log file to unicode as well before writing to it. Didn't exactly work... Now all the characters were in some kind of foreign language.

The code I'm using to write to the file is this:

    var {Cc, Ci, Cu} = require("chrome");
    var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm");
    var file = FileUtils.getFile("Desk", ["mylogfile.txt"]);
    var stream = FileUtils.openFileOutputStream(file, FileUtils.MODE_WRONLY | FileUtils.MODE_CREATE | FileUtils.MODE_APPEND);
    stream.write(data, data.length);
    stream.close();

I looked at the description of FileUtils.jsm over at MDN and as far as I can tell there's no way to tell it which encoding I want to use?

If you don't know a fix could you give me some good search terms because I seem to be coming up short on that front. Since I know basically nothing on the subject I'm flailing around in the dark a bit at the moment.

edit: This is what I ended up with (for now) to get this thing working:

var {Cc, Ci, Cu} = require("chrome");
var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm");

var file = Cc['@mozilla.org/file/local;1']
           .createInstance(Ci.nsILocalFile);

file.initWithPath('C:\\temp\\temp.txt');
if(!file.exists()){
  file.create(file.NORMAL_FILE_TYPE, 0666);
}

var charset = 'UTF-8';
var fileStream = Cc['@mozilla.org/network/file-output-stream;1']
.createInstance(Ci.nsIFileOutputStream);
fileStream.init(file, FileUtils.MODE_WRONLY | FileUtils.MODE_CREATE | FileUtils.MODE_APPEND, 0x200, false);

var converterStream = Cc['@mozilla.org/intl/converter-output-stream;1']
.createInstance(Ci.nsIConverterOutputStream);

converterStream.init(fileStream, charset, data.length,
Ci.nsIConverterInputStream.DEFAULT_REPLACEMENT_CHARACTER);
converterStream.writeString(data);
converterStream.close();
fileStream.close();

Answer 1

Dumping just the raw bytes (well, raw jschars actually) won't work. You need to first convert the data into some sensible encoding.

See e.g. the File I/O Snippets. Here are the crucial bits of creating a converter output stream wrapper:

var converter = Components.classes["@mozilla.org/intl/converter-output-stream;1"].
                createInstance(Components.interfaces.nsIConverterOutputStream);
converter.init(foStream, "UTF-8", 0, 0);
converter.writeString(data);
converter.close(); // this closes foStream

Another way is to use OS.File + TextConverter:

let encoder = new TextEncoder();                                   // This encoder can be reused for several writes
let array = encoder.encode("This is some text");                   // Convert the text to an array
let promise = OS.File.writeAtomic("file.txt", array,               // Write the array atomically to "file.txt", using as temporary
    {tmpPath: "file.txt.tmp"});                                    // buffer "file.txt.tmp".

It might be even possible to mix both. OS.File has the benefit that it will write data and access files off the main thread (so it won't block the UI while the file is being written).