How to export Chinese/Korean words to csv

Question

I managed to export English text to a csv file and toimplement localization. Latin letters and words work fine for any language (e.g.: German) but my program cannot export Chinese/Korean words to the csv, instead showing weird characters:

For reference, the English version looks like this:

Here's the code I use to generate the file:

ofstream file(filename);
// file.imbue(locale(file.getloc(), new std::codecvt_utf16<wchar_t, 0x10ffff, std::little_endian>()));
file << outputListWaveform << "\n";

//this is the part get the header of each column
for (int x = 0; x < data.size(); ++x)
{
    file << get<0>(data.at(x));
    if (x != data.size() - 1)
        file << ",";
}
file << "\n";

for (int i = 0; i < get<1>(data.at(0)).size(); ++i)
{
    for (int j = 0; j < data.size(); ++j)
    {
        auto header = get<0>(data.at(j));
        auto dVal = get<1>(data.at(j));
        auto bVal = get<2>(data.at(j));

        file << ((header == FileOpConstants::BOST || header == FileOpConstants::EOST) ? bVal.at(i) : dVal.at(i));
        if (j != data.size() - 1)
            file << ",";
    }
    file << "\n";
}

file.close();

And here are the codes to import the CSV file back

 StepTable data;
    ifstream file(filename);

    if (!file.is_open())
    {
        string errMsg = "Could not open file: " + filename;
        throw runtime_error(errMsg);
    }

    string line, colname;
    if (file.good())
    {
        getline(file, line); // metadata (e.g. "Output List Waveform" as generated using Cyclops)
        getline(file, line); // column header
        stringstream ss(line);
        while (getline(ss, colname, ','))
        {
            data.push_back({colname, vector<double>{}, vector<bool>{}});
        }
    }

    // Get Column Values (row by row)
    while (getline(file, line))
    {
        int i = 0;
        auto val = Utility::split(line, ',');
        for (const auto& v : val)
        {
            auto header = get<0>(data.at(i));
            if (header == FileOpConstants::BOST || header == FileOpConstants::EOST)
            {
                get<2>(data.at(i)).push_back(stoi(v));
            }
            else if(header == VoltageHeader || header == CurrentHeader || header == PowerHeader || header == ResistanceHeader || header == TimeHeader )
            {
                get<1>(data.at(i)).push_back(stod(v));
            }
            else{
                return {};
            }
            i++;
        }
    }

    file.close();
    return data;

I tried this method ( https://www.itg.ias.edu/content/how-import-csv-file-uses-utf-8-character-encoding-0 )on my exported CSV file and it looks like this:

I am able to export and import the Chinese CSV file without any error. But what I would like to view the exported CSV file with the correct Chinese word without any extra steps.

Answer 1

Microsoft products are notorious for using BOM in UTF-8 (which was initially invalid as by the Unicode specs, but due to widespread use in practice, is now allowed, but not recommended).

Excel uses it to determine the encoding of CSVs when you open them (e.g. by double click). If there is no BOM, it uses a locale 8-bit encoding (probably cp1252).

To enforce it to read as UTF-8, write a BOM, like this:

ofstream file(filename);
file << char(0xEF) << char(0xBB) << char(0xBF);

You will have to deal with it when reading your files back