java servlets utf-8 internationalization globalization

How to read Arabic or Persian (key and values) from java file with ResourceBundle

I am trying to read Arabic and Persian keys & values from java file instead of properties file with ResourceBundle but it displays unknown characters.

What I did is that I Coded 3 files as: 1. LabelsBundle_en_US.java 2. LabelsBundle_ar_AE.java 3. LabelsBundle_fa_IR.java

My LabelsBundle_en_US.java file looks like:

public class LabelsBundle_en_US extends ListResourceBundle{
            static final Object[][] contents = {
                                                {"REGISTER","Registration Form"},
                                                {"USERNAME","Email"},
                                                {"PASSWORD","Password"},
                                                {"CONFIRM_PASS","Confirm Password"},
                                                {"SUBMIT","Register"}
                                                };

            protected Object[][] getContents(){
            return contents;
            }   
    }//class

My LabelsBundle_fa_IR.java file looks like:

public class LabelsBundle_fa_IR extends ListResourceBundle{
        static final Object[][] contents = {
                                            {"REGISTER","ثبت نام"},
                                            {"USERNAME","ایمیل"},
                                            {"PASSWORD","رمز"},
                                            {"CONFIRM_PASS","مرور رمز"},
                                            {"SUBMIT","ارسال"}
                                            };

        protected Object[][] getContents(){
        return contents;
        }   
}//class

Here is my servlet:

protected void doGet(HttpServletRequest req, HttpServletResponse res)throws ServletException, IOException{      
        res.setContentType("text/html; charset=UTF-8");
        req.setCharacterEncoding("UTF-8");
        res.setCharacterEncoding("UTF-8");      
        StringWriter sWriter    = new StringWriter();  
        PrintWriter out         = new PrintWriter(sWriter);         

        String country          = req.getParameter("country");
        String language         = req.getParameter("language");
        Locale locale=null;
        if(country == null){
            locale = new Locale("en","US");
        }
        else{
            locale = new Locale(language, country);
        }       
        ResourceBundle rb = ResourceBundle.getBundle("com.i18n.resource.bundles.LabelsBundle",locale);
        req.setAttribute("resource", rb);

        out.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>"+
                    "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.1//EN\" \"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\">"+
                    "<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"fi\" dir=\"auto\">"+
                    "<head>"+
                    "<meta http-equiv='Content-Type' content='text/html; charset=UTF-8' />"+
                    "<body>"+
                    "<center><h1>"+rb.getString("REGISTER")+"</h1></center>"+
                    "<table border=0 width=540 align=center>"+
                        "<tr><td colspan=2 align=center><h1>"+rb.getString("REGISTER")+"</h1></td></tr>"+
                        "<tr><td>"+rb.getString("USERNAME")+"</td><td><input type=text name=username></td></tr>"+
                        "<tr><td>"+rb.getString("PASSWORD")+"</td><td><input type=password name=password></td></tr>"+
                        "<tr><td>"+rb.getString("CONFIRM_PASS")+"</td><td><input type=password name=cPass></td></tr>"+
                        "<tr><td colspan=2 align=center><input type=submit value="+rb.getString("SUBMIT")+"></td></tr>"+
                    "</table></html>");
        res.getWriter().print(sWriter.toString());
    }//doGet

This code(s) are working fine when language is en_US but when I change it to Arabic or Persian then its displaying as:

ط«ط¨طھ ظ†ط§ظ…

ط§غŒظ…غŒظ„
ط±ظ…ط²
ظ…ط±ظˆط± ط±ظ…ط² ط§ط±ط³ط§ظ„

Please advise

Solution

Personally, I see no good reason to use ListResourceBundle for strings. But hey, whatever ...

The problem here is the encoding of the source .java file.

You might encode the file as utf-8, but the Java compiler does not know that, and uses the system encoding.

a. You can pass the desired encoding to the compiler with -encoding:

javac -encoding utf-8 LabelsBundle_fa_IR.java

b. You can Unicode-escape the strings making the file ASCII. The characters will be unreadable (will be "...\u0646..."), but there is no confusion for the compiler anymore:

native2ascii -encoding utf-8 LabelsBundle_fa_IR.java asciifile
rename asciifile LabelsBundle_fa_IR.java

I assume you don't send out a .java file for localization :-) Most localization tools extract the localizable stuff from the file, translate it, and generate the java back. There is often an option to generate it with unicode-escape, so there is no need to go all the way to .java and convert there.

===

This is a bad answer:

rb.getString("USERNAME").getBytes(), "UTF-8");

getBytes depends on the system encoding, but that is the runtime encoding.

So it is the encoding of you server, vs. the one used to compile your java files to begin with. It might be different.

===

Why I prefer .properties to ListResourceBundle:

localization (by translators, not by techies) does not happen on .java files. Strings have to be extracted, and put back. Since the .java file can contain code and what not, it is easier to damage in the process. Most localization tools handle .properties well, but not ListResourceBundle
the encoding of the .properties file is well defined: Latin 1. There is no "magic" and does not depend on the system. So you don't need special flags at compile time.
ListResourceBundle is nice for locale-specific objects. Think business logic, for instance complicate tax rules, or special workflows. That's why it is a map from String to Object. You can of course put strings there, but .properties or .xml were designed specifically for strings.