Character encoding with utf-8 breaks all data if just one illegal character is present

Question

I have a strange behaviour when reading data via a REST service from within my app.

I always encode my REST services with utf-8. This has proven to be a safe choice - until now. I am from Denmark and we have a handfuld of special characters, like: æøå.

So with this particular set of data the user has entered a smiley in the app - and the data has been sync'ed to the server. When the data subsequently is sent back to the phone ALL data in the request gets "scrambled". If I try the same request in any other tool (like various browsers and e.g. PostMan) then everything looks Ok (character set is utf-8 and only the smiley cannot be shown). However, in my app all of the non-english characters get scrambled.

Click here to see the url with the smiley - and here to see the url without the smiley (just a different time cutoff). Edit: Links are not active any more....

I have created a small test-app to show the problem from inside Appcelerator:

View:

<Alloy>
    <Window class="container">
        <View>
            <Button id="label1" class="heading" onClick="reload1">Load with smiley</Button>
            <Button id="label2" class="heading" onClick="reload2">Load without smiley</Button>
            <ListView id="nameList" defaultItemTemplate="templateName">
                <Templates>
                    <ItemTemplate name="templateName">
                        <Label bindId="name"/>
                    </ItemTemplate>
                </Templates>
                <ListSection>
                    <ListItem/>
                </ListSection>
            </ListView>
        </View>
    </Window>
</Alloy>

Style:

".container": {
    top: 20,
    backgroundColor:"white",
    orientationModes: [Ti.UI.PORTRAIT]
}
"Label": {
    width: Ti.UI.SIZE,
    height: Ti.UI.SIZE,
    backgroundColor: 'transparent',
    left:10, 
    color: "#000"
}

".heading": { top:15, 
    font: {
        fontSize: '18dp',
        fontStyle: 'bold'
    }
}

"#label1":{left: 10}
"#label2":{right: 10}
"#nameList":{
    top:'50dp'
}

Controller:

function reload1(){
    reload('http://url1');
}
function reload2(){
    reload('http://url2');
}

function reload(url){
    var list = [];
    $.nameList.sections[0].items = [];

    var xhr = Ti.Network.createHTTPClient({
                timeout : 20000 
              });

    var name = 'speciesName';
    xhr.open('GET', url);
    xhr.onload = function(e) {
        var responseJSON = {};
        Ti.API.info("Response headers: " + this.getAllResponseHeaders());
        try {
            responseJSON = JSON.parse(this.responseText);
            if(responseJSON.data){
                _.each(responseJSON.data,function(rec){
                    if(rec[name] && rec[name] != ''){
                        var item = {template: "templateName",name : { text: rec[name] }};
                        list.push(item);
                    }
            });
            $.nameList.sections[0].items = list;
            }
        } catch (e) {
            Ti.API.error('[REST API] apiCall PARSE ERROR: ' + e.message);
            Ti.API.error('[REST API] apiCall PARSE ERROR: ' + this.responseText);
            status = false;
            error = e.message;
        }
    };
     // function called when an error occurs, including a timeout
    xhr.onerror = function(e) {
         Ti.API.debug(e.error);
         alert('error');
    };
    xhr.send();
}

$.index.open();
reload1();

When you launch the app it will show the data with "scrambled" characters. Then you can switch between the data set with/without the smiley using the buttons at the top.

The problem is consistent on iOS and Android. I am running Ti SDK 5.1.2 - I do not dare to upgrade until I have submitted an update to my "real" app shortly :-)

Ideally, I would also want to be able to show the smiley on the mobile device. However, I can live with it not showing correctly - as long as it does not spoil the entire dataset.

Any thoughts/ideas/insight is greatly appreciated :-)

/John

Answer 1

Ok. This turned out to be a problem that was not related to Titanium/Appcelerator - apart from the entire request being broken if just one character didn't fit into the character set.

The backend server is an IBM XWork server and it turned out the REST service used a ResponseWriter (Java) that sends characters. Changing this to return at DataOutputStream (Java) that sends a byte stream instead solved the issue.

If anyone else run into similar problems then I have written a blog article about this where you can learn more. The solution is not specific for IBM XWork - but applies to any backend using Java :-)

/John