HttpResponseCache doesn`t clear old files?

Question

I will soon work on a project, which uses a lot of HTTPRequests for mainly JSONs and Images, so I thought it is a good idea to think about caching. Basically I'm looking for a solution for

1. Start a HTTPRequest with a given lifetime (f.e. 3,6,12 hours)
2. Check, if that Request is available in the Cache and still valid (lifetime)
3. If Request is still valid, take it from Cache, otherwise make the Request and save its Response

I found HttpResponseCache class in Android. It is working, however it is not working like I'm expecting.

My test case is an AsyncTask to cache several Images. Code looks like the following:

URL url = new URL(link);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();

Bitmap myBitmap;
try {
    connection.addRequestProperty("Cache-Control","only-if-cached");

    //check if Request is in cache      
    InputStream cached = connection.getInputStream();

    //set image if in cache
    myBitmap = BitmapFactory.decodeStream(cached);

} catch (FileNotFoundException e) {
    HttpURLConnection connection2 = (HttpURLConnection) url.openConnection();
    connection2.setDoInput(true);
    connection2.addRequestProperty("Cache-Control", "max-stale=" + 60); 
    connection2.connect();

    InputStream input = connection2.getInputStream();
    myBitmap = BitmapFactory.decodeStream(input);

}

return myBitmap;

} catch (IOException e) {
    e.printStackTrace();        
}

Two questions:

1. I set max-stale=60seconds for testing purposes. However, if I call the same URL 5minutes later, it tells me, it is loading the image from cache. I would assume, that the image is reloaded because the HTTPRequest in the cache is out of date? Or do I have to clean the cache myself?
2. In my catch block, I have to create a second HttpURLConnection, because I cannot add properties, after I opened an URLConnection (this happens in the connection.getInputStream()?!). Is this bad programming?

After all, I find that HttpResponseCache poorly documented. I came across Volley: Fast Networking, but this seems even less documented, even if it is offering exactly the things I need (Request queuing and prioritization...). What do you use for caching? Any links to libraries, tutorials, are welcome.

UPDATE I'm not targeting Android versions lower than 4.0 (still maybe intresting for other users?)

Answer 1

Both HttpResponseCache and volley are poorly documented. However, I have found that you can very easily extend and tweak volley. If you explore source code of volley, especially of: CacheEntry, CacheDispatcher and HttpHeaderParser, you can see how it is implemented.

A CacheEntry holds serverDate, etag, ttl and sofTtl which can represent cache state pretty well, also it has isExpired() and refreshNeeded() methods as convenience.

CacheDispatcher is implemented accurately as well:

// Attempt to retrieve this item from cache.
Cache.Entry entry = mCache.get(request.getCacheKey());

if (entry == null) {
    request.addMarker("cache-miss");
    // Cache miss; send off to the network dispatcher.
    mNetworkQueue.put(request);
    continue;
}

// If it is completely expired, just send it to the network.
if (entry.isExpired()) {
    request.addMarker("cache-hit-expired");
    request.setCacheEntry(entry);
    mNetworkQueue.put(request);
    continue;
}

// We have a cache hit; parse its data for delivery back to the request.
request.addMarker("cache-hit");
Response<?> response = request.parseNetworkResponse(
        new NetworkResponse(entry.data, entry.responseHeaders));
request.addMarker("cache-hit-parsed");

if (!entry.refreshNeeded()) {
    // Completely unexpired cache hit. Just deliver the response.
    mDelivery.postResponse(request, response);
} else {
    // Soft-expired cache hit. We can deliver the cached response,
    // but we need to also send the request to the network for
    // refreshing.
    request.addMarker("cache-hit-refresh-needed");
    request.setCacheEntry(entry);

    // Mark the response as intermediate.
    response.intermediate = true;

    // Post the intermediate response back to the user and have
    // the delivery then forward the request along to the network.
    mDelivery.postResponse(request, response, new Runnable() {
        @Override
        public void run() {
            try {
                mNetworkQueue.put(request);
            } catch (InterruptedException e) {
                // Not much we can do about this.
            }
        }
    });
}

One interesting tidbit: If cache is "soft expired", volley will deliver data from local cache immediately, and re-deliver it from server again after some time, for single request.

Finally, HttpHeaderParser does its best to cope to server headers:

headerValue = headers.get("Date");
if (headerValue != null) {
    serverDate = parseDateAsEpoch(headerValue);
}

headerValue = headers.get("Cache-Control");
if (headerValue != null) {
    hasCacheControl = true;
    String[] tokens = headerValue.split(",");
    for (int i = 0; i < tokens.length; i++) {
        String token = tokens[i].trim();
        if (token.equals("no-cache") || token.equals("no-store")) {
            return null;
        } else if (token.startsWith("max-age=")) {
            try {
                maxAge = Long.parseLong(token.substring(8));
            } catch (Exception e) {
            }
        } else if (token.equals("must-revalidate") || token.equals("proxy-revalidate")) {
            maxAge = 0;
        }
    }
}

headerValue = headers.get("Expires");
if (headerValue != null) {
    serverExpires = parseDateAsEpoch(headerValue);
}

serverEtag = headers.get("ETag");

// Cache-Control takes precedence over an Expires header, even if both exist and Expires
// is more restrictive.
if (hasCacheControl) {
    softExpire = now + maxAge * 1000;
} else if (serverDate > 0 && serverExpires >= serverDate) {
    // Default semantic for Expire header in HTTP specification is softExpire.
    softExpire = now + (serverExpires - serverDate);
}

Cache.Entry entry = new Cache.Entry();
entry.data = response.data;
entry.etag = serverEtag;
entry.softTtl = softExpire;
entry.ttl = entry.softTtl;
entry.serverDate = serverDate;
entry.responseHeaders = headers;

So, ensure the server sends proper headers as well as honors etag,time-stamp and cache control headers.

Finally, you can override getCacheEntry() of Request class to return custom CacheEntry make cache behave exactly according to your needs.