Search code examples
javaandroidmultithreadingperformancethreadpoolexecutor

Android parallel (simultaneous) images download

Guys help please it's important.

I need to download about 2000 images in less than 5 minutes. So I decided to make parallel image downloading.

If someone can suggest me better way to do parallel downloads or told me where I'm going wrong, I'd really appreciate that.

In error log number of completed tasks is always various...

Help please.

I'm using Async task Executor by Artem Zinnatullin 

public class AsyncTaskExecutor {

private static final int CORE_POOL_SIZE;
private static final int MAXIMUM_POOL_SIZE;
private static final int KEEP_ALIVE;
private static final TimeUnit TIME_UNIT;

private static final BlockingQueue<Runnable> concurrentPoolWorkQueue;
private static final ThreadFactory concurrentThreadFactory;
    private static final ThreadPoolExecutor concurrentExecutor;

private AsyncTaskExecutor() {}

static {
 CORE_POOL_SIZE = 5;
 MAXIMUM_POOL_SIZE = 128;
 KEEP_ALIVE = 1;
 TIME_UNIT = TimeUnit.SECONDS;

 concurrentPoolWorkQueue = new LinkedBlockingQueue<Runnable>(10);
 concurrentThreadFactory = new AsyncTaskThreadFactory();
 concurrentExecutor = new ThreadPoolExecutor(CORE_POOL_SIZE,MAXIMUM_POOL_SIZE, KEEP_ALIVE, TIME_UNIT, concurrentPoolWorkQueue, concurrentThreadFactory);
} 
/**
* Concurrently executes AsyncTask on any Android version
* @param task to execute
* @param params for task
* @return executing AsyncTask
*/
@SuppressLint("NewApi")
public static <Params, Progress, Result> AsyncTask<Params, Progress, Result>
 executeConcurrently(AsyncTask<Params, Progress, Result> task,
 Params... params) {
 if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
 task.executeOnExecutor(concurrentExecutor, params);
 } else {
 task.execute(params);
 }
 return task;
}

    /**
* Thread factory for AsyncTaskExecutor
* @author Artem Zinnatullin
*
*/
private static class AsyncTaskThreadFactory implements ThreadFactory {
 private final AtomicInteger count;

 {
 count = new AtomicInteger(1);
 }

@Override
public Thread newThread(Runnable r) {
return new Thread(r, "AsyncTask #" + count.getAndIncrement());
}
}

}

Then I'm using loop to download all my images.

for (PhotoToDBHelper photoToDBHelper : notDownloaded) {
            try {
                photoToDBHelper.download(m_context);
            } catch (Exception e) {
                // TODO: handle exception
            }

        }

where download method look like this:

public void download(Context context) throws Exception {
    HttpClient httpClient = new DefaultHttpClient();
    HttpGet httpGet = new HttpGet(new URL(getPhotosUrl() + m_fileName).toString());
    HttpResponse response = httpClient.execute(httpGet);
    HttpEntity entity = response.getEntity();

    InputStream inputStream = entity.getContent();
    filePath = Environment.getExternalStorageDirectory().toString() + getFoldeRelativePath() + m_fileName;

    Utils.createFileOnExternalStorage(getFoldeRelativePath(), filePath);
    FileOutputStream fos = new FileOutputStream(filePath, false);

    byte[] buffer = new byte[20000];
    int byteRead = 0;

    while ((byteRead = inputStream.read(buffer)) != -1) {
        fos.write(buffer, 0, byteRead);
    }
    fos.close();
}

And when start my android application I am getting this error

02-25 01:08:30.133: E/AndroidRuntime(32357): FATAL EXCEPTION: main
02-25 01:08:30.133: E/AndroidRuntime(32357): java.lang.RuntimeException: Unable to start activity ComponentInfo{"package".TabHostActivity}: java.util.concurrent.RejectedExecutionException: Task android.os.AsyncTask$3@2c0809b0 rejected from java.util.concurrent.ThreadPoolExecutor@2be18f48[Running, pool size = 128, active threads = 1, queued tasks = 10, completed tasks = 27]
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1967)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1992)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.access$600(ActivityThread.java:127)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1158)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.Handler.dispatchMessage(Handler.java:99)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.Looper.loop(Looper.java:137)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.main(ActivityThread.java:4441)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.lang.reflect.Method.invokeNative(Native Method)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.lang.reflect.Method.invoke(Method.java:511)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at dalvik.system.NativeStart.main(Native Method)
02-25 01:08:30.133: E/AndroidRuntime(32357): Caused by: java.util.concurrent.RejectedExecutionException: Task android.os.AsyncTask$3@2c0809b0 rejected from java.util.concurrent.ThreadPoolExecutor@2be18f48[Running, pool size = 128, active threads = 1, queued tasks = 10, completed tasks = 1032]
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1967)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:782)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1303)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.os.AsyncTask.executeOnExecutor(AsyncTask.java:564)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".data.AsyncTaskExecutor.executeConcurrently(AsyncTaskExecutor.java:58)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".TabHostActivity.checkUpdateState(TabHostActivity.java:130)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at "package".TabHostActivity.onCreate(TabHostActivity.java:88)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.Activity.performCreate(Activity.java:4465)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1049)
02-25 01:08:30.133: E/AndroidRuntime(32357):    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1931)
02-25 01:08:30.133: E/AndroidRuntime(32357):    ... 11 more
Solution

  • There are a few problems that need to be addressed:

    1. 2000 images in 5 minutes is about 7 images per second. Depending on the image size, network bandwidth, processor and storage speed, etc... this may not be feasible with ANY code or amount of concurrency. The math for the throughput would be: 7 * N bytes-per-second where N is the average image size. If there are large images it may be a dead point.
    2. The exception indicates the thread-pool has reached the maximum number of queued tasks (10). This may be configurable based on the thread-pool-manager implementation. Overall, attempting to simultaneously start 2000 threads for example, will likely exacerbate this condition. Recommend queuing tasks (which uses threads) in groups, monitoring their completion and then starting more, all in small groups of say five (5) at a time.
    3. If there is any server control, consider using the sprite model where multiple images are stored in one image (file/resource), broken out by their relative x/y coordinates within the image. This will minimize the number of HTTP or whatever network requests from 2000 to 1 (possibly), improving performance significantly.