Out of memory while downloading file

Issue

I am trying to download a CSV file in android. Once it is downloaded, I am using Jackson to map and then persist to sqlite3 db. The problem I am facing is that when I make the call that downloads the file, I am getting an OutOfMemoryError. The file size is around 12MB. I am able to download the file when I try the same in a browser, so I don't think its a problem with the endpoint.

To download, I am using an AsyncTask.

class DownloadCSVTask extends AsyncTask<Void, String, Void> {

    private final Call<ResponseBody> csvDump;
    private final Context context;

    public DownloadCSVTask(Call<ResponseBody> csvDump, Context context) {
        this.csvDump = csvDump;
        this.context = context;
    }

    @Override
    protected Void doInBackground(Void... voids) {
        try {
            Response<ResponseBody> response = csvDump.execute();
            File fileFromResponse = createFileFromResponse(response);
            mapFileContentsToModelAndPersist(fileFromResponse);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "CSV pull failed");
        }
        return null;
    }

    private File createFileFromResponse(Response<ResponseBody> response) throws IOException {
        File path = context.getFilesDir();
        File csvFile = File.createTempFile(TEMP_PREFIX, CSV_EXTENSION, path);
        FileOutputStream fileOutputStream = context.openFileOutput(csvFile.getName(), Context.MODE_PRIVATE);
        fileOutputStream.write(response.body().bytes());
        Log.i(getClass().getCanonicalName(), "Writing file completed" + csvFile.getAbsolutePath());
        fileOutputStream.close();
        return csvFile;
    }

    private void mapFileContentsToModelAndPersist(File fileFromResponse) {
        try {
            MappingIterator<City> cityIter = new CsvMapper().readerWithTypedSchemaFor(City.class).readValues(fileFromResponse);
            List<City> cities = cityIter.readAll();
            Stream.of(cities).forEach(City::persist);
        } catch (IOException e) {
            Log.e(getClass().getSimpleName(), "Failed to map contents of file to City Class");
        }
    }
}

The error log that I am getting:

 java.lang.RuntimeException: An error occured while executing doInBackground()
                         E      at android.os.AsyncTask$3.done(AsyncTask.java:300)
                         E      at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
                         E      at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:242)
                         E      at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
                         E      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                         E      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                         E      at java.lang.Thread.run(Thread.java:841)
                         E  Caused by: java.lang.OutOfMemoryError
                         E      at java.lang.String.<init>(String.java:354)
                         E      at java.lang.String.<init>(String.java:393)
                         E      at okio.Buffer.readString(Buffer.java:616)
                         E      at okio.Buffer.readString(Buffer.java:599)
                         E      at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:263)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:42)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at com.someclass.somepath.network.util.NetworkInterceptor.intercept(NetworkInterceptor.java:23)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                         E      at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                         E      at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
                         E      at okhttp3.RealCall.execute(RealCall.java:60)
                         E      at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
                         E      at retrofit2.ExecutorCallAdapterFactory$ExecutorCallbackCall.execute(ExecutorCallAdapterFactory.java:89)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:60)
                         E      at com.someclass.somepath.service.DownloadCSVTask.doInBackground(CsvPullHandler.java:47)
                         E      at android.os.AsyncTask$2.call(AsyncTask.java:288)
                         E      at java.util.concurrent.FutureTask.run(FutureTask.java:237)

Am I doing something wrong here? Is there another way to do it?

Solution

The amount of memory for Java code on Android is limited to something like 16 Mb (varies by device). So you either need to move this part to native NDK code, where you can malloc() as much memory as your device has, or save the file directly to disk, then process it without loading the whole file into Java buffers. Use URLConnection.getInputStream() or URLConnection.read() in a loop with a 16Kb byte array instead of copying the whole file into response array. Or you can read it line-by-line directly from the HTTP stream, using BufferedReader.readLine(), and fill your database line-by-line.

Answered By - pelya