Downloading a Very Large file silently dies

Issue

So I'm trying to download a very large file through Retrofit2. Files can range up to 5-10 GB. I'm launching an asynchronous structure from an activity/fragment (I've tried both AsyncTask and IntentService) and streaming the file and writing the bytes to a file on the internal storage. I'm publishing progress of the filewrite after each buffer read.

Files up to 150 MB or so work fine, but when I try a 5 GB file the stream silently dies after about 1 GB. There are no logs or logcat that I can see, no exceptions are thrown just poof.

Does anyone have an idea on what's happening, or perhaps I wrote something wrong?

public interface IFileshareDownload {
    @Streaming
    @GET("File/Download/{guid}")
    Call<ResponseBody> downloadFileByGuid(@Path("guid") String guid);
}

public class FileshareDownloadService extends IntentService {
    private static final String TAG = "[FileDownloadService]";
    private static final String PATH = "/ftp/";
    private static final int FILE_CHUNK_SIZE = 2 * 1024 * 1024; //2MB buffer

    private String mFilename;
    private String mFileshareDirectory;
    private String mAbsoluteFilePath;

    private String mBaseUrl;
    private String mGuid;
    private Long mFileSize;
    private Retrofit mRetrofit;
    private IFileshareDownload mDownloader;
    private ResultReceiver mReceiver;

    public FileshareDownloadService() {
        super("FileshareDownload");
    }

    @Override
    protected void onHandleIntent(@Nullable Intent intent) {
        this.mBaseUrl = intent.getStringExtra("baseUrl");
        this.mGuid = intent.getStringExtra("guid");
        this.mFilename = intent.getStringExtra("fileName");
        this.mFileSize = intent.getLongExtra("fileSize", -1);
        this.mFileshareDirectory = getApplicationContext().getFilesDir().getAbsolutePath() + PATH;
        this.mAbsoluteFilePath = mFileshareDirectory + mFilename;

        this.mRetrofit = new Retrofit.Builder()
                .baseUrl(mBaseUrl)
                .callbackExecutor(Executors.newSingleThreadExecutor())
                .build();

        this.mDownloader = mRetrofit.create(IFileshareDownload.class);
        this.mReceiver = intent.getParcelableExtra("listener");

        downloadFile();
    }

    public void downloadFile() {
        Call<ResponseBody> call = mDownloader.downloadFileByGuid(mGuid);
        try {
            Response<ResponseBody> response = call.execute();
            if(response.isSuccessful()) {
                File file = new File(mAbsoluteFilePath);
                file.createNewFile();

                try(FileOutputStream fos = new FileOutputStream(file)) {
                    InputStream is = response.body().byteStream();
                    setUpdateProgress(SHOW_PROGRESS);
                    int count = 0;
                    long bytesRead = 0;

                    byte[] buffer = new byte[FILE_CHUNK_SIZE];
                    try {
                        while ((count = is.read(buffer)) != -1) {

                            fos.write(buffer, 0, count);
                            fos.flush();
                            bytesRead += count;

                            int progress = getPercent(bytesRead, mFileSize);
                            Log.d(TAG, String.format("Read %d out of %d bytes.", bytesRead, mFileSize));
                            setUpdateProgress(UPDATE_PROGRESS, progress);
                        }
                    } catch (Throwable t)
                    {
                        Log.e(TAG, "What happened?", t);
                    }
                }
                setUpdateProgress(HIDE_PROGRESS);
            } else {
                setUpdateProgress(HIDE_PROGRESS);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Solution

Here is a great tutorial about downloading files. It probably mentions what you need:

https://futurestud.io/tutorials/retrofit-2-how-to-download-files-from-server

If it doesn't, take a look at the @Multipart annotation, as well as @Part. I'm not sure if you can use it with GET, but since you have no answers yet, I'll just post it here so you can take the shot if you want.

This is an example from a project I had, in which we create a multipart body to upload an image. I know you want a GET, but the example should still be relevant:

// Setting up the multipart file
File newAvatar = new File(getRealPathFromURI(avatarUri)); // the new avatar
RequestBody filePart = RequestBody.create(MediaType.parse(getActivity().getContentResolver().getType(avatarUri)), newAvatar);

And your request (a POST in this example) should be something like this:

@Multipart
@POST("/api/v1/me/account/upload-cover") // upload avatar
Call<ResponseChangeAvatar> sendChangeAvatarRequest(@Part MultipartBody.Part file, @Header("Authorization") String token);

The retrofit documentation (just search for multipart):

http://square.github.io/retrofit/

A tutorial, in which he creates a multipart body to upload a file to a server:

https://futurestud.io/tutorials/retrofit-2-how-to-upload-files-to-server

Hope this helps. Let me know if you found the solution.

Answered By - Luís Henriques