So I am using the guidelines for resumable/multipart upload of files to google drive as per the specifications of the google docs here
Everything goes great I have written this code a long time ago and it had worked for so many files for a long time until just yesterday i tried using the same code to upload an sample file which is not an multiple of 256KB[again specified from google docs]
Here are the related classes
1) URLMaker :- HttpUrlFactory for constant chunk uploads
/*A Utility Class for constantly creating URL connections for each chunk upload as an instance can be used to send only one request */
final class URLMaker
{
static HttpURLConnection openConnection(String session, String method, String[] params, String[] values, boolean input, boolean output)throws IOException
{
HttpURLConnection request = (HttpURLConnection)new URL(session).openConnection();
request.setRequestMethod(method);
request.setDoInput(input);
request.setDoOutput(output);
request.setConnectTimeout(10000);
//params & values array are of equal length so just one->one key:value pass it as headers
for(int i=0;i<params.length;i++){request.setRequestProperty(params[i],values[i]);}
return request;
}
}
2) Session :- Used To Retrive an resumable uri session which i save to an file and read from later on
final class Session
{
private static final File SESSION=new File("E:/Session.txt");
private static String read()
{
try(ObjectInputStream ois=new ObjectInputStream(new FileInputStream(SESSION)))
{
URISession session=(URISession)ois.readObject();
return session.uri;
}
catch(Exception ex){return null;}
}
static String makeSession(String token,File file)throws Exception
{
if(SESSION.exists()){return read();} //If I Previously saved an resumable URI reload that from file
String body = "{\"name\": \"" + file.getName() + "\"}";
String fields[]=new String[5];
fields[0]="Authorization";
fields[1]="X-Upload-Content-Type";
fields[2]="X-Upload-Content-Length";
fields[3]="Content-Type";
fields[4]="Content-Length";
String values[]=new String[5];
values[0]="Bearer "+token;
values[1]="*audio/mp3*"; //generic octet stream
values[2]=String.format(Locale.ENGLISH, "%d",file.length());
values[3]="application/json; charset=UTF-8";
values[4]=String.format(Locale.ENGLISH, "%d", body.getBytes().length);
HttpURLConnection request =URLMaker.openConnection("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable"
,"POST"
,fields,values, true, true);
try(OutputStream outputStream = request.getOutputStream()){outputStream.write(body.getBytes());}
request.connect();
if(request.getResponseCode()==HttpURLConnection.HTTP_OK)
{
String uri=request.getHeaderField("location");
SESSION.createNewFile();
write(uri);
return uri;
}
else
{
System.err.println(request.getResponseCode()+"/"+request.getResponseMessage());
System.out.println(new String(request.getErrorStream().readAllBytes()));
}
return null;
}
private static void write(String uri)
{
try(ObjectOutputStream oos=new ObjectOutputStream(new FileOutputStream(SESSION)))
{
oos.writeObject(new URISession(uri));
}
catch(Exception ex){ex.printStackTrace(System.out);}
}
private static class URISession implements Serializable
{
private final String uri;
private URISession(String uri){this.uri=uri;}
}
}
the class where my problem lies
3)Uploader :- Uploads the data from file to uri
final class Uploader
{
private static final int
KB=1024,
_256KB=256*KB;
public static void uploadFile(String session,File file)throws IOException
{
long uploadPosition,bytesDone,workLeft;
if((uploadPosition=getLastUploadedByte(session,file.length()))==-1){return;} //read the last uploaded byte position[0-length-1 value] if this upload previously failed due to network/server issues
bytesDone=uploadPosition; //if +ve value it means my work is a little less
String
fields[]={"Content-Type","Content-Length","Content-Range"},
values[]=new String[]{"audio/mp3*","",""};
try(RandomAccessFile raFile = new RandomAccessFile(file,"r"))
{
raFile.seek(uploadPosition);
byte buffer[]=new byte[_256KB];
while((workLeft=raFile.read(buffer))!=-1) //read certain amount of work/or bytes to be uploaded
{
int code=0;
while(workLeft>0)// it is possible that google might not have uploaded all my bytes I have sent so I keep trying until what I have read has been completely uploaded
{
values[1]=String.format(Locale.ENGLISH, "%d",workLeft);
values[2]="bytes "+uploadPosition+"-"+(uploadPosition + workLeft - 1)+"/"+file.length();
System.out.println("Specified Range="+values[1]+"/"+values[2]);
HttpURLConnection request=URLMaker.openConnection(session,"PUT",fields,values, true, true);
try(OutputStream bytes=request.getOutputStream()){bytes.write(buffer,0,(int)workLeft);}
request.connect();
code=request.getResponseCode();
if(code==308) //Means successful but more data needs to be uploaded
{
String range = request.getHeaderField("range");
int bytesSent=Integer.parseInt(range.substring(range.lastIndexOf("-") + 1, range.length())) + 1
System.out.println("Sent Range="+range);
workLeft-=bytesSent; //Work Reduced By this many bytes
bytesDone+=bytesSent; //Progress increased by this many bytes
System.out.println("Bytes="+bytesDone+"/"+file.length());
}
else if(code!=200)
{
System.err.println("Upload Error:"+request.getResponseMessage());
System.err.println("Upload Error Msg:"+new String(request.getErrorStream().readAllBytes()));
code=-1;
break;
}
request.disconnect();
}
if(code==-1){return;}
else if(code==200 || code==201){System.out.println("Upload Completed");}
}
}
}
private static long getLastUploadedByte(String session,long fileLength)throws IOException
{
HttpURLConnection request = URLMaker.openConnection(session
,"PUT"
,new String[]{"Content-Length","Content-Range"}
,new String[]{"0","bytes */" + fileLength},true,true);
try(OutputStream output=request.getOutputStream()){output.flush();}
request.connect();
long chunkPosition=-1;
if(request.getResponseCode()==308 || request.getResponseCode()==200)
{
String range = request.getHeaderField("range"); //previously sent byte position[ it's a 0-length-1 value]
if(range==null){return 0;} //suppose it's my first time ever using this uri then there would have been no bytes uploaded and hence no range header
chunkPosition = Long.parseLong(range.substring(range.lastIndexOf("-") + 1, range.length())) + 1;//format[0-value]
}
else
{
System.err.println("Seek Error:"+request.getResponseCode()+"/"+request.getResponseMessage());
System.err.println("Seek Error Msg:"+new String(request.getErrorStream().readAllBytes()));
}
request.disconnect();
return chunkPosition;
}
}
And My Main Class
class FileUpload
{
private static final Credential getCredentials(NetHttpTransport HTTP_TRANSPORT,String authorize,Collection<String> SCOPES,boolean clearTokens)throws IOException,GeneralSecurityException
{
// Load client secrets.
InputStream in = GDrive.class.getResourceAsStream(CREDENTIALS_FILE_PATH);
if (in == null){throw new FileNotFoundException("Resource not found: " + CREDENTIALS_FILE_PATH);}
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));
// Build flow and trigger user authorization request.
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(HTTP_TRANSPORT,JSON_FACTORY,clientSecrets,SCOPES)
.setDataStoreFactory(new FileDataStoreFactory(new File("tokens")))
.setAccessType("offline")
.build();
LocalServerReceiver receiver = new LocalServerReceiver.Builder().setPort(8888).build();
return new AuthorizationCodeInstalledApp(flow,receiver).authorize(authorize);
}
public static Credential makeCredentials(String authorize,Collection<String> SCOPES,boolean clearTokens)throws IOException,GeneralSecurityException
{
NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
return getCredentials(HTTP_TRANSPORT,authorize,SCOPES,clearTokens);
}
public static void main(String args[])throws Exception
{
Credential credentials=makeCredentials("user",List.of(DriveScopes.DRIVE_FILE));//authorize,scopes
File media=new File("E:\\Music\\sample.mp3");
String session=Session.makeSession(credentials.getAccessToken(),media);
System.out.println("Session="+session);
if(session==null){return;}
Uploader.uploadFile(session,media);
}
}
Now here is the problem i have an 2.01 MB (21,13,939 bytes) you can even download the file and test it yourself from here or use any similar file
This is the log file
Specified Range=262144/bytes 0-262143/2113939 Ok so I start of good uploading the first 256KB of data at every step
Sent Range=bytes=0-262143
Bytes=262144/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=524288/2113939
Specified Range=262144/bytes 0-262143/2113939 //No Problems here still an multiple of 256KB
Sent Range=bytes=0-262143
Bytes=786432/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=1048576/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=1310720/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=1572864/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=1835008/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=2097152/2113939
// HERE IS WHERE LOGIC IS THROWN OUT THE WINDOW THE LAST CHUNK //
Specified Range=16787/bytes 0-16786/2113939 <---- I am clearly telling google that this is the last 16786 bytes of data of my file
Sent Range=bytes=0-262143 <---- But google dosen't seem to care and uploads the whole 256KB byte array along with the garbage data from the previous read
Bytes=2359296/2113939 <--- And now I have uploaded more bytes than the actual file itself [format is bytesDone/fileSize]
and the last response code i get before the app exits with no error is 308 I expect to see the message "Upload Completed" but it never does and the File never shows up on my google drive.
As You can already guess this code worked for files <256KB or multiples of it thats why i have never seen any problems with this code until now
I posted all the classes so that anyone can just copy and test it for themself but the main problem i suspect lies in the Uploader class and that's where i need your help.
I finally found your problem, from a piece of your output:
Specified Range=262144/bytes 0-262143/2113939 Ok so I start of good uploading the first 256KB of data at every step
Sent Range=bytes=0-262143
Bytes=262144/2113939
Specified Range=262144/bytes 0-262143/2113939
Sent Range=bytes=0-262143
Bytes=524288/2113939
Specified Range=262144/bytes 0-262143/2113939 //No Problems here still an multiple of 256KB
Sent Range=bytes=0-262143
Bytes=786432/2113939
Specified Range=262144/bytes 0-262143/2113939
Looking at your code the part that actually goes inside Content-Range is the third position of the values array:
values[2]="bytes "+uploadPosition+"-"+(uploadPosition + workLeft - 1)+"/"+file.length();
It seems that you are not updating the variable uploadPoistion so you are always sending the same range over and over. You can see that even from the response you get from drive:
String range = request.getHeaderField("range"); // This is always the same
int bytesSent=Integer.parseInt(range.substring(range.lastIndexOf("-") + 1, range.length())) + 1
System.out.println("Sent Range="+range);
Sent Range=bytes=0-262143
This value repeats in your log over and over, so you are just informing the first 262144 bytes of information.
You should try to increase the Content-Range every time to cover all the bytes of the file:
bytes 0-262143/2113939
bytes 262144-524287/2113939
bytes 524288- ....