403 From Google Drive API When Using API Key

Issue

I've got a Kotlin application that retrieves publicly availably PDFs stored on Google drive. To download the PDFs, I do the following

@Throws(IOException::class)
fun download(url: String?, destination: File?) {
    val connection: URLConnection = URL(url).openConnection()
    connection.setConnectTimeout(60000)
    connection.setReadTimeout(60000)
    connection.addRequestProperty("User-Agent", "Mozilla/5.0")
    val output = FileOutputStream(destination, false)
    val buffer = ByteArray(2048)
    var read: Int
    val input: InputStream = connection.getInputStream()
    while (input.read(buffer).also { read = it } > -1) output.write(buffer, 0, read)
    output.flush()
    output.close()
    input.close()
}

My url is of the form https://www.googleapis.com/drive/v3/files/${fileId}?key=<MY_KEY>&alt=media.

Google seems to be rejecting requests after it serves about 10 requests. I checked the API usage, and it says I get 20,000 requests per 100 seconds (https://developers.google.com/drive/api/guides/limits). I can see my requests on the API usage chart, so the API key is being recognized. I'm using 10-15 requests then getting the 403. It's not coming back as json, so here is the detailed message:

We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
See Google Help for more information.

I assume I'm missing something obvious. In that HTML blob, it says but your computer or network may be sending automated queries. To protect our users, we can't process your request right now., which is obviously what I'm trying to do.

Do I need to use a different method to pull a couple hundred PDFs from Drive?

Solution

You should be using Oauth2 to request that much data tbh. However if you insist on using an api key try adding quotaUser and userIp as part of your request.

Standard Query Parameters

Note: Per-user quotas are always enforced by the Drive API, and the user's identity is determined from the access token passed in the request. The quotaUser and userIp parameters can only be used for anonymous requests against public files.

If all the files are in the same directory you could use a service account and not have to worry about this error.

Oauth tokens.

An Api key is created on Google cloud console. They are used to access public api end points only. They identify your application to google and no more. You can only access public data not private user data. How to create an api key

Access token + refresh token. Are the results of an Oauth2 authorization request by a user. Access tokens are short lived they work for an hour then expire, they give you access to a users data, by sending an authorization header with the access token along with your request for data. Refresh tokens are long lived and can be used to request a new access token on behalf of the user when the one you have has expired Understand Oauth2 with curl

Answered By - DaImTo