Automate Google Drive uploads and file management with Python

Overview

Automate Google Drive tasks from Python: upload new files, update existing ones, list folder contents, and delete files. This guide shows a minimal, practical setup using the official Drive v3 API.

Quickstart

1. Enable the API and get credentials

• In your Google Cloud project, enable “Google Drive API”.
• Create an OAuth 2.0 Client ID (Desktop app) and download the credentials file as credentials.json into your project folder.

1. Install packages

• pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

1. Prepare a test file

• Create a small file (for example, example.txt) in your project folder.

1. Save the script below as drive_auto.py

2. Run the script

• python drive_auto.py
• On first run, a browser will ask you to authorize. A token.json will be stored for future runs.

1. Verify

• The script creates a Drive folder named Backups (if missing), uploads example.txt, lists the folder, and shows how to delete by id.

1. Schedule (optional)

• Use cron or Task Scheduler to run drive_auto.py at intervals for unattended uploads.

Minimal working example

import os
import mimetypes
from pathlib import Path
from typing import Optional, List, Dict

from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload

# Full Drive access for managing files. Narrow scopes if you can.
SCOPES = ["https://www.googleapis.com/auth/drive"]

def get_service() -> any:
    creds = None
    token_path = Path("token.json")
    if token_path.exists():
        creds = Credentials.from_authorized_user_file(str(token_path), SCOPES)
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file("credentials.json", SCOPES)
            creds = flow.run_local_server(port=0)
        token_path.write_text(creds.to_json())
    return build("drive", "v3", credentials=creds)

def ensure_folder(service, folder_name: str, parent_id: Optional[str] = None) -> str:
    query = [
        "mimeType = 'application/vnd.google-apps.folder'",
        "trashed = false",
        f"name = '{folder_name}'",
    ]
    if parent_id:
        query.append(f"'{parent_id}' in parents")
    else:
        query.append("'root' in parents")
    res = service.files().list(
        q=" and ".join(query),
        spaces="drive",
        fields="files(id,name)",
        pageSize=1,
    ).execute()
    files = res.get("files", [])
    if files:
        return files[0]["id"]
    meta = {
        "name": folder_name,
        "mimeType": "application/vnd.google-apps.folder",
    }
    if parent_id:
        meta["parents"] = [parent_id]
    created = service.files().create(body=meta, fields="id").execute()
    return created["id"]

def upload_or_update(service, local_path: str, folder_id: str) -> str:
    file_name = os.path.basename(local_path)
    mimetype = mimetypes.guess_type(file_name)[0] or "application/octet-stream"
    query = [
        f"name = '{file_name}'",
        f"'{folder_id}' in parents",
        "trashed = false",
    ]
    res = service.files().list(
        q=" and ".join(query),
        fields="files(id,name)",
        pageSize=1,
    ).execute()
    media = MediaFileUpload(local_path, mimetype=mimetype, resumable=True, chunksize=10 * 1024 * 1024)
    if res.get("files"):
        file_id = res["files"][0]["id"]
        updated = service.files().update(fileId=file_id, media_body=media, fields="id, name, version").execute()
        return updated["id"]
    else:
        body = {"name": file_name, "parents": [folder_id]}
        created = service.files().create(body=body, media_body=media, fields="id, name").execute()
        return created["id"]

def list_folder(service, folder_id: str) -> List[Dict[str, str]]:
    res = service.files().list(
        q=f"'{folder_id}' in parents and trashed = false",
        fields="files(id,name,mimeType,size,modifiedTime)",
        pageSize=1000,
    ).execute()
    return res.get("files", [])

def delete_file(service, file_id: str) -> None:
    service.files().delete(fileId=file_id).execute()

if __name__ == "__main__":
    service = get_service()

    # Adjust these for your use case
    folder_name = "Backups"
    local_file = "example.txt"

    folder_id = ensure_folder(service, folder_name)
    file_id = upload_or_update(service, local_file, folder_id)
    print(f"Uploaded/Updated: {local_file} -> id={file_id}")

    print("Files in folder:")
    for f in list_folder(service, folder_id):
        size = f.get("size", "-")
        print(f"- {f['name']} (id={f['id']}, size={size})")

    # Example: delete the uploaded file (comment out to keep it)
    # delete_file(service, file_id)
    # print(f"Deleted id={file_id}")

How it works

• OAuth flow stores a short-lived access token and a refresh token in token.json.
• ensure_folder finds or creates a Drive folder by name.
• upload_or_update searches by name within a folder. If found, it uploads a new revision; otherwise it creates a new file.
• list_folder queries metadata you can use to decide what to sync or prune.
• delete_file removes a file by id.

Automating regular uploads

• cron (Linux/macOS):

# Every hour
0 * * * * /usr/bin/python3 /path/to/drive_auto.py >> /path/to/drive_auto.log 2>&1

• Windows Task Scheduler: run python.exe with the script path on a schedule.
• To upload all files in a directory, wrap upload_or_update in a loop over Path("data").glob("*").

Common variations

• Rename a file:

service.files().update(fileId=file_id, body={"name": "new-name.txt"}).execute()

• Move a file to another folder:

# Add new parent, remove old parent(s)
service.files().update(
    fileId=file_id,
    addParents=new_folder_id,
    removeParents=old_folder_id,
    fields="id, parents",
).execute()

• Upload to a specific MIME type (e.g., plain text):

media = MediaFileUpload("notes.txt", mimetype="text/plain", resumable=True)
service.files().create(body={"name": "notes.txt", "parents": [folder_id]}, media_body=media).execute()

Pitfalls and how to avoid them

• Overly broad scopes: Using full drive scope is convenient but high-risk. Prefer drive.file if your app only needs access to its own files.
• Name collisions: Multiple files with the same name can exist in a folder. Search by name can return the wrong one. Store and reuse file IDs when possible.
• Shared drives: Calls must include supportsAllDrives=True and driveId or corpora if you target shared drives. Permissions may differ from My Drive.
• Shortcuts vs folders: Drive shortcuts mimic folders but are different types. Filter by mimeType to avoid misclassifying items.
• Quota/rate limits: Implement retries with exponential backoff for 403/429/5xx errors. Keep response fields minimal.
• Large files: Always use resumable uploads. Choose a chunk size (e.g., 10–32 MB) that balances memory and throughput.
• Time drift in scheduling: If running frequent jobs, ensure clock sync to avoid duplicate work based on timestamps.

Performance notes

• Resumable uploads: MediaFileUpload(..., resumable=True) recovers from transient failures without re-sending the entire file.
• Chunk size: Larger chunks reduce HTTP overhead but increase memory. 8–32 MB is a reasonable range. Tune based on network.
• Reduce response size: Use fields="id,name" or similar to speed responses and cut bandwidth.
• Reuse clients: Build the Drive service once and share it across operations.
• Batching: Group metadata-only operations where possible; for large-scale migrations, parallelize uploads carefully and honor Drive API quotas.
• Change detection: Only update when a file has changed. Compare file size, mtime, or a hash before calling update.
• Cache IDs: Cache folder IDs to avoid repeated list queries per file.

Tiny FAQ

• How do I avoid the consent screen every run? The token.json stores your refresh token. Keep it safe and reuse it.

• Can I use a service account instead of user OAuth? Yes. Service accounts work well for server-side tasks, but they need access to the target Drive or a shared drive.

• How do I target a shared drive? Add supportsAllDrives=True to list/create/update calls and set driveId and corpora parameters appropriately.

• Does update create a new version? Yes. Drive stores revisions. You can list or delete revisions via the Revisions API if needed.

• Maximum file size? Up to 5 TB if you have sufficient storage and use resumable upload.

• How do I make a file viewable by others? Create a permission on the file_id with type="user" or "anyone" and role such as "reader".