Getting a Gmail mailbox to show up in your product is easy. Keeping it current, seconds after something changes, for thousands of users, without burning quota or silently drifting out of sync, is a different engineering problem. This article breaks down the pipeline most teams converge on: a push signal from Pub/Sub, a cursor called historyId, a renewal job nobody should forget, and a recovery path for the days when things go wrong.
Introduction
Any product that mirrors a user’s Gmail (a CRM timeline, a shared inbox, an AI triage assistant) eventually faces the same requirement: changes in the mailbox must reach the product quickly and exactly once. The gmail api gives you the pieces to do that, but not the assembled pipeline. The pieces are a watch request, a Google Cloud Pub/Sub topic, and the History endpoint. How you connect them, and what you do when one of them misbehaves, determines whether your sync is trustworthy. This article assumes OAuth is already working and focuses only on the data flow after consent.
The three moving parts and how they fit
Think of the pipeline as a doorbell, a ledger, and a bookmark.
The doorbell is the push notification. You call users.watch on a mailbox and name a Pub/Sub topic. From then on, when the mailbox changes, Gmail publishes a small message to that topic. Your topic needs to grant publish rights to Gmail’s dedicated service account, otherwise the watch call succeeds and nothing ever arrives, which is a confusing first afternoon.
The ledger is the mailbox history. Every change in Gmail (a message added, deleted, labeled, unlabeled) is recorded with a monotonically increasing historyId. The users.history.list method returns the changes that happened after a given historyId.
The bookmark is the historyId you stored last time you processed a mailbox. It is the single most important value in your database for that account.
The key insight is that the doorbell carries almost no information. The Pub/Sub payload contains the email address and a historyId, not the message. It tells you “something changed, up to here”. The actual changes always come from the ledger.
From notification to change set
A worker that consumes the subscription should do very little with each notification itself. A pattern that holds up well:
on notification(email, newHistoryId):account = lookup(email)if newHistoryId <= account.cursor: ack and returnenqueue sync(account) # coalesced per accountacksync(account):page through history.list(startHistoryId = account.cursor)apply messageAdded / messageDeleted / labelsAdded / labelsRemovedfetch full messages only for ids you have never seenaccount.cursor = latest historyId returned
Two details matter. First, coalesce: several notifications for the same account often arrive close together, and they should collapse into one sync job, not trigger parallel History calls that race each other. Second, advance the cursor only after the changes are durably applied. If the worker crashes halfway, it will replay from the old cursor, so your write path must be idempotent, keyed on the Gmail message ID.
You can narrow history.list with the historyTypes parameter and a labelId when you only care about, say, new messages in the Inbox. That reduces payload and processing, but think ahead: if the product later needs label changes, you will want them in the history you already consume.
Quota is measured in units, so spend them on the right calls
The Gmail API does not simply count requests. Each method has a cost in quota units, and limits apply both per user and per project. Lightweight reads such as history.list and messages.get cost little, while sending and the watch call itself cost considerably more. Google publishes the current unit table and limits, and they are worth checking before capacity planning rather than assuming.
The design consequence is clear. Lean on history.list for detecting change, fetch message bodies only for genuinely new IDs, request the format you need (metadata when headers suffice, full when you need content), and use batch requests to group message fetches. When you hit a rate limit, Gmail answers with a 429 or a 403 carrying a rate-limit reason, and the right response is exponential backoff with jitter on that account while others continue.
Renewal: the job that keeps the doorbell wired
A watch does not last forever. The response includes an expiration timestamp, and the watch must be renewed at least every seven days. Google recommends simply calling watch again once a day, which is idempotent for the same topic and mailbox.
Treat renewal as a first-class scheduled job, not a side effect of user activity. Accounts that nobody opens for a week are exactly the ones whose watch will lapse unnoticed. Store the expiration per account, renew well before it, and alert when an account has gone too long without either a renewal or a notification.
Recovery paths for the bad days
Production sync is defined by its failure handling. Plan for at least these cases.
- Stale cursor. History records are retained only for a limited time. If your stored historyId is too old, history.list returns a 404. The only correct response is a full resync of that mailbox (or of the labels you track), followed by storing a fresh historyId from the mailbox profile.
- Missed notifications. Pub/Sub delivery is reliable but not a guarantee of completeness, and Google’s own guidance is to expect occasional delays or drops. A light periodic sweep that calls history.list for accounts with no recent notification catches silent gaps.
- Duplicate deliveries. Pub/Sub may deliver the same message more than once. The cursor comparison and idempotent writes make this harmless.
- Revoked access. When a user removes your app or a token expires for good, refresh fails with invalid_grant. Stop syncing that account, mark it as needing reconnection, and surface that clearly in the product.
- Deployments. Pull subscriptions keep unacknowledged messages, so a short deploy does not lose events, provided your workers acknowledge only after enqueueing work.
Build the pipeline or consume one
None of this is exotic, but it is a lot of always-on infrastructure for one provider: a Pub/Sub topic and subscription, a coalescing worker queue, a renewal scheduler, a resync path, and monitoring for each. Teams that also need Outlook and IMAP end up writing two more pipelines with different mechanics. That is where unified email APIs come in. Unipile, for example, runs the Gmail watch and history handling for each connected account and delivers changes to your backend as webhooks, alongside Outlook and IMAP accounts, acting on behalf of the authenticated user. It is SOC 2 Type II certified and CASA verified for Google restricted scopes. Whether you build it or adopt it, judge the result with one test: delete, label, and receive messages in a real Gmail account, take your workers down for ten minutes, bring them back, and confirm that every change still lands exactly once.
