Making a mailbox migration boring

01 / 10

Why this reads as a list of things that cannot happen

The data being moved is often the only copy the user has, and the tool doing the moving holds credentials to both ends at once. That combination is why most of what follows is about what cannot happen rather than about throughput. A migration engine earns trust by removing dangerous options from itself, not by promising that an operator will avoid them.

EMaigrator is a self-hostable engine licensed Apache-2.0, published as prebuilt multi-arch images for amd64 and arm64 — so the artifact making the safety claims is the same artifact you can read and run. Version 1 leads with WorkMail to Microsoft 365, pairing an IMAP source with a Microsoft Graph destination, and ships Gmail and generic IMAP connectors alongside it.

02 / 10

The two guarantees almost everyone collapses into one

There are two safety claims here, and keeping them apart is worth the effort, because a tool can satisfy either one without the other.

Read-only at the source means migration is a copy: nothing at the origin mailbox is deleted, moved or rewritten, so there is no destructive path to get wrong under pressure and no cleanup step a half-finished run can leave in a bad state. No body persistence is a separate claim about our own storage: message bodies and attachments are never stored, and retained metadata is limited to subject, date, folder and status.

A delete-after-copy migrator could still stream bodies and persist nothing. A scrupulously non-destructive one could still spool every message to disk so retries are cheap. EMaigrator claims both.

03 / 10

Streaming pass-through, and the absence that is the feature

The way the no-storage promise is kept is embarrassingly simple, which is the point: a worker reads a message from the source and writes it to the destination in-flight, so bodies transit worker memory and are never persisted. There is no staging area to secure, purge, encrypt, or forget about.

What lifts this above a design preference is that it is written down as an invariant with consequences. Never persisting bodies is a declared security invariant, and violating it is treated as a reportable security bug. An invariant with a reporting channel behaves differently from a best practice in a README — it gives a contributor, an auditor or a user a defined way to say this build is wrong, rather than a matter of taste to argue about in review.

Streaming has one real cost, and the rest of the architecture is shaped by paying it. If nothing durable is written as the messages go past, an interrupted run has no artifact of its own progress to consult. Something else has to remember what was already copied.

04 / 10

The ledger, and two words worth separating

The ledger lives in Postgres and is the single source of truth for migration state, keyed by a canonical message identity. Every other component is allowed to be forgetful. Workers hold nothing, queues may redeliver, a process may be killed mid-batch — the question has this message already been copied? is always answered in one place, by identity rather than by position in a stream.

That is what makes restarting cheap. Resume means scanning the ledger for not-done items and re-enqueueing them, rather than replaying from the beginning. The distinction matters most exactly when it is most expensive to get wrong: on a large mailbox interrupted late, replay would re-cross the network for everything already copied and lean on the destination to reject the duplicates.

Idempotent means re-running a copy produces no duplicate. Resumable means a run can stop and pick up where it left off. Resumability without idempotency is exactly how migrations grow duplicate mail: the run continues, but any work in flight at the moment of interruption is done twice. Idempotency is what makes resume safe rather than merely possible.

05 / 10

One canonical model, not a matrix of connectors

Every migration is routed source to canonical model to destination — hub-and-spoke, never N by M direct connectors. Each provider is taught to speak the canonical model once, as a reader and as a writer, and any source-destination pairing falls out of that rather than being built.

This is worth judging rather than accepting as dogma. With two providers, hub-and-spoke is pure overhead — you have written a canonical model to serve one pairing you could have written directly. It pays for itself as the provider count grows, because direct integration grows with the product of sources and destinations while the plugin model grows with their sum. The moment to adopt it is before that curve bites, which is why it was a v1 decision rather than a later refactor.

The shape is held in place by a dependency rule rather than by discipline: Core depends on nothing, and connectors depend only on Core's interfaces. If Core cannot reference a connector, no provider's peculiarity can leak into the hub except deliberately.

06 / 10

Where a guarantee would otherwise leak

A connector is a self-contained assembly that adapts one provider's SDK to the Core abstractions and exposes exactly one discoverable plugin entry point. Connectors register with TryAddEnumerable, so each appends to the plugin collection rather than replacing it — avoiding the classic dependency-injection footgun where the last registration silently wins and a deployment ends up one provider short. That bug shows up as a missing option in a dropdown, not as an error.

But the interesting part of the connector contract is not the interfaces. It is the invariant it re-states: body bytes must never land on a persisted field. This is the extension point where the safety guarantee would otherwise leak — the hub can be immaculate while a well-meaning connector caches an attachment to make a retry cheaper. A guarantee that is not restated where third parties write code is a guarantee with a hole in it.

07 / 10

The real throttle is never your thread count

Worker concurrency determines how much work is in flight inside EMaigrator; the provider decides how much of it is accepted. Turning the first dial past the second buys throttling, not throughput — and usually costs more than it gains, because rejected calls still consume the budget.

The complication is where limits apply. Provider limits are enforced per account, but many workers may hold batches for the same account simultaneously, so coordination has to be shared rather than per-worker. Any scheme giving each worker its own budget is wrong by construction: n workers each politely staying under the limit collectively exceed it by a factor of n, and the result is the mass-429 stall that makes naive migrators look slow.

The mechanism is a token bucket per provider-and-account pair, held in Redis and updated by an atomic Lua script. Workers stay stateless and interchangeable, because pacing is a property of the account rather than of the worker. The same granularity governs pushback: on a 429 or a Retry-After, only that account's bucket is drained, and other accounts keep moving.

08 / 10

One bad thing never stops a good thing

Each level of the execution hierarchy is also a containment boundary. A message failing must not fail its batch; a folder failing must not fail its siblings; a throttled account must not pause the pool.

For the message that simply cannot be copied there is a defined end state rather than an infinite retry: it is parked in the dead-letter queue after retries and surfaces in the post-run needs-decision list. Both halves matter. Parking stops one malformed item consuming the retry budget forever; surfacing stops it being quietly lost — which is the failure mode users discover months later, looking for an email nothing ever flagged as missing.

One piece of end-to-end evidence: a live Google Workspace to Microsoft 365 reconcile copied 320 messages across roughly 650 labels with zero write failures. Read that for what it is — a real run against real providers that exercised the label-to-folder fan-out and the write path. It is a useful smoke test of the containment story, not a scale or endurance result, and we will not dress it up as one.

09 / 10

Turning failures into decisions

Everything above keeps a migration safe. This is what the product competes on. The error-resolution engine is built on a deterministic rule catalog mapping a provider and an error signature to a diagnosis and a remediation. The bet is that migration failures are a finite, recurring set rather than an open-ended space — the same handful of provider errors, seen over and over, each with a known cause and a known fix.

Determinism is doing real work in that sentence. A lookup table from signature to remediation can be tested case by case and gives the same answer offline as hosted. It also makes the catalog the natural home for contributed knowledge: a failure someone diagnosed once becomes a rule everyone gets.

Alongside it sits the pre-flight scan. Each provider plugin declares its own constraints — folder depth, path length, illegal characters — and pre-flight checks them before any data moves. Transient failures such as throttling are handled automatically with backoff, while anything that would change data is surfaced as a decision rather than silently defaulted.

10 / 10

Where the free engine stops

The commercial boundary is drawn as sharply as the safety ones, and enforced physically by a repository boundary rather than by convention. The consequence for a self-hoster is the one that matters: what is published is not a crippled edition waiting for a licence key. It is the migration engine.