Beet

Self-hosted mail

This is a runbook, not a tutorial. The tutorials teach beet's concepts from an empty file and type out every line; this one stands up a real mail server on real infrastructure with a real domain, and it assumes you can fill in blanks, hold context across a week and read an error message without being told what it means.

It is also the longest-running thing in the docs. You start it on a Monday and finish it the following week, and most of that time is spent waiting on an Amazon support case. Plan for that rather than being surprised by it.

The shape is: Stalwart owns your mailboxes, receiving and policy on one small box you control. Its embedded SQLite database lives on a persistent EBS volume, while message bodies live in S3. Outbound leaves through whichever relay each domain declares, or through no relay at all. The whole system is declared as beet_infra blocks in one .bsx file.

This runbook takes the Amazon SES path throughout, because that is the one it was written against and the one with the longest tail of real operational detail. Section 1a says what the choice is and how to take a different one; nothing else in the runbook changes if you do.

1. What you get, what it costs, what it demands

You get mailboxes whose message bodies sit in a bucket you own and whose metadata sits in an embedded SQLite database on a volume you own. You get JMAP on port 443, which is the reason to do this at all if agents are going to read your mail: it is JSON over HTTP with batching and push, not a 1986 line protocol with a decade of extensions bolted on. You get IMAP, submission and CardDAV/CalDAV alongside it, so ordinary mail clients work with no special handling. And with a relay in front of delivery, your messages leave from IP addresses somebody else spends a great deal of money keeping trusted.

It costs roughly US$25 a month in Sydney on-demand pricing: EC2 t4g.small around $15, an elastic IP around $4, the root and persistent EBS volumes around $5, plus S3 and SES usage. A savings plan takes roughly 30% off the compute later.

It demands four things:

What you do not get is escape from operations. This is a server that other servers connect to on port 25 at three in the morning, and it stays that way.

1a. Choosing a relay, or not having one

Outbound delivery is composed beside a domain rather than assumed by it. A relay is a component you spread onto the <MailDomainBlock/>, or onto an ancestor of it so every domain underneath inherits:

<MailDomainBlock domain="example.com" mail_host="mail.example.com" {SesRelay}/>
<MailDomainBlock domain="example.com" mail_host="mail.example.com" {ComailRelay}/>
<MailDomainBlock domain="example.com" mail_host="mail.example.com"/>

The three are genuinely different trades, and none is the safe default in the abstract.

No relay is what a bare declaration means: the box dials each recipient's MX itself, publishes v=spf1 mx -all, and signs with the key this stack holds. Everything that makes that respectable is already in the stack, which is the point: a stable elastic address whose reverse record matches the SMTP banner, MTA-STS, TLS-RPT and DMARC at p=reject from the first deploy. What you take on is that a fresh address has no reputation, so early mail to the large receivers lands in spam until weeks of ordinary correspondence fix it, and that a listing is yours to get delisted from. What you keep is that nobody can revoke your sending, and no third party is in the path of your mail.

Amazon SES buys that reputation. It adds a sending identity and a configuration set per domain, three rolling Easy DKIM selectors, a custom MAIL FROM subdomain that makes SPF align, an event stream and reputation alarms, and one static IAM SMTP credential because the protocol demands one. It is what the rest of this runbook does, including the support case in section 4.

comail is an atproto-identity SMTP relay: you enrol a domain against a DID you already own, and it signs that domain's mail with a pair of selectors it publishes. Nothing is created in AWS. Four things are worth knowing before you choose it:

examples/infra/mail.bsx, mail_ses.bsx and mail_comail.bsx are the three, each minimal.

Whichever you pick, the sovereign stalwart._domainkey selector is published in all three. That is deliberate: a receiver needs one signature to pass, so holding a key nobody else can revoke is what makes the relay replaceable at all rather than a dependency you are stuck with.

Moving between them

There is no prescribed sequence, because the right one depends on what you are protecting. The facts that constrain it:

Two shapes people actually use. Put a greenfield domain (a newsletter subdomain, say) straight onto the new relay and let it warm while nothing depends on it. Or stage the change on a domain that matters using the same records="IdentityOnly" machinery section 8 uses for the apex cutover, so the new selectors verify while the incumbent relay keeps carrying the mail, then flip in one apply. Which one you want is a judgement about your traffic, not something a runbook can decide for you.

2. Build on a staging domain

This is the most useful idea in the tutorial, so it comes before any of the machinery.

You already have mail somewhere. That mail is how your invoices arrive and how your clients reach you, and it cannot wait two weeks while you learn what a Stalwart 0.16 config object looks like. So do not build on your apex domain. Build the entire system on stalwart.<your domain>, prove it, live in it, and move the apex across in one short window at the end.

The reason this works rather than merely deferring risk is that MX, SPF, DKIM and DMARC are all per-hostname records. A subdomain is a wholly separate mail domain as far as SMTP is concerned. Your v=spf1 include:amazonses.com -all at stalwart.example.com does not collide with your incumbent provider's SPF at the apex, because the two-record permerror only applies to two SPF records at the same name. A subdomain's own _dmarc takes precedence over the organizational domain's policy, so you can publish p=reject on the staging domain from the first deploy without imposing anything on the apex. Nothing you do during the build touches production mail.

One detail makes the eventual cutover short rather than a second build: the box is not a mail domain. Name it mail.<your domain> from the beginning and keep that name forever. Its reverse DNS record, its TLS certificate and its SMTP banner then never change across the cutover, which removes the riskiest churn from the riskiest step. Only the mail domains move.

Treat "no apex mail record before the cutover" as an invariant, not a preference. Write it as a test: every record name a staging block emits should be asserted to be under the apex and never at it.

3. Credentials and preflight

Hand step. Declare the credentials the repo deploys with at the entry's root, and generate the state passphrase (the secrets tutorial covers the identity and the document):

<AwsDeployCredential/>
<CloudflareDeployCredential/>

beet secrets/set TF_STATE_PASSPHRASE --role=env_var --generate --note="opentofu state encryption" --roll="manual:not re-mintable, losing it loses the state"
beet mail/deploy --stage=prod --dry-run   # what each credential would hold, asking for nothing

The first deploy is then an elevated one (beet mail/deploy --stage=prod --elevated), and the dry run says so. Neither credential is sealed by hand. The AWS pair is this repo's own IAM user, given one managed policy per app the entry declares, lowered from what the stacks render, and sealed as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the policies it was converged against; converging it is an administrator's act, so the elevated deploy asks for the six-digit code and buys an administrator session for it, since the deploy credential deliberately holds no iam: action over itself.

The region, the Cloudflare account and the zone are not credentials: they are identifiers, public in every dashboard url, and they are declared in the markup as spreads on the stack or its root ({AwsRegion("ap-southeast-2")}, {CloudflareAccount("..")}, {CloudflareZone{domain:"example.com", id:".."}}, see section 6), never read from the environment. The passphrase is the one unrecoverable value in the document: it encrypts the OpenTofu state client-side, and state carries the SES SMTP credential because sensitive = true on a tofu value redacts it from plan and apply output but not from state. It is generated in-process and never printed; the document is committed, so the identity that opens it is the backup.

Not every mail credential is in there. A comail api key is parked in parameter store by you and read by the deploy verbs directly, and the sovereign DKIM private half never leaves parameter store either; only its public half is a tofu variable. What state carries is what terraform derives, which for mail is the SES pair.

The Cloudflare token is not typed by hand either: the elevated deploy lowers its scopes from the resources the entry's stacks render and the groups each of its Cloudflare actions declares on itself (CloudflareAccess), and seals the result as CLOUDFLARE_API_TOKEN with what it grants. For the mail stack that is DNS Write and Workers Routes Write on the one zone (the records and the audit, the MTA-STS custom domains), and at the account Workers Scripts Write and Account Settings Read (the MTA-STS Worker, which wrangler deploys) and Workers R2 Storage Read (the cold bucket, its lifecycle and its lock, which every plan refreshes). beet mail/deploy --dry-run prints the exact list for your entry, with what asks for each group, and needs no credential.

What it deliberately does not carry is Account API Tokens Write. A token that may edit the account's api tokens can mint itself anything the account can hold, a wider token included; Cloudflare has no permissions boundary to cap that with and no mfa condition to put on it, so the only thing that keeps it out of reach is that no credential a deploy reads carries it. That group lives in a second token kept nowhere: the mint token, which an elevated deploy asks the operator for at the terminal after showing where to roll it on the dashboard, and which the age key must not open. So the two providers answer the same question the same way: the frequent credential is scoped and recoverable and sits in the document, and the one that can escalate needs something the document cannot give you (an mfa code for AWS, a dashboard login for Cloudflare).

The deploy token reads the cold bucket's configuration rather than writing it, because Workers R2 Storage Write is account-wide and also lifts a bucket's lock. A deploy that would change the bucket, its lifecycle or its lock is refused before it writes anything, and names the elevated run that makes the change: beet mail/deploy --stage=prod --elevated asks for the mint token, mints a token holding the deploy groups and that one write, runs the deploy with it and deletes it. The same holds at AWS for the backup and blob buckets' versioning and lifecycle and the box's data volume.

Hand step, and worth doing before creating anything. Prove every permission you are about to need, cheaply, in the order that fails fastest:

curl -s https://api.cloudflare.com/client/v4/accounts/<account id>/tokens/verify \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"
aws sts get-caller-identity

Then one cheap read per AWS service you will touch (ec2, sesv2, ssm, iam, s3, logs) in your chosen region, and a throwaway TXT record create and delete against the zone to prove DNS write rather than assume it. A missing permission discovered here costs seconds; the same permission discovered during an apply leaves a partly-created stack.

Gap: this should be a Preflight action. It is the same check every stack wants and nothing in beet performs it yet.

Hand step. Enumerate the whole zone before you touch it, rather than reasoning about what ought to be in it:

beet secrets/exec -- sh -c 'curl -s "https://api.cloudflare.com/client/v4/zones/<zone id>/dns_records?per_page=200" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"'

The default page size is 100, so a zone that outgrew one page will quietly report only its first hundred records. Ours held two records that were in nobody's plan: the ACM DNS-validation CNAMEs for the website's certificate, whose names are computed by ACM and can therefore only ever be allowed as a pattern. You will want that inventory again in section 6, when you write the audit allowlist.

Hand step. Sort out CAA before anything requests a certificate. If your zone carries an amazonaws.com-only CAA set, it covers ACM and blocks everything else, including the Let's Encrypt issuance Stalwart's ACME will need and Cloudflare's Universal SSL. Publish issue and issuewild rows for letsencrypt.org, pki.goog; cansignhttpexchanges=yes and ssl.com alongside the amazonaws pair. CAA is a zone-wide singleton: exactly one stack should own those rows, and every other block reconciles with them rather than emitting its own.

Hand step. Confirm your registrar. If the domain is registered at Cloudflare, enabling DNSSEC publishes the DS record to the parent automatically. Anywhere else and that is a manual registrar step you should schedule now rather than discover later.

Also stand up a website at the domain before you go further. This is not decoration: a human at Amazon opens that URL while reading your support case. Do it before section 4, not after.

4. SES production access, which is the long pole

Start this on day one. Everything else in this tutorial can proceed inside the SES sandbox, including the full deploy and both halves of the delivery probe, so the case runs in parallel with the build. But it gates real mail to real people, and it is measured in days.

The reality is a two-step negotiation, not a form.

Step one is the request, in the SES console or by api: aws sesv2 put-account-details --production-access-enabled --mail-type TRANSACTIONAL --website-url .. --use-case-description .., which needs no support plan. A one-paragraph request is auto-denied within a day, with a reply asking for detail and pointing out that you need a verified identity first. That is the normal path, not a rejection, and it is avoidable: verify the identities first, and put everything the reply below would say into step one's description, so there is nothing left to ask.

Step two is a written reply through the Support Center. It goes by hand: aws support requires a Business or Enterprise support plan and returns SubscriptionRequiredException on Basic, so there is no CLI path at any support tier worth paying for to get one.

Before you write it, make the claims true. Reviewers check, and a present-tense claim that is aspirational is the thing that turns a three-day case into a three-week one. So section 5's work, at minimum, happens before you reply: verified domain identities, DKIM verified, custom MAIL FROM verified, configuration sets with suppression and event publishing live, a DMARC record actually published, and a test message whose headers you have read.

A good reply is organised and specific. Cover, roughly in this order:

Send it within days. Idle cases auto-resolve, and while a resolved case can be reopened, momentum matters.

Ours was granted three days after the initial request and two after the detailed reply, moving the account out of the sandbox with a quota of 50,000 messages a day at 14 per second, effective immediately in the region. That grant is per account and per region, not per identity, so a grant earned while sending from a staging subdomain carries over to your apex at cutover with nothing to re-request. It does not carry to another region or another account: a repo moving into an account of its own starts in the sandbox there, so the request goes in first, from the new account, saying it is the same organisation and the same mail moving house and that the old account stops sending at cutover.

Port 25 is the other per-account, per-region limit. AWS throttles OUTBOUND 25 from a new account's instances and releases it through a request form; inbound is never throttled. A box that relays all its outbound mail through SES, as this stack's does, never sends on 25 itself and needs no release, so check the relay configuration before asking (and test it from the box: nc -vz gmail-smtp-in.l.google.com 25 hangs while throttled).

Not automatable. A support case argued in prose is a human writing to a human. Do not plan around a future action for this one.

5. Identities, DKIM, MAIL FROM, configuration sets, and the topic

Everything in this section is eventually declared by MailDomainBlock and applied by tofu. Do it by hand first anyway, for the two sending domains, because the support reply in section 4 needs it to already be true and because doing it once by hand is how you learn what the block is doing on your behalf.

Per sending domain:

aws sesv2 create-email-identity --email-identity "$DOMAIN" \
  --dkim-signing-attributes NextSigningKeyLength=RSA_2048_BIT

aws sesv2 put-email-identity-mail-from-attributes --email-identity "$DOMAIN" \
  --mail-from-domain "bounce.$DOMAIN" --behavior-on-mx-failure USE_DEFAULT_VALUE

The enum is RSA_2048_BIT. RSA_2048 is rejected.

Then publish, all DNS-only and none of them proxied:

Verification on a zone you control authoritatively took under thirty seconds for us, not the hours the AWS documentation warns about. Those warnings are written for people whose DNS changes propagate through a provider they do not control.

Then a configuration set per sending domain, with reputation metrics on, per-set suppression for BOUNCE and COMPLAINT, and an event destination matching BOUNCE, COMPLAINT, REJECT and DELIVERY_DELAY. Make it the identity's default with put-email-identity-configuration-set-attributes. Reputation metrics being enabled is load-bearing rather than decorative: without it the CloudWatch metric never appears, and an alarm on a metric that never appears sits in INSUFFICIENT_DATA looking exactly like a healthy one.

All the sets can publish to one account-wide SNS topic. Events carry their configuration set, so splitting the topic only pushes routing from the consumer into IAM.

The trap in this section, and it is a good one, is the topic's access policy. The default policy on a new SNS topic is owner-only, and SES publishing into it fails silently: no error appears in SES, nothing appears in CloudWatch, and you discover it whenever you first go looking for a bounce that should have been recorded. The policy must grant SNS:Publish to the ses.amazonaws.com service principal under an AWS:SourceAccount condition.

Prove the chain before you claim it. In the sandbox you can only send to verified recipients, so verify a personal address as an EMAIL_ADDRESS identity, send one message from each domain with aws sesv2 send-email, and then read the headers. In Gmail that is "Show original", and what you want to see is SPF PASS, DKIM PASS with d= equal to the sending domain, and DMARC PASS. A partial pass is the interesting failure: DKIM alone still delivers to plenty of receivers, which is exactly how a broken SPF record ships unnoticed for months.

Gap: every step above except the support case is a MailDomainBlock field today, so on a second domain you would declare it rather than type it. The by-hand pass exists because of the ordering: the support reply needs verified identities, and the block cannot exist before you have written it. If you are doing this a second time, declare it.

Hand step, with a judgement call attached. You may publish the apex DKIM CNAMEs early if you want to. They are namespaced by selector token, so they cannot collide with your incumbent provider's selectors, and publishing early takes identity verification out of the cutover window entirely. We declined, on the grounds that "no apex mail record before cutover" is a rule worth keeping absolute, and because verification turned out to take seconds anyway. Either answer is defensible. Decide deliberately.

6. The stack itself

Now the automated part. One .bsx file declares the whole system, and the boundary between this section and the previous two is the honest answer to how much of this is automated.

<Stack app_name="acme" {(AwsRegion("ap-southeast-2"), CloudflareAccount(".."), CloudflareZone{domain:"example.com", id:".."})}>
	<DeployRoutes/>

	<VpcBlock bx:ref="net" label="net" zones={["a"]} private_tier=false ipv6=true/>
	<SnsTopicBlock label="ses-events" name="acme-ses-events"
		publishers={["ses.amazonaws.com"]}/>
	<S3BucketBlock label="mail-blobs" deploy_versioned=false force_destroy=false
		runtime_write=true object_versioning=true/>
	<S3BucketBlock label="archive" deploy_versioned=false runtime_write=true
		object_versioning=true expire_prefixes={[{prefix:"sqlite/", expire_days:180}]}/>

	<StalwartBlock label="mail" hostname="mail.example.com"
		blob_bucket="mail-blobs" backup_bucket="archive" dns_stage="prod"
		{VpcRef($net)}
		ssh_public_key="ssh-ed25519 AAAA... deploy"/>

	<MailDomainBlock
		domain="stalwart.example.com"
		mail_host="mail.example.com"
		alarms_topic="acme-ses-events"
		{SesRelay{events_topic:"acme-ses-events"}}
		dns_stage="prod"
		mailboxes={[
			{localpart:"pete", admin:true},
			{localpart:"probe"},
		]}
		aliases={[
			{localpart:"postmaster", target:"pete"},
			{localpart:"abuse", target:"pete"},
			{localpart:"dmarc", target:"pete"},
			{localpart:"tlsrpt", target:"pete"},
		]}/>
</Stack>

A domain is one MailDomainBlock plus the relay spread beside it, and everything else composes from the pair: the SES identity, every record that makes the domain deliverable and discoverable, and the mailboxes it holds. A second domain is a second tag rather than an edit, which is exactly what makes the cutover in section 8 one more tag beside the others.

The two topics are two different things and are named separately for that reason. alarms_topic on the domain is where a reputation alarm fires, and every relay has one. events_topic on the SesRelay is where the raw stream of bounce, complaint, rejection and delay objects is published, and only SES emits that. In practice they are the same topic, as here; one of them survives a change of relay. The topic itself is the SnsTopicBlock: account-wide, so declared once and named by its bare name everywhere else, with a policy that lets SES publish under an AWS:SourceAccount condition (the default policy grants a service nothing, and SES drops events it cannot publish without a word). If you made the topic by hand in section 5, adopt=true on the block has the first apply import it and its policy rather than fail on the duplicate name; take the flag off after that deploy, since an import of a topic a later destroy has removed refuses to apply. What stays by hand is the subscription that pages you: its endpoint is your address, and its confirmation is a click in your inbox whatever owns the resource.

Some notes on the declarations that are not obvious from reading them:

The order of the MailDomainBlock tags is load-bearing, and nothing at the tag says so. The first one declared becomes the server's primary domain, which settles two things you would not go looking for in a list of domains. It is what system('domain') resolves to, so it is the domain every outbound report is addressed from (noreply-dmarc@<primary>, section 7). And it is the only domain whose ACME order carries the box's own hostname, since that name belongs to no mail domain and has to ride along on one. Reordering the tags therefore moves the server's identity and reissues a certificate, and deleting the first tag hands both to whichever domain happens to be next. Put the domain you intend to keep second, directly behind the staging domain it will replace, so that retiring staging promotes the right one rather than the newsletter domain.

The VPC declares one public subnet in zone a. The stack is one public EC2 box and one EBS volume that must share its availability zone, so zones={["a"]} private_tier=false says exactly that and emits one subnet. The topology is a declaration rather than a default because nothing in the block can infer it: leave those attributes off and you get two zones with a public and a private subnet in each, which is what an <RdsPostgresBlock/> subnet group requires and what a multi-zone workload wants. Subnets are free either way; the reason to declare the small one is that a private tier nothing sits in is a tier somebody will later wonder about. ipv6=true makes that public subnet dual-stack and the box takes one address out of its /64 and publishes an AAAA beside its A; section 10 has what that changes.

The box is cattle; its data volume is not. Machine config is cloud-init user data, and any change to that user data replaces the instance. Inbound SMTP during a rebuild is covered by sender retries, which run for days. What must survive that replacement is the mail itself, so Stalwart's SQLite database sits on its own encrypted gp3 volume at /var/lib/stalwart, a separate resource in the subnet's availability zone that is detached from the old box and reattached to the new one. Production defaults to prevent_destroy with a final snapshot; non-production defaults to disposable, so the same declaration can run a restore drill and then be destroyed. Set data_volume_protected explicitly only to override that stage-derived data grade.

Everything about the mount is written assuming the detection can fail. Cloud-init waits for the exact nvme-Amazon_Elastic_Block_Store_<volume-id> by-id path rather than the /dev/sdf it asked for, because Nitro does not honour the requested name. It formats only a device blkid has positively reported as blank: blkid exits 2 for "no filesystem here" and non-zero for every other reason, and treating those alike is how a transient read failure comes to mkfs the mail store. It records the filesystem UUID in fstab with nofail, so a volume that never appears leaves a box you can still ssh into rather than an emergency shell you cannot; RequiresMountsFor on the units is what keeps Stalwart from starting against a missing store. The one machine-config value that is a terraform reference is the volume id, and it is substituted after the user data's ${ escape pass. See section 9 for why that ordering has its own paragraph.

The one static credential in the stack is the SES SMTP key pair, because the SMTP protocol forces it. Terraform derives it into two SecureString parameters and it belongs to a dedicated IAM user whose only permission is ses:SendRawEmail. Everything else, including the instance's access to its own buckets, is an instance profile with IMDSv2 required. That user exists exactly while some domain relays through SES: move the last one off and the next apply removes it, which is the point of the relay being composed rather than assumed.

The ports are 22, 25, 443, 465, 587 and 993, and nothing else. The management HTTP port is deliberately not among them: provisioning reaches it through an SSH tunnel, and the listener that serves it is destroyed at the end of the first provision.

dns_stage is not decoration. A mail name is not stack-composed: mail.example.com is the real name of real mail, so a second stage deploying the same declaration publishes a second record at that name and receivers round-robin between your live box and your experiment, taking production mail down without erroring anywhere. dns_stage names the one stage that owns the shared names; outside it the blocks emit their infrastructure and touch no record.

The deploy route is a sequence, and its order is the design:

<Route path="deploy" {ExchangeSequence}>
	<EnsureSecret secret="mail-admin-password"/>
	<EnsureDkimKey/>
	<StalwartSnapshot/>
	<TofuApply/>
	<EipReverseDns/>
	<StalwartProvision ssh_key="~/.ssh/id_ed25519_mail"/>
	<MtaStsPublish/>
	<MailDane ssh_key="~/.ssh/id_ed25519_mail"/>
	<TofuApply/>
	<MailProbe mailbox="probe" sender_domain="news.example.com"/>
	<ZoneAudit/>
</Route>

The bootstrap admin secret is minted before the box starts. EnsureDkimKey is create-if-missing for a sharper reason than the password: a rolled key under an already-published selector is a fortnight of unverifiable mail. The key is minted before the apply and the apply publishes its public half, so the selector the world resolves and the key the server signs with are one parameter read twice. Letting the server generate its own key would mean reading it back and publishing in a second apply, with a window in between where mail is signed by a selector nothing answers for. StalwartSnapshot then finds the persistent volume by its Name, Project and Stage tags and waits for its snapshot to complete. No volume exists on the first deploy, so that case is an intentional skip; retries within one deploy use its UUID as EC2's idempotency token.

The apply starts only after the snapshot completes. Reverse DNS comes after the apply because AWS validates the forward record before publishing the reverse one. Provisioning comes after that, because Stalwart 0.16 keeps listeners, routing, domains and accounts as objects inside its data store rather than in any file terraform writes. The MTA-STS policy body is published after the apply that published the record pointing at it. The DANE pin is read off the box after provision and published by a second apply, for the reason section 10 gives. The probe proves a message goes out and a message comes back authenticated. And the audit runs last, because it is the only check that can see what the deploy did not do: a record left behind by a block that stopped declaring it.

Two things about running these verbs:

plan and deploy do not produce the same diff, and reading a plan without knowing that is alarming. deploy runs the DKIM step before the apply and plan does not, so a plan will show the DKIM record going to p= empty. The empty p= is worth recognising because it is also a genuine failure mode: p= with nothing after it is the wire form of a revoked key, and a bare apply that reaches past the deploy route will publish exactly that.

Run validate, then plan, then the audit, before you deploy anything. plan against real providers costs nothing and is the last cheap check before a phase that costs real money and real DNS. Check that the planned DNS records match the specification table exactly and that nothing moves the apex early. Running audit before the first deploy is likewise worth it: against a zone of 30 records and a stack declaring 31 it reported clean, which means the allowlist was proven right while it was still free to be wrong.

The allowlist belongs on the stack rather than on the audit verb, so that the audit at the tail of deploy and the standalone audit route read the same list. Every entry is a decision with a reason attached, and on a staging build most of them retire at the cutover:

<ZoneAudit allowed={[
	{name:"example.com", record_type:"MX",
		reason:"the incumbent provider serves the apex until cutover"},
	{name:"example.com", record_type:"TXT",
		reason:"the incumbent's apex SPF; a second SPF at this name would be a permerror"},
	{name:"_*.example.com", record_type:"CNAME",
		reason:"acm dns validation, name computed by acm"},
	{name:"mta-sts.stalwart.example.com",
		reason:"wrangler's worker custom domain: record and certificate both"},
]}/>

Two subtleties in writing one. Match DKIM selector names as patterns, because the SES selector token is computed and unknown until after the first apply, so the declared name is a terraform interpolation; collapsing ${...} to a wildcard label keeps the audit useful before the first apply, which is exactly when a stray record does the most damage. And restrict a pattern by record type where you can, so that _*.example.com as a CNAME does not also swallow your _dmarc, _mta-sts and _smtp._tls TXTs.

7. The first deploy, the probe, and the soak

Hand step. Mint a dedicated deploy key before the first apply, because EC2 installs a key only at launch and swapping one later replaces the instance:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_mail -N '' -C mail-deploy

The public half goes into <StalwartBlock ssh_public_key=..> and the path to the private half into <StalwartProvision ssh_key=..>. A dedicated key is the right answer here: reusing one that already authenticates somewhere else means one compromise reaches both.

Hand step, if you did section 5 by hand. Reconcile what you built by hand with what the stack now declares. SESv2 will not let terraform adopt an existing identity or configuration set silently, so the path is delete-and-let-the-apply-recreate: aws sesv2 delete-email-identity, aws sesv2 delete-configuration-set, and delete the hand-added zone records over the DNS API. Capture the configuration sets' event destinations first if you want to re-add them, and note that event destinations are create-only from terraform's point of view, so a hand-made one that exists in AWS but not in state fails the apply with AlreadyExistsException.

Then:

beet --entry=infra/mail.bsx --stage=prod deploy

Budget an hour for the first one and expect it to fail at least once. Most of that hour is the box: the AMI, the pinned release, the volume it waits for and the certificate it orders. A converged, idempotent re-deploy afterwards runs in under two minutes.

When it fails

Some failure shapes worth recognising, because each cost us hours:

The box is silent and provisioning times out.systemctl status stalwart says the unit could not be found, which means cloud-init never got that far. sudo cloud-init status and then /var/log/cloud-init-output.log is the path. Ours was a tarball download dying mid-transfer on a transient network error. User data runs once per instance under set -e, so a flaky download is a permanently dead box that answers SSH perfectly: the service just does not exist. Fix it in the block (retry flags on the download) rather than on the box, and redeploy, which replaces the instance and proves the fix on a fresh boot.

An instance replacement changes the host key, and a stale known_hosts entry makes the next SSH fail with REMOTE HOST IDENTIFICATION HAS CHANGED. ssh-keygen -R <ip> between replacements.

Deleting a block whose resource is protected leaves a resource nothing can destroy. Removing an <RdsPostgresBlock/> declaration and applying fails with Cannot delete protected DB Instance: terraform plans a destroy, but the attribute that forbids it is only settable through the declaration that is no longer there, so there is nothing left to turn it off with. Retiring a protected resource is therefore two applies, in this order: set deletion_protection=false (or data_volume_protected=false) on the block while it is still declared and apply, then delete the block and apply again. Clearing the flag out of band with aws rds modify-db-instance --no-deletion-protection gets you unstuck once, but it leaves the live resource disagreeing with the state until the destroy lands, so prefer the two applies when you have the choice. The protection did its job either way; it is meant to cost you a deliberate step.

A config parse error reporting "line 1 column N". Check whether N is the length of the file. An error at EOF means the parser consumed everything and a required field was missing at the top level, which is to say the document shape is wrong rather than a field in it being wrong.

A partial apply is not a rolled-back apply. One of ours replaced the instance and then errored on a later resource, leaving a rebuilt box with provisioning never run. Mail was down until the re-run, with inbound covered by sender retries. Killing a deploy mid-provision is safe by contrast: the apply has already committed its state and provisioning is idempotent, so the recovery is to fix the cause and run deploy again.

A hung deploy is not necessarily a broken box. SSH in and run sudo ss -lntp. A box listening on every declared port and none of the undeclared ones has recovered, and the problem is in what the deploy is waiting for. systemctl is-active stalwart and test -f /etc/stalwart/config.json are the other two lines worth running.

ACME fails on a name that has no A record. A mail domain is MX and TXT records with nothing resolving at the name itself, so it must not be a certificate SAN. Let's Encrypt fails that name and the entire order dies with it, including the valid names beside it. The SANs are exactly the names that resolve to the box: autoconfig.<domain>, autodiscover.<domain>, and the box hostname once. This failure is visible only in the server's own log, not in anything the deploy prints.

The probe

MailProbe is the assertion that matters, and it works inside the SES sandbox. The outbound leg sends through the box's own submission listener with curl speaking implicit-TLS SMTP with AUTH, which exercises the listener exactly as a mail client would, TLS verification included. The inbound leg sends from your publication domain via aws sesv2 send-email to the probe mailbox, then reads the message back over JMAP and asserts its Authentication-Results header carries spf=pass, dkim=passanddmarc=pass. Both simulator addresses and a once-verified recipient identity are permitted in the sandbox, so no production access is needed to prove the whole loop.

Both legs follow the sending domain's relay, since that is what decides what a receiver can conclude. A comail domain sends its inbound leg over the HTTP send API and is judged on dkim=pass and dmarc=pass only, because its envelope sender is a VERP address at the relay's domain and can never align; a 429 during the warming window is logged and skipped rather than failing the deploy. A domain with no relay has nothing external to ask, so its inbound leg is submitted through the box's own port and only exercises the local route, and the probe says so in its logs. Its outbound leg is the one that matters there: a real send to a third-party sink over a real MX lookup.

Assert all three verdicts, not any of them. A partial pass is how a broken SPF record ships unnoticed.

Getting a password out

Every credential in this stack is generated and never displayed, which is right, and it leaves one real gap: setting up a mail client needs the value. That is what export-passwords is for.

beet --entry=infra/mail.bsx --stage=prod export-passwords

It composes the parameter names off the declaration rather than making you type /acme/prod/mail-account-<localpart>-at-<domain-with-dots-as-hyphens> from memory, which is the part that goes wrong. It lists every mailbox on every served domain plus admin@, the account the server itself creates when the data store is first claimed and which no declaration names. --infra adds the relay credentials in use and the DKIM private keys, kept behind a flag because reading a mailbox password is setting up a client while relay and signing credentials are infrastructure access.

It prints secrets to stdout, deliberately and uniquely in this stack. Mind what is recording your session.

The hand version, for when you want one value and not all of them:

beet secrets/exec -- aws ssm get-parameter --region ap-southeast-2 \
  --name "/acme/prod/mail-account-pete-at-stalwart-example-com" \
  --with-decryption --query Parameter.Value --output text

Setting up a client

Thunderbird took the address and the password and discovered everything else from the autoconfig records, and ticking its address book and calendar sync worked against the same 443 with the same credentials, so CardDAV and CalDAV came along for free. The one prompt that needs explaining is the bare "mail.example.com requests username and password" from the DAV sync: it wants the full address as the username, with the same password.

The soak, and a realistic expectation

Send a message to a personal Gmail address and read the headers. With SPF, DKIM (both signatures), DMARC under p=reject and a TLS 1.3 hop all passing, Gmail still filed our first message from the days-old domain into spam.

That is the correct outcome and it is worth stating plainly: authentication earns deliverability, and only history earns the inbox. There is no configuration that shortcuts it. The remedy is the soak itself, which is to mark it not-junk and then correspond normally for a couple of weeks. Plan for a partial repeat at cutover, since Gmail keys reputation off the From and DKIM domain, and that domain changes at the apex swap even though the SES addresses, the box and its reverse DNS all carry over whole.

Two more soak-period notes:

The box logs to two files, and the agent tails both into one CloudWatch group: stalwart.log is the unit's stdout (bootstrap mode, the backup scripts, a panic) and the daily-rolled stalwart.<date> is the claimed server's own tracer, declared without ANSI colour so the timestamp leads every line and the agent dates events at write rather than at ingestion. The agent follows only the newest file matching a glob, which is exactly right for a rolled series; our first week shipped nothing but bootstrap lines because only the first file was tailed.

watch's port-25 banner check cannot pass from a residential connection. ISPs block outbound 25 as an anti-spam measure, so the check times out looking exactly like a dead server while SES delivers through that same port perfectly well. Verify the banner from the box itself (exec 3<>/dev/tcp/localhost/25 && head -1 <&3), or trust the probe, and run watch from somewhere with port 25 egress if you run it at all.

dig is not on every machine. The dependency-free substitute, which is also the only check that means anything for DNS because it asks something other than your own provider's API:

curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=<name>&type=TXT'

Hand step. Enable DNSSEC on the zone (PATCH /zones/<id>/dnssec {"status":"active"} on the Cloudflare API). If Cloudflare is also your registrar the DS record publishes to the parent automatically, but not quickly: ours sat pending for just over a fortnight (enabled 2026-08-28, at the .com parent by 2026-09-15) with nothing to do but re-query, and the support ticket we had drafted in our heads was never needed. Verify with a DoH query for type=DS at the apex, and an apex MX query checking the AD flag; GET /zones/<id>/dnssec reports active once the parent has it. The zone keeps resolving unsigned-style until the DS lands, so mail is unaffected either way, and until it lands there is no DANE to be had.

Hand step, and a judgement call. MTA-STS ships in testing mode. Flip it to enforce after two clean weeks of TLS-RPT reports and not before. A testing policy withholds no mail whether or not it is fetched; an enforce policy withholds mail from senders whose TLS your box fails to satisfy, which is the point of it and also the risk. Expect the report stream to be EMPTY rather than clean: a reporter (Google and Microsoft, in practice) writes only for days on which it delivered to you, so a domain nobody has mailed yet produces no reports at all, and two weeks of silence is not two weeks of evidence. When that is where you are, check directly what a report would have checked for you: the policy's mx: equals your MX record; port 25 offers STARTTLS with a certificate for that exact name from a publicly trusted CA (openssl s_client -starttls smtp -verify_hostname <mx> -verify_return_error, run on the box against 127.0.0.1:25 with -servername, since neither a residential ISP nor EC2's default egress block will let you reach 25 from outside); the policy fetches over valid TLS with no redirect; and an outside vantage on 25 agrees, for which internet.nl is free and checks trust chain and name match. Then flip, knowing the shape of being wrong: an enforce failure is a deferral at the sender, not a bounce, and the way back is the one attribute and a deploy, which bumps the _mta-sts id and has senders re-fetch. One message from a Gmail account to your domain will provoke a Google report the next day, which is the cheapest way to have enforce judged after the fact. Also note that a wrangler custom domain's certificate is not instantly live at every edge: our deploy verified both policy fetches, a curl two minutes later got a TLS error from the same machine, and five minutes after that it was clean. Do not roll back over that.

You are now a reporter, whether or not you meant to be

A server that checks DMARC on inbound mail also generates DMARC reports on outbound. Nothing in the build asks for this and no tag declares it. It is simply what a conforming receiver does, and it starts the day your MX goes live.

The day after our apex cutover, two SES bounce notifications arrived in a personal inbox. Both were reports our own box had sent: a DMARC failure report, and a day later the aggregate report covering the same window. Both were addressed to the dmarc@ mailbox that a heavily spoofed finance domain publishes for rua= and ruf= alike, and both were bounced by that domain's provider.

The alarming reading is the wrong one. At the volume a new mail server sends, one bounce a day against two or three messages looks like a 50% bounce rate and an account about to be suspended. It is not. AWS does not count transient bounces toward the reputation metric, and ours read 0.0 on both days, with EnforcementStatus: HEALTHY and an empty suppression list throughout. Read the metric before you react to the mail:

aws cloudwatch get-metric-statistics --namespace AWS/SES \
  --metric-name Reputation.BounceRate --period 86400 --statistics Maximum \
  --start-time <from> --end-time <to>

What is worth acting on is the split between the two kinds of report. Aggregate (rua) reports are one message per reporting domain per day carrying counts. They are how the mechanism works at all, and they are what answers the rua= you publish yourself. Failure (ruf) reports are one message per spoofed message, they forward the headers of mail addressed to your users out to whatever address a third party published, and almost nothing consumes them. Their volume is a function of how much spam you receive, which is not a quantity you control.

So send the aggregate half and not the failure half. On Stalwart 0.16 that is three singletons, x:DmarcReportSettings plus the pre-DMARC x:DkimReportSettings and x:SpfReportSettings, and one constant:

{ "aggregateSendFrequency": { "else": "daily" },
  "failureSendFrequency": { "else": "disable" } }

Leave TLS-RPT alone. It is aggregate, it is daily, and it is the evidence the MTA-STS enforce flip above is judged against.

Two details that generalise past this server. A frequency is an expression constant, so it is unquoted where a string literal in the same position would carry its quotes, and a quoted 'disable' is a string that no constant matches, which leaves the shipped default silently in place. And the two settings are off by different mechanisms: the aggregate one has a real "never" value, while the failure ones are [count, period] rates whose conversion rejects anything that is not a pair of positive numbers, and the send path reads that rejection as off. Both spell it disable, and only one of them means it.

One last thing the bounce notifications are worth checking for. The report sender is noreply-dmarc@ at the server's default domain, which on a box serving several domains is decided by something you probably did not think of as a decision. See section 6's note on declaration order.

8. Cutover

This section is a lived path: our own cutover ran on 2026-09-01, and the timings below are what it measured. From the first hand DNS change to a fully verified apex was about fifteen minutes.

The cutover changes which domain the working system serves and nothing else. The box, its IP, its reverse DNS, its certificate and its banner are all unchanged, which is why this is a short window rather than a second build.

Ahead of the window, each of these is safe on its own and none of them move mail:

  1. Add the apex MailDomainBlock with records={MailRecords::IdentityOnly}, so the SES identity, DKIM and configuration set exist and verify while every record that would move mail stays unpublished. Ours verified in under a minute, weeks ahead.
  2. Write the rollback: the exact incumbent MX and SPF values, captured from the live zone rather than from memory, ready to paste back. Keep the incumbent account paid and live until the window is a week behind you.
  3. Confirm the TTLs rather than lowering them on faith. On Cloudflare, records with the proxy off are TTL 1 meaning "auto", which is authoritative 300 seconds; there was nothing to lower.
  4. Run plan and read it. Ours showed the base AMI had drifted, which would have rebuilt the box inside the cutover apply. If the plan contains anything that is not a DNS record, deploy that part FIRST as its own baseline deploy, with the apex still IdentityOnly: it absorbs the churn and proves the whole path end to end while the incumbent still holds the apex, and the cutover apply then replaces nothing.

Do NOT provision the apex mailboxes weeks ahead. Mailboxes need the domain on the server, and a local domain hijacks every submission addressed to it away from the MX the world still resolves. They land at the start of the window instead, which shrinks that hazard to minutes.

The window itself. Ours resolved the one open mechanical question as: one records="All" deploy, with the hand steps interleaved around it, and the hand steps only ever delete records so nothing collides with what terraform creates:

  1. Hand-create temporary autoconfig.<apex> and autodiscover.<apex> CNAMEs to the box. Provisioning a served domain issues its ACME certificate, and the order dies on names that do not resolve, so these two must exist before the records that will own them are applied. Then flip the entry: records="All", members, mailboxes, aliases.
  2. Run provision (no apply, no DNS change). The apex becomes local, the mailboxes and role aliases exist empty, and the certificate issues against the temporary names. From here until the MX swap, submissions from your own accounts to apex addresses land locally instead of at the incumbent; ours lasted about two minutes. The win: the box accepts apex mail from the first second the MX lands, so there is no window where your published MX 550s inbound mail.
  3. Delete the two temporary CNAMEs and delete the incumbent's apex SPF TXT, then immediately run deploy. The apply publishes the SPF replacement, _dmarc, _mta-sts, _smtp._tls, the autoconfig and autodiscover CNAMEs, the SRVs and the apex MX in one run. There must be exactly one SPF TXT at the apex at every moment: the delete-then-apply is the replace.
  4. The moment the apply completes, delete the incumbent's MX records. Do not rely on the zone audit to catch a missed one: the audit matches declared records by name and type, and the incumbent's MX shares both with yours. Enumerate the zone raw and look.
  5. Verify from the outside: DoH the MX, SPF, DMARC and _mta-sts; fetch the MTA-STS policy over https; then round-trip real mail both directions (below). Watch the old mailbox for stragglers for a few days; sender retries cover the propagation gap in either direction.
  6. Days later, not in the window: retire the incumbent's DKIM CNAMEs and drop them from the audit allowlist, so the audit goes back to asserting that the zone is exactly yours. Keep the staging addresses for a deprecation window, then remove the staging block and let the audit clean up its records.

The rule behind the ordering is: sending first, receiving second. A half-applied cutover then degrades to "we can send but mail still arrives at the old provider", which is inconvenient, rather than to mail arriving nowhere.

What the window's round trip caught

Verify the cutover with the worst-case message, not the best one. Our post-cutover round trip (curl submission to a throwaway inbox, raw bytes back over an API, dkimpy verifying every signature by index) found the sovereign DKIM signature failing on real mail while both of SES's signatures passed and DMARC stayed green, the same hiding pattern as the Message-ID rewrite the soak had already caught, one layer deeper. SES re-parses and re-emits the address headers, so the bare angle form (To: <a@b.c>, the shape every script and agent composes) is rewritten to To: a@b.c in transit, and a signature covering To, Cc or Reply-To dies with it. The display-name form survives, which is why a round trip from a mail client passes and one from curl does not: test with both, or you will prove the form your users happen to use and ship broken signatures for the rest.

The diagnosis method generalises: send a message local-to-local (it never touches the relay) and verify the signature on the raw stored copy. If pure server output verifies and relayed output does not, diff the delivered covered headers against the stored ones byte by byte; the rewritten header is visible to the eye. The fix is the same shape as before (the signed set contains no header the relay rewrites, which for address headers means none but the RFC-mandatory From) and it ships as a provision patch in under a minute.

9. Backups, and the drill

Start with the paragraph that justifies everything else in this section.

Our first restore drill found that the nightly dump had never once produced a backup. Every check the stack makes was green: the timer was scheduled, the unit existed, the deploy passed, the zone audit was clean. The bucket had been empty since the day the timer was armed and nothing anywhere said so.

Nothing observes a backup except restoring one. Read that before you read any of the machinery below.

The same shape bit the layer above it. <StalwartSnapshot/> had been in the deploy route through a whole migration without ever once running, because the volume it snapshots did not exist until late and every deploy before that took the first-deploy skip. The first time it genuinely fired it died at the aws cli on an option that does not exist, and the account had zero snapshots to show it. A step that has never executed is not a step that works, however carefully it reads; the tell is an artefact count, not a green deploy.

The layers, ordered by how much losing them would hurt:

  1. The data volume. Stalwart's SQLite database is at /var/lib/stalwart/stalwart.db on its own encrypted gp3 volume, which is not the box. In production it carries prevent_destroy and asks AWS for a final snapshot, so a stray destroy cannot eat the mail. This is the layer that replaced RDS's 14-day point-in-time recovery, and losing that window is the one thing this trade genuinely gave up.
  2. The pre-apply snapshot.<StalwartSnapshot/> sits immediately before the full <TofuApply/>, finds the volume by its Name/Project/Stage tags, and waits for a complete EBS snapshot before an apply is allowed to replace the box. A retry within one deploy finds that deploy's own snapshot by its DeployId tag rather than taking a second, because CreateSnapshot offers no idempotency token to use instead (only the multi-volume CreateSnapshots has one). Nothing else prunes a snapshot lineage, so the step prunes its own: the newest snapshot_retain (3 by default) of the snapshots it tagged survive, and a snapshot you took by hand before something frightening is never one of them.
  3. The Recycle Bin. A deleted snapshot or data volume is not gone at once. The repo's account (<AwsAccount/>) holds region-level Recycle Bin rules, locked, so a deleted snapshot stays restorable for snapshot_retention_days (14 by default) and a deleted volume for volume_retention_days (at most 7); a locked rule changes only after an unlock and its delay, and the deploy credential may not unlock one. This covers the layers above against a mistake or a leaked deploy key, and nothing against the account itself.
  4. Off-box. A systemd timer on the box runs SQLite's online .backup, checks PRAGMA integrity_check on the result, uploads it under sqlite/ in the archive bucket, downloads it again, compares the bytes and re-verifies the copy that came back. The backup runs on the box rather than from a deploy machine, because a backup that only happens while somebody is deploying is not a backup. The sqlite/ prefix expires at 180 days, which is only safe because the live database is on the volume and not in that bucket.
  5. Blobs. S3 versioning, a lifecycle expiring noncurrent versions, a public access block and server-side encryption. Message bodies were never in the database and are not in these snapshots.
  6. Off-account. Everything above lives in the one AWS account, so a compromised credential or a closed account takes the record and every copy of it in the same afternoon. <R2BucketBlock label="cold-backups" location="weur" retain_days=30/> is a bucket in Cloudflare's account, declared by the same apply; the box's cold_bucket names it and gains a second nightly timer that copies the newest snapshot and every blob across with rclone, reading both sides back and comparing bytes before it calls the snapshot copied. Copy, never sync: nothing in that script deletes and every copy is --ignore-existing, so a deletion upstream stops at the archive instead of reaching it and this side only ever appends. R2 keeps no object versions, so retain_days is the undo's replacement: a bucket lock R2 enforces itself, refusing a delete or an overwrite of anything younger than its window to every credential that reaches the objects, the parked token and a lifecycle rule included. A bucket declaring no window is refused at render unless it says accept_data_loss=true. The section below is about this layer, since it is the one with a human step in it.
  7. Everything else regenerates: config from git, the box from user data, secrets re-mintable. The exceptions are the DKIM private key once its selector is published and any relay credential a human minted by hand, so every deploy and provision ends by exporting the stack's whole secret store into a secrets document, twice: the committed infra/secrets/<app>--<stage>.toml in the repo (<SecretsExport document={$mail_secrets}/>, overwritten only when a secret changed, so git holds the history) and a dated series in the cold bucket (<SecretsExport document={$mail_cold} dated=true/>). Both are sealed with age to the humans listed in the entry's own secrets.toml, one identity per person, each living with its human and in neither cloud; beet secrets/restore is the inverse, and the secrets tutorial is the whole story.

The last rung is still manual: rclone copy :s3:<cold-bucket> <local> from hardware you control, initiated from outside both clouds. The cold copy closes "AWS account lost"; it does not close "every cloud credential lost at once". The token that writes it can also delete it (R2 tokens are read or read-and-write, with no append-only level), and what the lock buys is the window: nothing younger than retain_days can be deleted by it, which is exactly the copies a restore would use, and nothing older is protected at all. The lock itself is bucket configuration rather than data plane, so the parked token cannot lift it, and neither can the deploy token, which only reads bucket configuration: the write that could is held for one elevated deploy at a time, behind the dashboard login (section 3). Say that plainly in your own runbook rather than letting the bucket's existence imply more than it holds.

Who mints the cold copy's credential

The bucket answers to an R2 api token, which no IAM policy can grant, so one is minted for it alone: an account-owned token scoped to that one bucket's objects, whose id is the S3 access key id and whose value's SHA-256 is the secret. Both are parked as SecureString parameters under the stack's prefix (cold-backups-access-key-id, cold-backups-secret-access-key), the same way the box's SES relay pair is parked, and every cold verb reads that pair: the box's timer through its instance role (the bucket block's grants are exactly those two parameter reads), and the deploy machine's export, push, probe and drill through the same names.

The apply does not mint it, and that seam is the point. Creating a token needs Account API Tokens Write, so an apply that minted here would put the one escalating Cloudflare group into the deploy credential of every repo that declares a bucket, with nothing to cap it (section 3). The elevated deploy mints it instead, as the mint token, and converges: the repo's own deploy token first, then one token per declared bucket, parking the secret half before the access key id so a crash between the two writes heals on the next run rather than leaving a pair that looks current. Rolling it is --elevated --roll on the same deploy. If the pair is not parked, R2BucketBlock::missing_credential is what every deploy-machine consumer says, naming that deploy, and the box's own script echoes the fixed R2BucketBlock::UNPARKED, since anything a person may reword stays out of user_data.

Everything else in the stack is still minted by an apply, scoped to one resource, parked in the stack's secret store and read through a grant: the SES user and its access key, the DKIM key, the mailbox credentials. The version of this section that had you minting the R2 token in the dashboard and running aws ssm put-parameter by hand worked, and was the anomaly, a deploy credential with a human inside its loop, and the version that had the apply mint it was the opposite anomaly, a daily credential holding the one permission that can mint any other.

The one credential the stack does not mint is the age identity the exports are sealed to, because its whole point is to be held by a human off both clouds: one per person, the public halves listed in the entry document's default group, which every export copies. Making one, backing it up, adding and removing a person, restoring an export into a fresh account and reading one with the age command line alone are the secrets tutorial.

Then the deploy tail runs three steps after the mail probe, every time: the two <SecretsExport/>s, <MailColdPush/> (which starts the box's nightly unit right now over ssh and waits for it, so the path is exercised before the timer ever fires) and <MailColdProbe/>, which reads the newest snapshot back from the cold side and the live side and compares digests, does the same for one blob, checks the snapshot was taken within 36 hours by the clock in its own key (its last-modified in the cold bucket would only say when it was copied), and opens the newest export with your identity to prove it is this stack's and readable. mail/cold-probe is the same check on any other day.

The bug, because it generalises

A terraform reference resolved before the user data escape pass. The whole user data string is escaped (${ becomes $${) and then the deliberate references are substituted in. A helper that resolves its own reference runs before that escape, so the escape turns a live reference into literal text, and the box once ran pg_dump --host '${aws_db_instance.x.address}', which fails on every line of a name resolver and is discovered a fortnight later by a restore that has nothing to restore. Postgres is gone and the trap is not: build_user_data still makes exactly one late substitution, now the data volume id, and a box that mounts /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_$${aws_ebs_volume...} waits ten minutes and then refuses to boot. Every step that composes machine config must leave a token for the one late substitution, never a reference. Diagnosis is one line: sudo cat the script on the box and look at it.

Useful commands: systemctl list-timers stalwart-backup.timer says when a timer will next run and whether it ever has (LAST is - if it never has), and systemctl start <unit> runs a timer's unit right now without disturbing the schedule. Verify by listing the bucket, not by the unit's exit code:

systemctl list-timers stalwart-backup.timer
sudo systemctl start stalwart-backup.service
aws s3 ls s3://<archive-bucket>/sqlite/ --recursive

The drill

The drill is a whole parallel stage, not a spare database file. deploy it, restore-drill into it, destroy it. A non-production Stalwart volume is disposable by default precisely so this works: the drill tears down the same declaration production protects, without either stage knowing about the other. The destroy is whole: after tofu destroy has taken what terraform owns, <StackTeardown/> sweeps the stage's parameter store prefix, so the admin password, the DKIM key and the mailbox credentials the actions minted go with the stack and nothing is left to delete by hand. That sweep is unconditional, on production too, which is why the secrets exports at the end of every deploy exist: a destroyed stack's secrets live on in the committed repo document and the cold bucket's dated series, and nowhere else.

Run it twice if you keep a cold copy: restore-drill restores the archive's newest snapshot and restore-drill-cold (<MailRestoreDrill source_snapshot="Cold"/>) pulls the same key back out of the other vendor's bucket with the source stage's parked token, and everything after the download is identical. Until the cold one has passed, the cold bucket is an upload rather than a backup.

The drill refuses to restore a stage over itself, since replacing the live database is the incident. The day that IS the point (a box whose store is lost, or a stack moving into a new AWS account) the same restore runs over the stage's own box as <MailRestore/>, and asks for both halves of the decision on the command line: the snapshot by key, since the newest is whatever ran last, and --replace. A move stands the new box up with nothing resolving to it first, the storage and box layers alone (<TofuApply layer="box"/>), restores the old box's final snapshot by key, and only then runs the full deploy, so the first message the new box accepts lands in the restored store; the SES identities can go ahead of all of it (<TofuApply layer="ses"/>), since AWS reviews production access only for a verified identity and an Easy DKIM record is per account.

Three things about building one over a shared zone, each of which is a consequence rather than a preference:

The drill stage must not own any shared DNS name. That is what dns_stage is for, and it is why the drill needs its own entry file with drill-scoped names, including a throwaway mail domain that exists only so that the box can obtain a certificate.

The drill entry must not declare production's mail domains at all, even guarded. An SES identity is account-global, so a second stack naming a domain the live stack owns fails its apply outright. Name the source domain as a parameter instead.

What it must share is its labels and its app name. A resource name composes as <app>--<stage>--<label>, so identical labels resolved against the source stage are how the drill finds production's backup bucket and the credential it signs in with.

Then the thing that reframes the whole exercise, which only appears once you actually run one. Stalwart 0.16 keeps its configuration in the data store, so a restore carries the source's entire identity: hostname, domains, listeners and certificates all live in the database the mail lives in. Ten seconds after the restore the drill box had stopped answering to its own name and was serving production's certificate for production's names.

A restore of this kind is not "the data came back". It is "the server came back", and restoring production's snapshot onto a second box produces a second production. Your runbook should say so.

Two consequences:

The probe cannot be the drill's assertion. A restored store carries the source's domains and accounts, so the drill box serves probe@<source domain> while that domain's MX still points at production: the probe's inbound leg would be answered by the live box and pass without the drill having restored anything. What is provable is that a restored account authenticates against the drill box and its mailbox is readable, which is what the drill action asserts itself.

Reaching it means forcing the address. No name both resolves at the drill box and is covered by the certificate it now holds, so curl --resolve <autoconfig.source-domain>:443:<drill ip> is how you dial it, with TLS verification on, against the restored certificate. Passing that is part of the proof: a box that had restored nothing could not present that certificate. Resolving the name normally would reach the live box and pass having tested nothing.

Smaller things the drill taught:

10. The box's identity on the wire: IPv6 and DANE

Two things the free checker at internet.nl will fail a fresh mail domain on, both of which are one attribute once the pieces beneath them exist. Neither carries mail on its own: outbound rides the relay, and inbound works without either. They are about what a careful sender sees, and they are also where a mistake costs delivery rather than a score, so both are built to be derived rather than typed.

IPv6

ipv6=true on the VpcBlock gives the vpc an Amazon-provided /56, each public subnet a /64 at the same index as its /24, and the public route table a ::/0 to the gateway. The private tier stays v4-only, on purpose: a v6 address is globally routable, so a private subnet with one has exactly the path off the vpc the tier promises not to have. The box then takes one address out of its subnet's /64 (an in-place change on a running instance, no rebuild) and publishes an AAAA beside its A, read off the instance. Stalwart already binds [::], so that is the whole of the box's side.

What it changes: senders that prefer v6 reach you over it, which is inbound only in practice since outbound leaves through the relay, and a broken v6 path degrades to a sender's v4 fallback rather than to loss. Two things to know. Let's Encrypt validates over v6 first when an AAAA exists and falls back to v4 only on a network error, so the first renewal after publishing the record is the first real test of the v6 path; the restore drill, deployed dual-stack, rehearses a fresh issuance with an AAAA present every time it runs. And your deploy machine probably has no v6 (residential connections mostly do not), so verify the address from the box (ip -6 addr, curl -6 against its own 443) and the reachability from outside with internet.nl, not from your desk.

DANE

DANE is a TLSA record beside the MX target, at _25._tcp.<hostname>, naming the key the box serves on port 25. A validating sender (a minority, mostly European: Posteo, mail.de, the Dutch and German public sectors, Comcast) refuses any other certificate for that name outright, and unlike MTA-STS there is no testing mode and no report stream. That is the whole hazard: a pin that stops matching the served key is refused sessions, kept up until the record or the key changes. Everything below follows from refusing to let those two drift.

The precondition is DNSSEC on the zone, active at the parent: an unsigned pin is ignored by every validator (section 7 has the fortnight the DS took to reach the parent). The record is 3 1 1, the SHA-256 of the leaf's public key: the leaf rather than the issuer, because Let's Encrypt rolls intermediates unannounced; the key rather than the certificate, because a key can survive a renewal and a certificate cannot.

Can survive. Read your server's ACME code before publishing a pin, because whether the key is reused across renewals is the whole design question. Stalwart 0.16 mints a fresh P-256 pair per renewal by default and reuses the stored certificate's key only when the ACME provider's reuseKey is set; provision sets it, so from then on the pin is a one-time publish per key. The key still changes on a fresh data store (a destroy and redeploy, a migration), and provision is where a fresh store gets its first certificate.

So the pin is read after provision, never before it, and never typed. <MailDane/> runs on the box, reads the digest port 25 presents to a peer that sends SNI and to one that does not, parks it in parameter store, and the second <TofuApply/> publishes it; the record reads that parameter as content whose absence publishes nothing, so a fresh stack renders no TLSA at its first apply and exactly one at its second, and a stack whose key did not change applies a no-op. dane=true on the StalwartBlock is the whole declaration. The standing check is in watch: the published pin over DNS-over-HTTPS with the AD flag, against the key 443 serves for the hostname (25 is unreachable from most deploy machines, and the two ports serve one certificate for one name).

The finding that made the "with SNI and without" part necessary: a server holding one certificate per domain and no default answers a peer that sends no SNI with whichever certificate its hash map yields first, which is random per process. This box was serving the newsletter domain's certificate on port 25 to any peer that named nothing, a name that covers neither the MX nor the pin, and the phase that flipped MTA-STS to enforce had checked with SNI and, by luck of the seed, without. Not every MTA sends SNI. Provision now sets the server's default certificate to the one covering the box's own hostname, and the pin step refuses to publish while the two handshakes disagree. If you run more than one domain on a box, check the no-SNI handshake yourself before you trust either MTA-STS enforce or a pin: openssl s_client -connect 127.0.0.1:25 -starttls smtp -noservername on the box.

The way back from a wrong pin is the way back from everything else here: one deploy, which re-reads the served key and republishes. What you cannot do is turn the failures into a report, so a pin is a promise you make with the monitoring you already have. internet.nl will note the absence of a rollover scheme (a second TLSA for the key you will move to next); there is none because the key does not move under reuseKey, and when it does move, on a fresh store, the deploy that moved it publishes the new pin before it ends. Ours went from the two failures in section 7 to 100% on the deploy that added both, with the DANE record shown as 3 1 1 and valid.

What is automated, and what is not

The honest summary, because a tutorial that blurs this line strands its reader at the first step with no command.

Fully declared and applied by beet: the repo's own AWS account, with its locked Recycle Bin rules, its read role and its state bucket; the network (dual-stack if asked), the persistent SQLite data volume, both buckets, the box and its machine config, the events topic and its publish policy (adopted from a hand-made one with a flag), every SES identity and its DKIM and MAIL FROM records, the configuration sets with their suppression and their event destinations, the reputation alarms, every DNS record the mail domains need, the box's A, AAAA and DANE pin, the mail server's entire configuration (listeners, routing, domains, accounts, aliases, certificates, the default certificate, key reuse across renewals), reverse DNS, the MTA-STS policy host and body, the delivery probe, the zone audit, the backup timer, the credential export, the restore drill and its snapshot pruning, and the off-account cold copy: its bucket, the scoped token that reaches it, its timer, the encrypted secrets export, the read-back probe and the cold restore.

Genuinely not automatable, and labelled as such:

Hand steps that are hand steps only because of ordering: everything in section 5. The blocks declare all of it; you do it by hand the first time because the support reply needs it to be true before the stack exists.

Gaps, which is to say the backlog:

Where to go next

beet_infra covers the blocks, the terraform export and the deploy lifecycle these verbs sit on. If you have not written a beet router before, A guestbook from a scene is half an hour and shows why a command and an HTTP route are the same thing.