Terraform State Management at Scale: Drift Detection and Safe Plan/Apply Pipelines (2026)

Terraform’s configuration files are the part of your infrastructure everybody reviews. The state file is the part that actually decides what happens. It is the only record that says “this aws_db_instance.main block corresponds to that particular database in that particular account”, and every plan is a diff computed against it. Lose it, corrupt it, let two pipelines write it at once, or leak it, and no amount of tidy HCL will save you. At ten resources this is a footnote. At ten thousand resources spread across forty teams, terraform state management becomes the dominant engineering problem in your infrastructure-as-code programme.

This matters now because the tooling around state has changed more in two years than in the previous eight. S3 gained native locking, and the Terraform S3 backend documentation now marks DynamoDB-based locking as deprecated. OpenTofu shipped client-side state encryption. Write-only arguments and ephemeral values let secrets stay out of state entirely. Pipelines moved from “someone runs apply on a laptop” to PR-driven plan and apply with policy gates. Readers who finish this article will be able to design a backend, split state along sensible blast-radius lines, detect and reconcile drift on a schedule, and build a plan/apply pipeline that fails safely.

What this covers: what state is and why it is a liability, backend and locking mechanics, state-splitting strategy, drift causes and detection, import, moved and removed blocks, OpenTofu encryption, PR-based pipelines and policy as code, provider and module pinning, failure modes with recovery recipes, and how GitOps and Crossplane change the picture.

Context and Background

Terraform (and its fork OpenTofu) is a declarative engine with three inputs: your configuration, the real world as reported by provider APIs, and the state. The state maps each resource address in configuration to a remote object ID and caches the attributes last seen. During plan, the engine refreshes those attributes from the provider, compares them with configuration, and emits a diff. Without state the engine cannot tell a resource it should create from one it should adopt, nor one that was removed from configuration and must be destroyed.

The incumbent options for storing state have a clear lineage. Local files work for one person. Remote backends (Amazon S3, Google Cloud Storage, Azure Blob Storage, HCP Terraform, Consul, PostgreSQL and others) give shared storage plus, in most cases, locking. HashiCorp’s licence change in 2023 produced the Linux Foundation fork OpenTofu, which has since diverged on features such as state encryption; our companion piece on Terraform 1.16 versus OpenTofu 1.13 and the IaC fork divergence covers that split in detail. In this article I note differences where they affect state, and otherwise use “Terraform” for both engines.

The failure data for state is mostly anecdotal, but the categories are consistent across incident write-ups: concurrent applies that clobber each other, state files that grew until every plan took twenty minutes, secrets that leaked through a bucket policy, and drift that nobody noticed until a plan proposed to destroy a production database. The official guidance on sensitive data in state is blunt: state and plan files can contain secrets such as database passwords and API tokens, and marking a value sensitive only redacts it from CLI output and the UI, not from the file.

The rest of this article treats state as a production database with a schema, an access-control model, a consistency story and an operational runbook. That framing explains most of the design decisions that follow.

State as a Liability: The Reference Architecture

Short answer: treat Terraform state as a sensitive, write-contended production datastore. Keep it in a remote, versioned, encrypted backend with locking; split it into many small states aligned to blast radius and ownership; run every change through a pipeline that plans on pull request, applies from a saved plan under a lock, and schedules refresh-only plans to detect drift.

That paragraph is the thesis. The architecture in Figure 1 shows how the pieces connect.

Figure 1: Reference architecture. Engineers change code through pull requests; a pipeline plans and applies against a remote backend with locking; a scheduled scanner runs refresh-only plans and raises drift tickets.

In the diagram, engineers never hold write credentials to production state. A CI runner assumes a role per environment, takes the backend lock, produces a saved plan, and applies only that plan after approval. A separate scheduled job uses a read-mostly role to detect drift. Policy checks sit between plan and apply. Everything that mutates state flows through one funnel, which is what makes audit and recovery tractable.

What the state file actually contains

The state is a JSON document with a version, a serial, a lineage UUID, a map of outputs and a list of resources. Each resource instance records its address, provider, schema version and a flat attributes object holding every attribute the provider returned. The serial increments on every write; the lineage identifies the state’s origin so that Terraform refuses to overwrite a state from an unrelated history. These two fields are your first defence against stale writes and are worth knowing when you do surgery.

The key property is that attributes are stored verbatim. A random_password result, an RDS master password, a TLS private key from the tls provider, an access key secret: all land in plaintext JSON unless the provider uses write-only arguments. The official documentation explains that write-only arguments (typically ending in _wo, with a matching _wo_version) and ephemeral values are omitted from state and plan files, and that write-only arguments require Terraform 1.11 or later and are defined by each provider. Adoption varies by provider and resource, so check the provider docs before assuming a secret stays out of state.

Why blast radius is the real design variable

A single state file is a single lock, a single refresh, a single permission boundary and a single failure domain. If one state holds networking, databases, Kubernetes clusters and application IAM, then a typo in an IAM policy runs a plan that must refresh every resource, holds a lock that blocks every other team, and runs with credentials broad enough to destroy the network. Splitting state is therefore not about elegance. It buys four things at once: shorter plans, parallel work, narrower credentials and smaller worst-case mistakes.

A useful rule of thumb, which I offer as opinion rather than standard, is that a state should change at one cadence, be owned by one team and fit within one credential scope. A VPC changes monthly, belongs to the platform team and needs network-admin rights. An application’s service and its autoscaling group change daily, belong to a product team and need narrower rights. Putting both in one state makes the slow thing pay for the fast thing’s churn.

Remote backends and their consistency models

The backend determines three things: where bytes live, how writes are serialised and how access is controlled. The common choices differ along those axes.

Backend	Locking	Versioning / recovery	Access control	Notes
S3 (`use_lockfile`)	Native lock object (`.tflock`) via conditional writes	Bucket versioning recommended	IAM plus bucket policy, SSE-S3 or SSE-KMS	Current default for AWS shops
S3 + DynamoDB	DynamoDB item with `LockID`	Bucket versioning	IAM on bucket and table	Marked deprecated in Terraform docs
GCS	Native lock object	Object versioning	IAM, CMEK supported	Simple and robust
Azure Blob	Blob lease	Blob versioning / snapshots	RBAC, SAS, Entra ID	Lease-based locking
HCP Terraform	Managed run queue and workspace lock	Built-in state versions	Teams, workspace permissions	Runs execute remotely
PostgreSQL (`pg`)	Advisory lock	Database backups	Database roles	Fine for small self-hosted setups

Treat this table as orientation, not a guarantee; backend behaviour changes between releases, so verify against the documentation for the version you run.

Backends, Locking and Encryption in Practice

S3 native locking versus DynamoDB

For years the canonical AWS recipe was an S3 bucket for the state object plus a DynamoDB table with a LockID partition key for locking. The reason is historical: S3 lacked a compare-and-swap primitive, so locking needed a second system. After S3 added conditional writes, the Terraform S3 backend gained a use_lockfile argument. The current S3 backend documentation states that use_lockfile defaults to false, that DynamoDB-based locking “is deprecated and will be removed in a future minor version”, and that when the lock file is used s3:GetObject, s3:PutObject and s3:DeleteObject are required on the lock object, whose example path is path/to/my/key.tflock.

Mechanically the lock is an object created with a conditional write that fails if the object already exists. The first writer wins; later writers see a precondition failure and Terraform reports the lock holder’s metadata (who, when, operation, lock ID). Releasing the lock deletes the object. That removes a whole AWS resource, its IAM permissions, its capacity mode and its backup policy from your bootstrap code, which is a real operational saving.

OpenTofu documents the same use_lockfile option in its S3 backend, describing it as using conditional writes with the If-None-Match header, and states the team has no plans to deprecate the DynamoDB option. The two engines therefore differ in direction of travel here. If you share modules or pipelines across both engines, keep the locking configuration in a per-engine backend file rather than hard-coding it.

# backend.tf (Terraform, AWS)
terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod-eu-west-1"
    key          = "network/core/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:eu-west-1:111122223333:key/EXAMPLE"
    use_lockfile = true
  }
}

The bucket itself deserves bootstrap discipline. Enable versioning (the docs call it “highly recommended” for state recovery), block public access, enforce TLS in the bucket policy, enable access logging or CloudTrail data events for the state prefix, and apply a lifecycle rule that expires noncurrent versions after a window you can defend. The kms_key_id argument requires kms:Encrypt, kms:Decrypt and kms:GenerateDataKey on the key, so grant exactly those to the pipeline role and nothing broader.

# Minimal IAM policy fragment for a pipeline role (illustrative)
data "aws_iam_policy_document" "state_access" {
  statement {
    actions   = ["s3:ListBucket"]
    resources = ["arn:aws:s3:::acme-tfstate-prod-eu-west-1"]
    condition {
      test     = "StringLike"
      variable = "s3:prefix"
      values   = ["network/core/*"]
    }
  }
  statement {
    actions = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
    resources = [
      "arn:aws:s3:::acme-tfstate-prod-eu-west-1/network/core/terraform.tfstate",
      "arn:aws:s3:::acme-tfstate-prod-eu-west-1/network/core/terraform.tfstate.tflock",
    ]
  }
}

Scoping the role to a single key prefix is what turns “split state” from a convention into an enforced boundary: the network pipeline cannot read or overwrite the database state even if its code is compromised.

What locking does and does not protect

State locking serialises operations that write state. It stops two applies from interleaving their writes. It does not stop two engineers from planning against the same stale base, it does not make a partially failed apply atomic, and it does not protect resources from changes made outside Terraform. Many incidents blamed on “locking failures” are really plan staleness: a plan generated at 10:00 is applied at 10:40 after someone else’s apply changed the world. The remedy is to apply a saved plan file produced under the same revision and to re-plan if state moved. Terraform detects a changed state serial when applying a saved plan and refuses, which is the behaviour you want; do not defeat it with -lock=false.

The -lock-timeout flag is the correct tool for contention. The CLI reference says it makes Terraform retry acquiring a lock for a period before erroring. A pipeline with -lock-timeout=5m queues politely behind a colleague’s apply instead of failing and paging someone. Pair it with a pipeline-level concurrency group so that two runs for the same state never start simultaneously in the first place.

Encryption: at rest, in transit and client side

Server-side encryption (SSE-KMS in S3, CMEK in GCS) protects against stolen disks and misconfigured snapshots, but anyone with read access to the object and decrypt rights on the key reads plaintext. It does nothing against a leaked pipeline credential. For that threat you want narrow IAM and, optionally, client-side encryption.

OpenTofu supports the latter natively. Its state and plan encryption is configured inside terraform { encryption { ... } } using key_provider, method, and state or plan blocks. The documentation lists PBKDF2 (passphrase-derived), AWS KMS, GCP KMS, Azure Vault, OpenBao and external key providers, and an aes_gcm method. An enforced = true setting prevents unencrypted data from being written, and a fallback block lets OpenTofu read data written under an old method, which is how you migrate in and rotate keys.

terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "alias/tofu-state"
      region     = "eu-west-1"
      key_spec   = "AES_256"
    }
    method "aes_gcm" "main" {
      keys = key_provider.aws_kms.main
    }
    state {
      method   = method.aes_gcm.main
      enforced = true
    }
    plan {
      method   = method.aes_gcm.main
      enforced = true
    }
  }
}

The cautions in the documentation deserve repeating. Losing the key makes the state and plan files unrecoverable. Encryption does not defend against corruption, replay of older state or plan files, or the operator running the tool. Do not rename key providers or methods after encrypting without using a fallback or a fixed encrypted_metadata_alias. The docs also recommend a separate KMS key per state file and note that AES-GCM can suffer from key saturation, so use a long passphrase with PBKDF2 or a KMS with automatic rotation. Check the exact attribute names against your OpenTofu version before copying the snippet above; it is representative, not authoritative.

Terraform itself has no equivalent client-side layer in the open-source CLI as far as I can verify, so Terraform users rely on backend encryption, narrow access and keeping secrets out of state through ephemeral values and write-only arguments.

State Splitting, Cross-State Data and Dependencies

Choosing a split

There are three common axes: by environment (dev, staging, prod), by layer (network, data, compute, apps) and by service or team. Most mature estates use all three, producing a path like prod/eu-west-1/network/terraform.tfstate. Environment is the first cut because it enforces credential separation and lets non-production experiments never touch production. Layer is the second because layers change at different speeds. Service is the third because ownership and on-call follow service boundaries.

Over-splitting has a cost. Each state needs its own backend config, pipeline, permissions and mental model; each cross-state reference is a coupling you must version. Under-splitting costs in plan time and blast radius. A pragmatic heuristic from practice: split when a plan exceeds a few minutes, when two teams contend for the same lock weekly, or when one credential would otherwise need rights over unrelated systems. Treat those as illustrative thresholds, not rules.

terraform_remote_state versus data sources

When a downstream state needs a value from an upstream one, you have two mechanisms. The terraform_remote_state data source reads another state’s outputs directly. It is simple but couples the consumer to the producer’s backend, requires read access to the entire upstream state (including any secrets in it, not merely the outputs), and creates an implicit dependency on output names.

The alternative is to look the object up through the provider’s API with a normal data source: find the VPC by tag, the subnet by filter, the KMS key by alias. This decouples consumers from the producer’s backend and exposes only what the cloud API exposes under the consumer’s own permissions. The downside is that lookups depend on naming and tagging conventions, so those conventions become a contract.

# Consumer state: look up the VPC by tag, not by reading network state
data "aws_vpc" "core" {
  tags = { Name = "core-prod" }
}

data "aws_subnets" "private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.core.id]
  }
  tags = { Tier = "private" }
}

A third option is a publish-subscribe store: the producer writes selected values to SSM Parameter Store or a similar registry and consumers read them. That limits exposure to deliberately published values, supports different access policies per parameter and survives a backend migration. It adds a moving part, but for cross-team contracts it is usually the cleanest.

Figure 2: State splitting. Each box is its own state, backend key and pipeline. Dashed coupling goes through tag lookups or a parameter registry rather than direct state reads.

Figure 2 shows the shape I recommend for a mid-sized estate: environment-scoped accounts, a thin shared network layer, a data layer and per-service states that consume lookups. The arrows flow one way, from slow-changing layers to fast-changing ones, which keeps the dependency graph acyclic and the apply order obvious.

Workspaces are not environments

Terraform CLI workspaces let one configuration hold several states in the same backend. They are convenient for ephemeral copies of a stack and poor as an environment boundary: the workspaces share a backend, share credentials unless you engineer otherwise, and share code, so a variable typo can hit production from a dev session. For hard boundaries, use separate backends and accounts and separate pipelines. Keep workspaces for short-lived review environments where a leaked credential is low-impact.

Terragrunt and wrapper tools

Splitting multiplies boilerplate: every state wants a backend block, a provider block and common variables. Tools such as Terragrunt generate those blocks from a hierarchy and manage dependency ordering across states, and Terraform’s own stacks features and HCP Terraform workspaces address the same pain differently. I will not make tool-specific claims about versions here; the design principle is that backend configuration should be generated or templated, never hand-copied, because a copy-paste error that points two services at the same key is a classic way to corrupt state.

Drift Detection and State Surgery

Where drift comes from

Drift is any divergence between what configuration and state say and what exists. The causes are mundane. An on-call engineer widens a security group in the console during an incident and forgets to back-port it. An autoscaler or a managed service mutates an attribute that Terraform also manages. A cloud provider adds a default field in a new API version. Another tool, such as a Kubernetes controller or a Crossplane composition, owns part of the same object. A teammate deletes a resource by hand. Each has a different right answer: back-port the change, add ignore_changes, accept the provider default, split ownership, or recreate.

It helps to separate three kinds. Config drift is the console edit that Terraform will revert on the next apply. State drift is where state is stale relative to reality but the next refresh will fix it harmlessly. Ownership drift is where two controllers fight over one attribute, producing perpetual diffs. Only the first is a security or compliance concern by default; the third is a design bug.

Detecting it with refresh-only plans

The primary built-in tool is the refresh-only plan. The plan command reference describes -refresh-only as creating a plan only to update state and root module outputs to match changes made to remote objects outside Terraform. Combined with -detailed-exitcode, which returns 0 for no changes, 1 for error and 2 when changes are present, it gives you a scriptable drift check.

#!/usr/bin/env bash
# drift-check.sh: run per state directory by a scheduled job
set -uo pipefail
terraform init -input=false -lock=false >/dev/null
terraform plan -refresh-only -detailed-exitcode -input=false \
  -lock-timeout=2m -out=drift.tfplan
rc=$?
case $rc in
  0) echo "no drift" ;;
  2) echo "DRIFT DETECTED"; terraform show -no-color drift.tfplan > drift.txt
     # post drift.txt to the ticketing system or chat webhook here
     exit 2 ;;
  *) echo "scan error"; exit 1 ;;
esac

Two details matter. A refresh-only plan reports drift in resources already in state; it will not find resources created outside Terraform that you have never imported, which is the “shadow infrastructure” problem. And a plain terraform plan also surfaces config drift, because it proposes to revert the change, so a scheduled normal plan with -detailed-exitcode detects both “code changed but not applied” and “world changed”. Many teams run the normal plan nightly and treat exit code 2 as a ticket, then distinguish causes in triage.

For unmanaged resources, the earlier open-source answer was driftctl, which compared cloud inventory with state. My understanding is that it is no longer actively developed, so verify its status before depending on it. Practical substitutes are cloud-native inventory (AWS Config, Azure Resource Graph, GCP Cloud Asset Inventory) joined with state resource IDs, commercial IaC platforms that include discovery, and tag policies that make untagged resources an alert. The inventory join is simple in concept: list resource ARNs in a scoped account, subtract those present in any state, and review the remainder.

Reconciling: accept, revert or adopt

Once drift is found there are three responses and the choice is a human one. To revert, run a normal apply and let Terraform restore declared configuration. To accept, edit the configuration to match reality and confirm the next plan is empty. To adopt a resource that was never managed, import it. Applying a refresh-only plan updates state to match the world without changing infrastructure, which is useful when the drift is benign (a provider-computed tag) and you want a clean baseline, but it also records the drift as the new truth, so do it deliberately and review the diff first.

Figure 3: Drift loop. A scheduler runs plans, exit code 2 opens a ticket, triage chooses revert, accept or adopt, and the normal pull-request pipeline carries out the fix.

Notice in Figure 3 that the reconciliation path rejoins the normal pipeline. Drift fixes should not be a side channel with looser controls; the same review and policy gates apply, otherwise the drift process becomes the way people sneak changes around review.

Import blocks, moved blocks and removed blocks

Refactoring state used to mean running terraform state mv and terraform import by hand against live state. Declarative blocks replaced most of that with changes that are reviewed in a pull request and executed in a normal plan.

The import block pairs with a resource block: to names the resource address, id is the provider-specific identifier (a string known at plan time), identity is an alternative map form and mutually exclusive with id, and for_each lets one block import many similar resources. The plan shows the import as a planned action so reviewers see it before anything changes. The documentation I checked does not state the introduction version or cover config generation on that page, so consult the dedicated guide for -generate-config-out and your version’s support.

import {
  for_each = {
    logs    = "acme-logs-prod"
    exports = "acme-exports-prod"
  }
  to = aws_s3_bucket.managed[each.key]
  id = each.value
}

resource "aws_s3_bucket" "managed" {
  for_each = toset(["logs", "exports"])
  bucket   = each.key == "logs" ? "acme-logs-prod" : "acme-exports-prod"
}

A moved block records that a resource changed address without being replaced: renaming a resource, moving it into a module, or switching between count and for_each. Terraform rewrites the state address during the next apply and the remote object is untouched. Keep moved blocks in the codebase for as long as any consumer of your module might still hold the old address, which for public modules means a long time.

moved {
  from = aws_instance.web
  to   = module.compute.aws_instance.web
}

For deletion from state without destroying the real resource, Terraform offers a removed block with a lifecycle setting that forgets the resource instead of destroying it. I am deliberately not quoting exact syntax or the introduction version because I could not verify them against the documentation in this run; check the language reference for your version. The older imperative commands (terraform state rm, terraform state mv) remain available for emergencies, but they act on live state immediately and bypass review, so use them only with a backup and a lock held.

State surgery safely

When you must touch state by hand, follow a fixed procedure. First, make sure no pipeline can run: disable the schedule and hold the lock or take the workspace offline. Second, pull a copy with terraform state pull > backup.tfstate and record the serial. Third, perform the smallest possible operation (state mv, state rm) and verify with terraform plan that the result is empty or exactly what you intended. Fourth, if something went wrong, restore from the object version, not from your local copy alone. Never hand-edit the JSON unless you also bump the serial correctly, because Terraform uses it to reject stale writes; an edit that leaves the serial unchanged is likely to be mistaken for the old state or rejected.

Plan/Apply Pipelines, Policy and Pinning

The pipeline shape

A safe pipeline has five stages: validate, plan, review and policy, apply, and verify. Validation runs fmt -check, validate and linters. Plan runs on every pull request with read-only credentials and posts a readable summary. Policy evaluates the plan JSON. Review requires an approver who is not the author for sensitive paths. Apply runs only from the merged commit, using the saved plan or a fresh plan that is compared against the approved one. Verify runs a post-apply refresh-only plan or smoke checks and fails the run if drift appears immediately.

Figure 4: Plan and apply pipeline as a sequence. The pull request triggers plan with read-only credentials, policy and review gate the result, and apply runs from the merged commit with write credentials and the state lock.

The security-critical design choice is credential separation. The plan job needs read access to state and cloud APIs; the apply job needs write access. If a pull request from a branch could run apply credentials, any contributor could execute arbitrary provider code with production rights. Run plans from forks or untrusted branches without secrets, and make apply a separate job triggered only from the protected branch via OIDC federation with short-lived tokens rather than stored keys.

# .github/workflows/terraform.yml (condensed, illustrative)
name: terraform
on:
  pull_request:
    paths: ["stacks/network/**"]
  push:
    branches: [main]
    paths: ["stacks/network/**"]
concurrency:
  group: tf-network-prod
  cancel-in-progress: false
permissions:
  id-token: write
  contents: read
  pull-requests: write
jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-plan-readonly
          aws-region: eu-west-1
      - run: terraform -chdir=stacks/network init -input=false
      - run: terraform -chdir=stacks/network plan -input=false -lock-timeout=5m -out=tfplan
      - run: terraform -chdir=stacks/network show -json tfplan > plan.json
      - run: conftest test plan.json --policy policy/
  apply:
    if: github.event_name == 'push'
    environment: prod-network     # required reviewers configured here
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-apply-network
          aws-region: eu-west-1
      - run: terraform -chdir=stacks/network init -input=false
      - run: terraform -chdir=stacks/network apply -input=false -lock-timeout=10m -auto-approve

The condensed example applies a fresh plan after merge, which is simpler but means the applied plan can differ from the one reviewed if the world changed in between. The stricter variant stores the saved plan as an artifact keyed to the commit SHA and applies exactly that file. Note the CLI warning that saved plan files may contain sensitive data in cleartext, so the artifact needs the same access control as state, short retention and encryption. OpenTofu’s plan encryption addresses the artifact risk directly.

The concurrency group serialises runs per stack, with cancel-in-progress: false so an in-flight apply is never cancelled mid-write. Cancelling an apply is how partial applies and stuck locks are born.

Atlantis, Terragrunt, Spacelift and HCP Terraform

Several products implement this pattern so you do not have to assemble it yourself. Atlantis is a self-hosted pull-request automation server: comments such as atlantis plan and atlantis apply run the engine and post output. Its locking documentation describes a per-directory-and-workspace lock taken at plan time and held until the pull request is merged or closed, or the plan is deleted, so a second pull request planning the same directory gets an error pointing to the holder. Apply happens before merge, which is why that lock exists: main would otherwise not reflect infrastructure. This gives strong safety and a real cost, since long-lived pull requests block everyone touching the same directory.

Spacelift, Scalr, env0 and HCP Terraform are managed alternatives with their own policy engines, drift detection schedules, approval flows and private runners. HCP Terraform pairs workspaces with a managed run queue, state versions and Sentinel or OPA policy. Terragrunt sits underneath any of these as an orchestrator for multi-state dependency graphs. I will not rank vendors; pick on where runs execute (your network or theirs), how policy is authored, how credentials are issued and what the exit story is, because a pipeline is also a lock-in surface.

Policy as code

Policy checks catch the class of mistakes review misses: a public S3 bucket, an unencrypted volume, an IAM policy with *:*, a resource in a forbidden region. Three families are common. OPA with Conftest evaluates Rego against the JSON from terraform show -json. Sentinel is HashiCorp’s policy language integrated with HCP Terraform. Checkov and similar scanners analyse configuration and plans against rule libraries. Use scanners for broad coverage with low authoring cost and OPA or Sentinel for organisation-specific rules such as “production changes need a ticket label” or “no more than N resources destroyed in one plan”.

A destroy-count guard is the single most valuable custom policy I know. A Rego rule that fails any plan deleting more than a set number of resources, or any deletion of a resource type in a protected list (databases, KMS keys, state buckets), converts “oops, wrong directory” into a failed check. Pair it with lifecycle { prevent_destroy = true } on stateful resources, remembering that prevent_destroy only blocks plans that would destroy the resource while it remains in configuration; removing the block removes the protection.

package terraform.guard

import rego.v1

deny contains msg if {
  some rc in input.resource_changes
  rc.type in {"aws_db_instance", "aws_kms_key", "aws_s3_bucket"}
  "delete" in rc.change.actions
  msg := sprintf("blocked: deleting protected resource %s", [rc.address])
}

Provider pinning, lock files and module versions

Reproducibility has three layers. The required_version constraint pins the engine version. The required_providers block constrains provider versions, and the .terraform.lock.hcl file records exact provider versions and checksums selected by init. Commit the lock file, and generate checksums for every platform your CI and developers use with terraform providers lock -platform=linux_amd64 -platform=darwin_arm64. Without it, a provider release between plan and apply can change behaviour under you.

Modules need version constraints too. A registry module with version = "~> 5.0" floats within a major version; pinning an exact version is more conservative and makes upgrades explicit pull requests. For internal modules referenced by git ref, use tags, not branches. Treat provider and module upgrades as changes with their own plan review, ideally one at a time, because a provider upgrade can introduce diffs on untouched resources (new defaults, renamed attributes) that look like drift.

Engine upgrades deserve staging as well. State written by a newer engine version may not be readable by an older one, so a single developer running a newer CLI locally can upgrade shared state and lock out the pipeline. Pin the CLI version in CI, enforce it with required_version, and roll engine upgrades through non-production states first. The Terraform documentation in this period labels 1.16.x as the latest line, but check release notes for your own upgrade path rather than relying on this article.

Trade-offs, Gotchas, and What Goes Wrong

Stuck locks and partial applies

A crashed runner can leave a lock behind. The recovery tool is terraform force-unlock <LOCK_ID>, and the discipline is to confirm the holder is truly dead first. Check the CI system for a running job, read the lock info (who, when, operation) and only then remove it. Force-unlocking a live apply invites two writers. With native S3 locking, the lock is an object, so an operator could delete it by hand, but go through the CLI so the ID check applies.

A partial apply is different: the provider created some resources, failed on the next, and Terraform wrote what succeeded to state. Re-running apply usually converges. The dangerous partial apply is when the process was killed before state was written; then real resources exist that state does not know about. The fix is to import them or delete them manually, which is why apply jobs must never be cancelled casually and why the backend should write state after each resource operation (Terraform does) rather than only at the end.

Concurrent apply and the saved-plan trap

Two engineers running apply from laptops against the same backend is prevented by locking, but two pipelines for different branches that were both planned against the same base are not: the second apply will either fail on a stale saved plan or, if you re-plan at apply time, silently apply a different change set than the one reviewed. Pipelines should apply from the merged commit on a single serial queue so that the plan reviewed is the plan on main.

State file bloat and slow refresh

Every resource is refreshed during plan by default. In a state with several thousand resources, plan time is dominated by API reads, rate limits and provider throttling. Splitting is the structural fix; -refresh=false and -target are escape hatches that skip safety. -target in particular produces plans that ignore dependencies and should be reserved for recovery, with a follow-up full plan. Also watch for large for_each maps over thousands of items, which turn one block into thousands of state entries and slow every operation.

Secrets, outputs and “sensitive” misconceptions

The sensitive flag is cosmetic as far as state goes. Anything marked sensitive is still stored in plaintext in state and plan files, and terraform output -raw prints it. Combining sensitive with ephemeral values, using write-only arguments where the provider supports them, and fetching secrets at runtime from a secrets manager rather than creating them in Terraform are the durable fixes. Also remember that terraform_remote_state grants read access to everything in the upstream state, not just its outputs.

Anti-patterns worth naming

One giant state “to keep things simple”.
Using workspaces as the production boundary.
Disabling locking (-lock=false) to get around a stuck pipeline.
Committing local state or .tfvars with secrets to git.
Running apply from laptops with admin keys.
Auto-approving applies in production without a policy gate or destroy guard.
Importing resources without writing the configuration and then ignoring all changes.
Letting drift tickets age; a stale drift queue trains everyone to ignore it.

Terraform, GitOps and Crossplane: Where State Lives Elsewhere

Terraform’s model is plan/apply with an explicit state file and a push-style pipeline. GitOps controllers such as Argo CD and Flux invert it: a controller inside the cluster continually reconciles live objects with a git repository, and the Kubernetes API (etcd) is the state. Drift is corrected automatically unless you disable self-heal. Our Argo CD versus Flux decision record compares the two controllers.

Crossplane extends the controller model to cloud infrastructure: managed resources and compositions represent cloud objects as Kubernetes custom resources, and controllers reconcile them continuously. The tradeoff is real. You gain continuous drift correction, a single API for platform teams and no state file to protect, because desired and observed state live in the cluster. You lose the explicit, reviewable plan: a change takes effect when merged and reconciled, and the dependency on cluster availability and etcd integrity becomes the new single point of failure. Our overview of Crossplane compositions for an internal developer platform goes into that design.

A pragmatic hybrid is common: Terraform for foundational, slow-changing infrastructure (accounts, networks, clusters, IAM baselines) where a reviewed plan is valuable, and controller-based reconciliation for fast-moving platform resources that application teams request through claims. The boundary should be explicit, because two controllers managing one object is exactly the ownership drift described earlier.

Practical Recommendations

If I were setting this up from scratch, I would start with the backend and credentials, because they are the hardest to change later. Put state for each environment in its own account and bucket with versioning, encryption, block-public-access and key-prefix scoped IAM. Turn on native locking where your engine supports it and plan the migration off DynamoDB if you are on Terraform. Then split by environment, layer and service until plans take minutes, not tens of minutes, and replace direct remote-state reads with lookups or a parameter registry for cross-team contracts.

Next, make the pipeline the only writer. Plans run on pull requests with read-only identity; applies run from the protected branch with short-lived federated credentials and a per-stack concurrency group. Add policy gates, with a destroy guard first. Schedule nightly plans per stack and wire exit code 2 to a ticket queue with an owner and a service-level objective for resolution. Finally, rehearse recovery: restore a state from a bucket version, force-unlock a test lock and import a deleted resource, so the first time you do it is not during an incident.

[ ] Remote backend with versioning, encryption and native locking; no local state anywhere
[ ] State split by environment, layer and service; credentials scoped per key prefix
[ ] Secrets kept out of state via ephemeral values, write-only arguments or external secret stores
[ ] Plan on PR with read-only role; apply from main with federated role and concurrency group
[ ] Policy as code with a destroy guard and prevent_destroy on stateful resources
[ ] .terraform.lock.hcl committed with multi-platform checksums; CLI and providers pinned
[ ] Nightly drift plans with ticket automation and an owner per state
[ ] Documented, rehearsed runbooks for force-unlock, restore and import
[ ] Reviewed moved and import blocks instead of ad hoc state commands

Frequently Asked Questions

What is Terraform state and why does it need a remote backend?

State is a JSON record mapping resource addresses in your configuration to real remote objects, along with cached attributes. Terraform needs it to compute diffs, order operations and know what to destroy. A remote backend gives teams a shared source of truth, enables locking so concurrent runs do not corrupt it, supports versioning for recovery and lets you apply access control and encryption. Local state in a team setting leads to divergent copies and accidental duplicate resources.

Do I still need DynamoDB for Terraform state locking on S3?

The current Terraform S3 backend documentation marks DynamoDB-based locking as deprecated and says it will be removed in a future minor version. The replacement is use_lockfile = true, which creates a lock object next to the state using S3 conditional writes. It defaults to false, so you must enable it. OpenTofu documents the same option but states it has no plans to deprecate DynamoDB. Verify behaviour against the versions you run before migrating.

How do I detect drift in Terraform?

Run terraform plan -refresh-only -detailed-exitcode on a schedule for each state. Exit code 2 means the plan found differences, 0 means none and 1 means an error. A normal scheduled plan also reveals drift because it proposes to revert the change. Neither finds resources created outside Terraform that were never imported; for those, compare cloud inventory (AWS Config, Resource Graph, Cloud Asset Inventory) against the resource IDs in your states.

Is OpenTofu state encryption worth using?

It protects against exposure of state or plan files at rest or in artifacts, independent of the backend’s own encryption, so a leaked bucket read does not reveal secrets. The documentation lists PBKDF2, AWS KMS, GCP KMS, Azure Vault, OpenBao and external key providers with an AES-GCM method. The trade-off is key management: losing the key makes state unrecoverable, and it does not protect against replay or a malicious operator. Plan key backup and rotation first.

Should I use workspaces or separate states for environments?

Use separate states, backends and accounts for real environments. Workspaces share a backend, code and usually credentials, so a mistake in a dev session can reach production, and the boundary is only a naming convention. Workspaces are fine for short-lived copies of a stack, such as per-pull-request review environments. The stronger separation costs more boilerplate, which generation tools like Terragrunt or templated pipelines can absorb.

What should I do when a Terraform lock is stuck?

First confirm the holder is dead: check the CI job, read the lock metadata for user and time, and wait through any -lock-timeout. Then run terraform force-unlock <LOCK_ID> with the ID from the error message. Never use -lock=false as a workaround, since it permits concurrent writers. Afterwards inspect state for a partial apply by running a plan, and import or clean up any resources that were created but never recorded.