Infrastructure as Code: Terraform Practices That Scale

Why click-ops does not scale

Click-ops — provisioning cloud resources by hand in a web console — feels productive on day one and becomes a liability by month six. A human clicking through a portal produces infrastructure that no one can reproduce, review, or reason about. There is no diff, no history, no test. When the person who built it leaves, so does the only record of how it works.

The failure modes are predictable:

• No reproducibility. Rebuilding a region for disaster recovery means remembering hundreds of settings correctly, under pressure, in the wrong order.
• Configuration drift. Someone loosens a security group "just for a minute" and it stays open for a year. Nobody notices because nobody is comparing intended state to actual state.
• No review. A change that would never pass code review — a public database, an over-permissive role — sails through because a console click has no pull request.
• Snowflake environments. Staging and production diverge until "works in staging" stops meaning anything.

Infrastructure as code (IaC) replaces the console with declarative files under version control. Terraform is the de facto standard because it is cloud-agnostic, has a mature provider ecosystem, and expresses infrastructure as a dependency graph it can plan before it touches anything. Every change becomes a reviewable, revertible, testable artifact. The rest of this post covers the practices that keep Terraform manageable as your estate grows from ten resources to ten thousand.

Figure: IaC turns each change into a reviewed, policy-gated pipeline step — the automation, not a person, is what applies to production.

State is the source of truth

Terraform records what it manages in a state file that maps your code to real cloud resources. Get state wrong and nothing else matters, because state is how Terraform knows what already exists.

Never keep state on a laptop, and never commit it to Git. Local state cannot be shared, is trivially lost, and contains secrets in plain text. Use a remote backend instead:

• AWS: an S3 bucket with versioning enabled, plus a DynamoDB table for state locking.
• Azure: a Storage Account blob container, which provides native locking.
• Google Cloud: a GCS bucket with object versioning.
• Terraform Cloud / Enterprise: managed state with locking, history, and access controls built in.

Two rules are non-negotiable:

1. Locking. When two engineers run apply at once without a lock, they corrupt state and can destroy live resources. A locking backend serializes writes so only one apply runs at a time.
2. Encryption and access control. State contains resource IDs, connection strings, and sometimes generated passwords. Encrypt the backend at rest, restrict who can read it, and treat it as sensitive as any production database.

Split state deliberately. One giant state file for the whole company means every plan is slow and every apply risks everything. Break state along blast-radius boundaries — per environment, per major service — so a change to one system cannot corrupt another.

Structure code into modules

Copy-pasted Terraform is the infrastructure equivalent of copy-pasted code: it rots. Modules are how you get reuse and consistency.

• Root modules are what you actually apply — one per environment or per deployable unit. They are thin, wiring together child modules with environment-specific inputs.
• Child modules encapsulate a reusable pattern: a VPC with sane defaults, an encrypted database, a standard Kubernetes node pool. Write the security defaults once and every consumer inherits them.

Practical guidance:

• Keep modules small and focused. A module that does one thing well composes; a 2,000-line "does everything" module does not.
• Pin module and provider versions explicitly. An unpinned provider that jumps a major version overnight breaks every plan.
• Version shared modules in a registry (a private registry, or just tagged Git refs) so a fix ships on a schedule you control rather than silently.
• Expose a minimal input surface with validated variables and sensible defaults. Every knob you expose is a knob someone will set wrong.

This is also where security standards get baked in. A module that encrypts storage by default, refuses public exposure, and enforces tagging turns your cloud security posture into the path of least resistance rather than an afterthought.

Environments and workspaces

Development, staging, and production must be isolated so a mistake in dev cannot reach prod. There are two common patterns, and the distinction matters.

• Directory per environment (preferred for most teams). Each environment gets its own folder, its own backend and state, and its own variable file. Isolation is explicit and visible. Blast radius is contained by design.
• Terraform workspaces. Multiple named states from one configuration. Convenient for ephemeral, identical copies — a per-developer sandbox or a short-lived preview — but risky as a production boundary, because a single misread variable can point a prod apply at the wrong workspace.

Our guidance for client estates: use directories for the dev, staging, and prod boundary, and reserve workspaces for ephemeral environments spun up and torn down by automation. Keep the code identical across environments and let variables express the differences — instance sizes, replica counts, domains — so environments stay honest copies of each other.

Plan and apply in CI with policy as code

The moment more than one person writes Terraform, apply has to leave laptops and move into a pipeline. A workflow that scales:

1. Pull request opens. CI runs terraform fmt, validate, and plan, posting the plan as a comment so reviewers see exactly what will change.
2. Policy as code gates the plan. Open Policy Agent (OPA and Conftest) or HashiCorp Sentinel evaluates the plan against rules — no public S3 buckets, encryption required, approved regions only, mandatory tags. A violation fails the check before a human even looks.
3. Human review. An engineer approves the change and its plan.
4. Apply on merge. Only the pipeline holds cloud credentials, and it applies the exact plan that was reviewed. Nobody runs apply by hand against production.

Policy as code is what makes governance scale without a bottleneck. Instead of a security engineer manually reviewing every change, the rules are executable and run on every pull request. Write them to encode the same guardrails you would enforce anywhere — least privilege, encryption everywhere, no public data stores — and they enforce themselves consistently at 3 a.m. as well as 3 p.m.

Credentials belong to the pipeline, not to people. Use short-lived, federated credentials — OIDC from your CI system to the cloud — so there are no long-lived cloud keys sitting in CI secrets waiting to leak.

Drift detection

Even with everything in code, reality drifts. Someone makes an emergency console change during an incident, a managed service mutates a setting, or an out-of-band script edits a resource. Drift is the gap between your code and the live world, and it silently invalidates every assumption you have.

• Run scheduled terraform plan in read-only mode — nightly or hourly — against every environment. A non-empty plan means something changed outside the pipeline. Alert on it.
• Investigate before you reconcile. Drift can be a symptom of an unauthorized change or an active incident, not just a sloppy edit. Understand why before you re-apply.
• Reconcile deliberately. Either bring the code up to match an intended change, or re-apply to erase an unauthorized one — but decide, do not let drift accumulate.

Continuous drift detection feeds naturally into your broader monitoring. When a plan shows an unexpected open port or a changed IAM policy, that signal should reach the same team watching for threats across the estate.

Handling secrets

Secrets are where Terraform teams most often cut a corner they regret. The rules:

• Never hardcode secrets in .tf files or in variable files committed to Git. They live forever in history even after you delete them.
• Never commit state to Git — it can contain generated secrets in plain text (covered above, worth repeating).
• Source secrets at runtime from a dedicated secrets manager — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault — via data sources, so the value is fetched at apply time and never stored in code.
• Mark variables and outputs as sensitive so they are redacted from plan output and logs.
• Prefer generating and rotating over storing. Where possible, have the secrets manager generate and rotate credentials rather than Terraform managing static ones.

For estates handling regulated data or private workloads, keep this discipline tight — a leaked state file or a hardcoded key in a private cloud build undermines every other control you paid for.

Where to start

If you are still in the console, do not try to import everything at once. Pick one new service and build it in Terraform end to end — remote state, a module, a CI pipeline with a policy check — and use it as the template for everything that follows. If you already have Terraform but it lives on laptops with local state and hand-run applies, fix state and CI first; those two changes eliminate the largest risks.

Getting IaC right is less about Terraform syntax and more about the operating model around it — state, structure, pipelines, and policy. Our cloud infrastructure team builds and runs this model for organizations moving off click-ops, from a first governed module to a fully automated, policy-gated estate. Talk to us about an IaC assessment and we will map where your current setup is fragile and give you the sequenced plan to make it scale.