Drift Detection & Day-2 Operations

By the end of this lesson you will be able to detect drift in both Terraform and Ansible, decide what to do when you find it, schedule a daily drift check in CI, and handle common day-2 jobs such as rolling OS patches and version upgrades. Building infrastructure is day 1; keeping it healthy for years is day 2.

What drift looks like in real life

Drift is any difference between your code and the real systems. It creeps in quietly:

• Someone opens a port in the cloud console during a late-night incident and forgets to close it.
• A teammate edits /etc/nginx/nginx.conf on one server to test something.
• An automatic process changes a setting, such as a scaling group's size.

Think of a society's parking plan in Jaipur. The plan says flat 302 has slot 14. Over months, people park wherever there is space. The plan on paper still looks neat, but it no longer tells the truth. Drift detection is the monthly walk around the parking area with the plan in hand.

Step 1: Create drift on purpose

Go back to the hello-tf folder and make sure welcome.txt exists (terraform apply if needed). Now simulate a manual change by deleting the file outside Terraform:

rm welcome.txt
terraform plan

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # local_file.welcome has been deleted
  - resource "local_file" "welcome" {
      - filename = "./welcome.txt" -> null
    }

  # local_file.welcome will be created
  + resource "local_file" "welcome" {

Plan: 1 to add, 0 to change, 0 to destroy.

The output is trimmed. The first section lists what changed outside Terraform. The second section is what Terraform will do to bring reality back in line with the code.

Step 2: Turn drift into an exit code

A human can read the plan, but a scheduled job needs a simple signal. -detailed-exitcode gives one:

terraform plan -detailed-exitcode > /dev/null
echo "exit code: $?"

exit code: 2

Run terraform apply to restore the file, then the same command prints exit code: 0.

Step 3: Decide what to do with drift

Finding drift is half the job. For each difference, choose one of three answers:

1. Revert it. The manual change was a mistake. Run terraform apply, and the code wins.
2. Adopt it. The manual change was right. Update the code to match, open a pull request, and the next plan shows no changes.
3. Accept it into state only. Sometimes you only want Terraform's record to catch up with reality, without changing anything. Use a refresh-only plan and apply:

terraform plan -refresh-only
terraform apply -refresh-only

A refresh-only apply updates the state to match the real objects and makes no changes to them. It does not change your code, so you usually still need option 2 afterwards. (Older guides use terraform refresh, which does the same without showing you a plan first; it is deprecated.)

Step 4: Schedule a daily drift check

A drift check that nobody runs is useless. Add a scheduled GitHub Actions workflow, .github/workflows/drift.yml, that plans every morning and opens an issue when it finds changes:

name: drift-check

on:
  schedule:
    - cron: "30 2 * * *"   # 02:30 UTC = 08:00 IST
  workflow_dispatch:

permissions:
  contents: read
  id-token: write
  issues: write

jobs:
  drift:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/envs/prod
    steps:
      - uses: actions/checkout@v7
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: 1.17.1
          terraform_wrapper: false
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-plan
          aws-region: ap-south-1
      - run: terraform init -input=false
      - id: plan
        run: |
          set +e
          terraform plan -input=false -no-color -detailed-exitcode > plan.txt
          code=$?
          echo "exitcode=$code" >> "$GITHUB_OUTPUT"
          if [ "$code" -eq 1 ]; then exit 1; fi
      - if: steps.plan.outputs.exitcode == '2'
        uses: actions/github-script@v9
        with:
          script: |
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: 'Drift detected in prod',
              body: 'The daily plan found changes. See the workflow run logs.'
            });

GitHub's cron uses UTC, so 02:30 UTC is 8:00 in the morning in India. set +e stops the shell from failing on exit code 2, so the script can save the code first. The job uses the read-only plan role, so it can never change anything. HCP Terraform offers a built-in version of this called health assessments.

Drift in Ansible

Ansible has no state file, so drift shows up as tasks that would change. Run the playbook in check mode regularly:

ansible-playbook site.yml --check --diff

TASK [Write the Nginx site config] *********************************************
changed: [web2]

PLAY RECAP *********************************************************************
web1                       : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2                       : ok=6    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The output is trimmed; with --diff, the lines that differ appear above changed: [web2]. Someone edited the site config on web2 by hand. Any changed count above zero in check mode is drift. Running the playbook normally reverts it; if the edit was right, move it into the template or variables first.

Remember that Ansible only checks what it manages. A file it has never heard of can change freely, which is one reason to manage important config files completely, not line by line.

Day-2 operations

Beyond drift, a healthy IaC setup needs regular care. Common jobs:

• Rolling OS patches. Update servers a few at a time so the site stays up. Ansible's serial keyword runs the whole play on one batch of hosts before starting the next.
• Provider and module upgrades. One at a time, with init -upgrade, a no-change plan and a separate pull request.
• Terraform and ansible-core upgrades. Read the release notes and porting guides, upgrade dev first, and update the pinned versions in CI.
• Secret rotation. Change passwords and keys on a schedule, using vault files and write-only arguments.
• Clean-up. Remove unused resources, old workspaces and finished moved blocks.

Here is a rolling patch play for Ubuntu servers:

- name: Patch web servers one at a time
  hosts: web
  become: true
  serial: 1

  tasks:
    - name: Upgrade all packages
      ansible.builtin.apt:
        upgrade: dist
        update_cache: true

    - name: Check whether a reboot is needed
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_flag

    - name: Reboot if required
      ansible.builtin.reboot:
      when: reboot_flag.stat.exists

With serial: 1, web1 is patched (and rebooted if needed) before web2 starts, so one server always keeps serving. The reboot module waits for the server to come back before moving on. Behind a load balancer, you would also take each server out of rotation first.

Common mistakes

• Treating exit code 2 as a failure. It means "changes found", which is exactly what a drift job wants to report, not an error.
• Always reverting drift. Sometimes the manual fix was correct; adopt it into code instead of undoing it.
• Using terraform apply -refresh-only and stopping there. State now matches reality, but code still does not.
• Patching every server at once. Without serial, a bad update takes the whole site down together.

Key points

• Drift is any difference between code and reality; it comes from manual and automatic changes.
• terraform plan -detailed-exitcode returns 0 for no changes, 1 for errors and 2 for changes.
• For each difference, revert it with apply, adopt it into code, or record it with a refresh-only apply.
• Schedule a read-only drift check in CI and raise an issue when it finds changes.
• In Ansible, --check --diff with a non-zero changed count reveals drift on managed settings.
• serial gives rolling updates; upgrades and secret rotation are regular day-2 work.

#!/usr/bin/env bash
terraform plan -input=false -detailed-exitcode > /dev/null 2>&1
code=$?
case "$code" in
  0) echo "No drift" ;;
  2) echo "Drift found" ;;
  *) echo "Plan failed"; exit 1 ;;
esac

Drift found