Migrating from Vault to GCP Secret Manager

We ran HashiCorp Vault in HA mode on GKE for about three years. It worked, but it required more ongoing care than I wanted to give it — unsealing after cluster events, managing the Raft storage backend, coordinating upgrades across the HA cluster, handling the occasional Vault pod OOMKill during high-load periods. When we started evaluating alternatives, GCP Secret Manager kept coming up as the obvious answer for teams already running on GCP.

Here's what the migration actually involved.

The Case for Migrating

Vault is a powerful piece of software. It does things Secret Manager doesn't: dynamic secrets (on-demand database credentials that expire automatically), PKI management, encryption as a service, and a rich plugin ecosystem. If you're using those features, migration is a real project.

We weren't using most of them. Our Vault deployment was primarily:
- KV secrets store (about 90% of usage)
- Kubernetes auth method for pod-level secret access

That's the easy migration profile. If you're using Vault's dynamic secrets or PKI CA features, spend real time evaluating what replaces them before committing to migration.

The operational burden of self-managed Vault is meaningful. In practice our Vault cluster required attention several times a year — not constantly, but reliably. An unsealing event after a planned GKE maintenance window, a storage migration during a Vault upgrade, rolling back a Vault version once after a bad upgrade broke our Kubernetes auth. Secret Manager is fully managed; there's nothing to operate.

Auditing What's in Vault

Before touching anything, I ran an audit of every Vault path in use. Vault's audit log (if you have it enabled — turn it on immediately if you don't) gives you access patterns. We also did a manual sweep:

# List all KV mounts
vault secrets list

# Recursively list all paths in a KV v2 mount
vault kv list -format=json secret/ | jq -r '.[]' | while read path; do
  vault kv list -format=json "secret/$path" 2>/dev/null | jq -r ".[] | \"secret/$path\(.)\""
done

We mapped each path to an owner team and a consuming application. This gave us a migration wave plan: migrate by team, with each team responsible for updating their applications to read from Secret Manager instead.

Terraform for Secret Manager

The Terraform resource model for Secret Manager is straightforward:

resource "google_secret_manager_secret" "db_password" {
  secret_id = "my-service-db-password"
  project   = var.gcp_project

  replication {
    auto {}
  }
}

resource "google_secret_manager_secret_version" "db_password_v1" {
  secret  = google_secret_manager_secret.db_password.id
  secret_data = var.db_password  # passed in via tfvars or from another source
}

resource "google_secret_manager_secret_iam_member" "my_service_reader" {
  secret_id = google_secret_manager_secret.db_password.id
  role      = "roles/secretmanager.secretAccessor"
  member    = "serviceAccount:${google_service_account.my_service.email}"
}

One thing I did differently than the examples in the docs: I separated the secret metadata (the google_secret_manager_secret resource) from the secret version (the google_secret_manager_secret_version resource). The metadata — name, labels, replication policy, IAM bindings — goes in your main Terraform repo. The actual secret value gets written separately, either manually or through a rotation process. This way secret values never appear in Terraform state.

Kubernetes Workload Access via Workload Identity

The Vault Kubernetes auth method issued short-lived tokens to pods; the equivalent in GCP is Workload Identity. Workload Identity binds a Kubernetes service account to a GCP service account, which then has IAM permissions on Secret Manager.

# Create GCP service account
gcloud iam service-accounts create my-service-gsa \
  --project="${PROJECT_ID}"

# Grant Secret Manager access
gcloud secrets add-iam-policy-binding my-service-db-password \
  --member="serviceAccount:my-service-gsa@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

# Bind to Kubernetes service account
gcloud iam service-accounts add-iam-policy-binding \
  my-service-gsa@${PROJECT_ID}.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:${PROJECT_ID}.svc.id.goog[my-namespace/my-service-ksa]"

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-ksa
  namespace: my-namespace
  annotations:
    iam.gke.io/gcp-service-account: my-service-gsa@PROJECT_ID.iam.gserviceaccount.com

With this in place, code running in the pod can call the Secret Manager API directly using Application Default Credentials — no tokens to manage, no Vault agent sidecar, no injected environment variables from a mutating webhook.

The External Secrets Operator

For applications that read secrets via env.valueFrom.secretKeyRef in their pod specs — which was most of our apps — we used the External Secrets Operator to sync Secret Manager secrets into Kubernetes Secret objects automatically.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-service-secrets
  namespace: my-namespace
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: my-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: my-service-db-password
        version: latest

This produces a standard Kubernetes Secret in my-namespace with key DB_PASSWORD. The application pod spec doesn't change at all — it still does env.valueFrom.secretKeyRef. The only change is which system is populating that Kubernetes Secret.

This made the migration much less disruptive. Teams didn't need to change their application code or deployment manifests to switch from Vault-backed secrets to Secret Manager-backed secrets.

Rotation

Secret Manager has a built-in rotation feature using Pub/Sub. You configure a rotation period and a notification topic on the secret:

resource "google_secret_manager_secret" "db_password" {
  secret_id = "my-service-db-password"

  rotation {
    rotation_period   = "2592000s"  # 30 days
    next_rotation_time = "2023-07-01T00:00:00Z"
  }

  topics {
    name = google_pubsub_topic.secret_rotation.id
  }

  replication {
    auto {}
  }
}

When rotation is due, Secret Manager publishes a message to the Pub/Sub topic. A Cloud Function subscribes to that topic, generates new credentials (e.g., rotates the database password), calls Secret Manager to add the new secret version, and disables the old one. The External Secrets Operator picks up the new version on its next refresh cycle.

We implemented this for our database passwords and service account keys. It replaced a manual process that had been running on a shared calendar reminder for two years.

The Pricing Gotcha

Secret Manager charges per operation, not per secret stored. As of 2023 it's $0.06 per 10,000 operations, with the first 10,000 operations per month free. This is usually cheap — but the per-operation model means you should audit access patterns before migration.

One service we identified was hitting Vault on every request to fetch a database connection string — no caching, 100+ requests per second. Multiplied out, that's millions of Secret Manager API calls per day. We had to add a local in-memory cache with a TTL before migrating that service.

The pattern for caching with a reasonable TTL in Python:

import time
from google.cloud import secretmanager

_cache = {}
_CACHE_TTL = 300  # 5 minutes

def get_secret(secret_name: str) -> str:
    now = time.time()
    if secret_name in _cache:
        value, ts = _cache[secret_name]
        if now - ts < _CACHE_TTL:
            return value

    client = secretmanager.SecretManagerServiceClient()
    response = client.access_secret_version(
        name=f"projects/{PROJECT}/secrets/{secret_name}/versions/latest"
    )
    value = response.payload.data.decode("utf-8")
    _cache[secret_name] = (value, now)
    return value

Overall the migration took about four months end to end — mostly waiting on individual teams to update their applications during their normal sprint cycles. The operational overhead dropped to zero on the Secret Manager side, which was exactly the point. Vault is still running for a handful of services that use its PKI features; that migration is a separate project.