Blog
March 11, 2020 Marie H.

Mutual TLS in gRPC Services with Go

Mutual TLS in gRPC Services with Go

Photo by <a href="https://unsplash.com/@hutters?utm_source=cloudista&utm_medium=referral" target="_blank" rel="noopener">Stan Hutter</a> on <a href="https://unsplash.com/?utm_source=cloudista&utm_medium=referral" target="_blank" rel="noopener">Unsplash</a>

Mutual TLS in gRPC Services with Go

I work on key encryption services at IBM. Security is not optional and mTLS is not a nice-to-have — it's the baseline requirement for any internal service-to-service communication that touches cryptographic material. Let me show you how we actually set it up in Go.

Why mTLS, Not Just TLS

Standard TLS gives you server authentication and encryption. The client verifies the server's certificate, negotiates a session key, and traffic is encrypted. That's sufficient for browser-to-server traffic.

For service-to-service communication, you also need the server to verify the client. Without client authentication, anyone who can reach your gRPC port can make calls to it. Network segmentation and service mesh policies help, but they're not a substitute for cryptographic identity.

With mTLS, both sides present certificates. The connection only succeeds if both certificates are trusted by the other side's CA. Your key management service knows that only your key consumer service can call it — not because of firewall rules, but because the calling service can prove its identity with a private key it controls.

This also gives you a clear audit trail. When a call comes in, you know which service made it. That matters when you're investigating an incident or auditing access to sensitive material.

Generating Certs for Development

In production we use cert-manager on Kubernetes. For local development I generate a small CA and issue certs from it:

# Create a root CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 1825 -key ca.key -out ca.crt \
  -subj "/CN=dev-ca/O=IBM Internal"

# Server cert
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr \
  -subj "/CN=keyservice.internal/O=IBM Internal"
openssl x509 -req -days 365 -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out server.crt

# Client cert (for the service making the call)
openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr \
  -subj "/CN=keyconsumer/O=IBM Internal"
openssl x509 -req -days 365 -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt

In production, cert-manager handles the issuance and renewal lifecycle. It integrates with your cluster's CA or an external CA (ACME, Vault, AWS PCA). The Go code that loads the certs doesn't change — just where the cert files come from.

Loading Certificates in Go

func loadTLSCredentials(certFile, keyFile, caFile string) (credentials.TransportCredentials, error) {
    // Load the certificate and private key
    certificate, err := tls.LoadX509KeyPair(certFile, keyFile)
    if err != nil {
        return nil, fmt.Errorf("loading key pair: %w", err)
    }

    // Load the CA cert to verify the peer
    caCert, err := os.ReadFile(caFile)
    if err != nil {
        return nil, fmt.Errorf("reading CA cert: %w", err)
    }

    certPool := x509.NewCertPool()
    if !certPool.AppendCertsFromPEM(caCert) {
        return nil, fmt.Errorf("failed to add CA cert to pool")
    }

    config := &tls.Config{
        Certificates: []tls.Certificate{certificate},
        RootCAs:      certPool, // for clients: verify server cert
        ClientCAs:    certPool, // for servers: verify client cert
        ClientAuth:   tls.RequireAndVerifyClientCert,
        MinVersion:   tls.VersionTLS12,
    }

    return credentials.NewTLS(config), nil
}

ClientAuth: tls.RequireAndVerifyClientCert is the line that enables mutual TLS on the server side. Without this you have regular TLS — server auth only. With it, the handshake fails if the client doesn't present a valid certificate signed by one of the CAs in ClientCAs.

gRPC Server Setup

func main() {
    creds, err := loadTLSCredentials("server.crt", "server.key", "ca.crt")
    if err != nil {
        log.Fatalf("failed to load TLS credentials: %v", err)
    }

    server := grpc.NewServer(
        grpc.Creds(creds),
        grpc.UnaryInterceptor(clientCertInterceptor),
    )

    pb.RegisterKeyServiceServer(server, &KeyServiceImpl{})

    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    log.Println("server listening on :50051")
    if err := server.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

gRPC Client Setup

func newClient(address string) (pb.KeyServiceClient, error) {
    creds, err := loadTLSCredentials("client.crt", "client.key", "ca.crt")
    if err != nil {
        return nil, fmt.Errorf("loading TLS credentials: %w", err)
    }

    conn, err := grpc.Dial(
        address,
        grpc.WithTransportCredentials(creds),
    )
    if err != nil {
        return nil, fmt.Errorf("dialing %s: %w", address, err)
    }

    return pb.NewKeyServiceClient(conn), nil
}

For the client, RootCAs in the tls.Config is what verifies the server's certificate. The Certificates field provides the client's own cert for mutual authentication.

Verifying the Client Cert in an Interceptor

After the TLS handshake, the client's verified certificate is available on the connection. I extract it in a unary interceptor to get the client's identity:

func clientCertInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    p, ok := peer.FromContext(ctx)
    if !ok {
        return nil, status.Error(codes.Unauthenticated, "no peer info")
    }

    tlsInfo, ok := p.AuthInfo.(credentials.TLSInfo)
    if !ok {
        return nil, status.Error(codes.Unauthenticated, "no TLS info")
    }

    if len(tlsInfo.State.VerifiedChains) == 0 || len(tlsInfo.State.VerifiedChains[0]) == 0 {
        return nil, status.Error(codes.Unauthenticated, "no verified client cert")
    }

    clientCert := tlsInfo.State.VerifiedChains[0][0]
    callerID := clientCert.Subject.CommonName

    // Tag the active tracing span with the caller identity so it shows up in Jaeger
    if span := opentracing.SpanFromContext(ctx); span != nil {
        span.SetTag("caller.id", callerID)
        span.SetTag("grpc.method", info.FullMethod)
    }

    // Structured log with zap — plain log.Printf doesn't give you searchable fields
    logger.Info("authenticated gRPC call",
        zap.String("caller", callerID),
        zap.String("method", info.FullMethod),
    )

    // Propagate caller ID downstream for authorization checks in handlers
    ctx = context.WithValue(ctx, callerKey{}, callerID)

    return handler(ctx, req)
}

The VerifiedChains field is populated by the TLS stack only after a successful mutual handshake. If this is empty, the TLS layer already rejected the connection — but this is a belt-and-suspenders check.

In production we chain this interceptor with the OpenTracing gRPC interceptor from github.com/grpc-ecosystem/grpc-opentracing so that trace context propagates correctly across service boundaries. The mTLS interceptor runs first (to authenticate), then the tracing interceptor creates the child span for the handler. Order matters: if tracing runs before auth and the auth check panics or errors, you'd still emit a span for an unauthenticated call, which pollutes your traces with noise.

Certificate Rotation Without Restart

Hot-reloading certificates is important for services with high uptime requirements. You don't want to restart a key service every 90 days when your cert expires.

The trick is tls.Config.GetCertificate (for servers) and tls.Config.GetClientCertificate (for clients). These are called per-connection rather than once at startup:

type certReloader struct {
    mu       sync.RWMutex
    certFile string
    keyFile  string
    cert     *tls.Certificate
}

func (r *certReloader) GetCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) {
    r.mu.RLock()
    defer r.mu.RUnlock()
    return r.cert, nil
}

func (r *certReloader) Reload() error {
    cert, err := tls.LoadX509KeyPair(r.certFile, r.keyFile)
    if err != nil {
        return fmt.Errorf("reloading cert: %w", err)
    }
    r.mu.Lock()
    r.cert = &cert
    r.mu.Unlock()
    return nil
}

Start a background goroutine that watches for cert file changes (using fsnotify or a simple poll) and calls Reload(). New connections will pick up the new certificate immediately. Existing connections aren't interrupted.

reloader := &certReloader{certFile: "server.crt", keyFile: "server.key"}
// load initial cert
if err := reloader.Reload(); err != nil {
    log.Fatalf("initial cert load: %v", err)
}

config := &tls.Config{
    GetCertificate: reloader.GetCertificate,
    ClientCAs:      certPool,
    ClientAuth:     tls.RequireAndVerifyClientCert,
    MinVersion:     tls.VersionTLS12,
}

Updated March 2026: Go 1.18 changed the default minimum TLS version to TLS 1.2 and deprecated TLS 1.0 and 1.1 — the MinVersion: tls.VersionTLS12 I set explicitly here is now the default, but it's still worth being explicit in security-sensitive code. Go 1.18+ also enables TLS 1.3 by default, and in practice most internal gRPC connections negotiate TLS 1.3 now. The TLS 1.3 cipher suites are not configurable (the spec mandates the suite), so any CipherSuites config you set is ignored for 1.3 connections. For FIPS compliance contexts, verify that your Go build's crypto backend supports TLS 1.3 — BoringCrypto-based builds do.

What Can Go Wrong

The most common mistake: mixing up RootCAs and ClientCAs. RootCAs is for verifying the remote party's cert as a client. ClientCAs is for verifying the connecting client's cert as a server. I've seen these swapped and the error messages from Go's TLS stack are not always helpful in pointing to the exact problem.

The second most common issue: ServerName in tls.Config. When the client connects, the TLS library checks that the server's certificate CN or SAN matches the ServerName. If you're connecting by IP address instead of hostname, you either need an IP SAN in the server cert or you need to set InsecureSkipVerify: true (don't do this in production — add the IP SAN instead).

mTLS adds maybe 1-2ms to connection establishment. For long-lived gRPC connections, that's negligible. For very high-frequency short connections, consider connection pooling to amortize the handshake cost.