Fooling Go's X.509 Certificate Verification

Below are two X.509 certificates. The first is the Certificate Authority (CA) root certificate, and the second is a leaf certifcate signed by the private key of the CA.

ca.crt.pem

leaf.crt.pem

If you downloaded these certificates, you could visually see that the latter references the former as its Issuer. If you were to use a tool like openssl to verify that the leaf is signed by the private key of root, you would see that it is.

Unless of course you are reading this blog post from the year 2126 or you have changed the system time on your machine. If the former, I am exceedingly dissapointed that humanity is still using openssl.

openssl verify -CAfile ca.crt.pem leaf.crt.pem

Now, if you wanted to write a Go program that verified this chain of trust, it might look something like the following.

main.go

package main

import (
	"encoding/pem"
	"crypto/x509"
	"fmt"
	"os"
	"time"
)

func main() {
	b, err := os.ReadFile("ca.crt.pem")
	if err != nil {
		panic(err)
	}
	block, _ := pem.Decode(b)
	ca, err := x509.ParseCertificate(block.Bytes)
	if err != nil {
		panic(err)
	}
	b, err = os.ReadFile("leaf.crt.pem")
	if err != nil {
		panic(err)
	}
	block, _ = pem.Decode(b)
	lc, err := x509.ParseCertificate(block.Bytes)
	if err != nil {
		panic(err)
	}
	roots := x509.NewCertPool()
	roots.AddCert(ca)
	opts := x509.VerifyOptions{
		Roots:       roots,
		CurrentTime: time.Now(),
	}
	if _, err := lc.Verify(opts); err != nil {
		panic(err)
	}
    fmt.Println("Certificate verification successful.")
}

But if you ran that program, you might be surprised to see the following.

panic: x509: certificate signed by unknown authority

If you used this CA certificate instead, you would see the expected output.

ca.verifies.crt.pem

Certificate verification successful.

At first glance these certificates appear to be identical. You could use openssl to view the contents of both certificates, and you would get identical output.

openssl x509 -in ca.crt.pem -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            75:ae:14:be:51:73:c1:01:0e:fd:f0:f4:7f:88:40:9e:3f:b2:74:f6
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = Root CA
        Validity
            Not Before: Feb 27 19:47:46 2026 GMT
            Not After : Feb  3 19:47:46 2126 GMT
        Subject: CN = Root CA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:a2:f9:04:1f:5a:69:0d:93:b4:d8:14:ca:61:2c:
                    6b:ff:ac:d9:99:53:5d:59:65:d0:dd:b6:28:a5:b1:
                    87:43:7f:25:d8:07:72:3a:ca:7f:3a:91:95:5a:ed:
                    50:dd:86:5d:db:ec:74:b3:aa:d7:94:f1:06:af:7b:
                    5a:f1:34:ee:52
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                C0:54:4C:3D:76:84:B3:63:50:EE:3D:5C:4F:12:7C:04:A7:FF:95:24
            X509v3 Authority Key Identifier: 
                C0:54:4C:3D:76:84:B3:63:50:EE:3D:5C:4F:12:7C:04:A7:FF:95:24
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:44:02:20:7a:d2:4e:4f:ca:24:70:23:9f:73:02:fd:e8:2f:
        7f:13:0c:d4:da:46:18:4c:cc:07:b1:07:f9:55:d6:cf:b4:5f:
        02:20:32:87:0c:b8:13:d8:5c:f3:fa:ac:4c:8f:b8:88:a7:f6:
        a3:fc:cc:46:ee:47:d8:63:49:38:ba:8e:9e:08:25:6d

However, if you were to compare the bytes of the certificates, you would see that there is a very slight difference; two bytes to be exact.

diff <(openssl x509 -in ca.crt.pem -outform der | xxd) <(openssl x509 -in ca.verifies.crt.pem -outform der | xxd)

4c4
< 00000030: 1231 1030 0e06 0355 0403 1307 526f 6f74  .1.0...U....Root
---
> 00000030: 1231 1030 0e06 0355 0403 0c07 526f 6f74  .1.0...U....Root
8c8
< 00000070: 1307 526f 6f74 2043 4130 5930 1306 072a  ..Root CA0Y0...*
---
> 00000070: 0c07 526f 6f74 2043 4130 5930 1306 072a  ..Root CA0Y0...*

In both locations, there is a 0x13 byte in the certificate which failed verification with the Go program (ca.crt.pem), and a 0x0c byte in the certificate that passed (ca.verifies.crt.pem). If you are familiar with X.509 certificates, you’ll know that they are defined using Abstract Syntax Notation One (ASN.1) and are binary encoded using Distinguished Encoding Rules (DER). They are frequently Base64 encoded and stored and transmitted as Privacy-enhanced Mail (PEM) text files (which you have already seen in this post).

The ASN.1 specification defines a set of data types, each with an associated tag (non-negative integer identifier), which precedes the length and the value when using DER encoding (see this post from Let’s Encrypt for more information). openssl can once again be used to see the data types of different fields in the certificate that is successfully verified.

openssl asn1parse -in ca.verifies.crt.pem

    0:d=0  hl=4 l= 378 cons: SEQUENCE          
    4:d=1  hl=4 l= 289 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=  20 prim: INTEGER           :75AE14BE5173C1010EFDF0F47F88409E3FB274F6
   35:d=2  hl=2 l=  10 cons: SEQUENCE          
   37:d=3  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
   47:d=2  hl=2 l=  18 cons: SEQUENCE          
   49:d=3  hl=2 l=  16 cons: SET               
   51:d=4  hl=2 l=  14 cons: SEQUENCE          
   53:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   58:d=5  hl=2 l=   7 prim: UTF8STRING        :Root CA
   67:d=2  hl=2 l=  32 cons: SEQUENCE          
   69:d=3  hl=2 l=  13 prim: UTCTIME           :260227194746Z
   84:d=3  hl=2 l=  15 prim: GENERALIZEDTIME   :21260203194746Z
  101:d=2  hl=2 l=  18 cons: SEQUENCE          
  103:d=3  hl=2 l=  16 cons: SET               
  105:d=4  hl=2 l=  14 cons: SEQUENCE          
  107:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  112:d=5  hl=2 l=   7 prim: UTF8STRING        :Root CA
  121:d=2  hl=2 l=  89 cons: SEQUENCE          
  123:d=3  hl=2 l=  19 cons: SEQUENCE          
  125:d=4  hl=2 l=   7 prim: OBJECT            :id-ecPublicKey
  134:d=4  hl=2 l=   8 prim: OBJECT            :prime256v1
  144:d=3  hl=2 l=  66 prim: BIT STRING        
  212:d=2  hl=2 l=  83 cons: cont [ 3 ]        
  214:d=3  hl=2 l=  81 cons: SEQUENCE          
  216:d=4  hl=2 l=  29 cons: SEQUENCE          
  218:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Subject Key Identifier
  223:d=5  hl=2 l=  22 prim: OCTET STRING      [HEX DUMP]:0414C0544C3D7684B36350EE3D5C4F127C04A7FF9524
  247:d=4  hl=2 l=  31 cons: SEQUENCE          
  249:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Authority Key Identifier
  254:d=5  hl=2 l=  24 prim: OCTET STRING      [HEX DUMP]:30168014C0544C3D7684B36350EE3D5C4F127C04A7FF9524
  280:d=4  hl=2 l=  15 cons: SEQUENCE          
  282:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Basic Constraints
  287:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  290:d=5  hl=2 l=   5 prim: OCTET STRING      [HEX DUMP]:30030101FF
  297:d=1  hl=2 l=  10 cons: SEQUENCE          
  299:d=2  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
  309:d=1  hl=2 l=  71 prim: BIT STRING

In the diff view of the two CA certificates, the bytes that differed preceded the Root CA string in two different places: the Subject and the Issuer, which are the same since this is a self-signed certificate. They were also followed by a 0x07 byte, which aligns with the number of characters in Root CA (i.e. the length of the value). The differing leading byte suggests differing ASN.1 data types for these fields. The CA certificate for which validation is successful uses UTF8String (0x0c), and you can use openssl with the failing CA certificate to see that it uses PrintableString instead (0x13).

openssl asn1parse -in ca.crt.pem

    0:d=0  hl=4 l= 378 cons: SEQUENCE          
    4:d=1  hl=4 l= 289 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=  20 prim: INTEGER           :75AE14BE5173C1010EFDF0F47F88409E3FB274F6
   35:d=2  hl=2 l=  10 cons: SEQUENCE          
   37:d=3  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
   47:d=2  hl=2 l=  18 cons: SEQUENCE          
   49:d=3  hl=2 l=  16 cons: SET               
   51:d=4  hl=2 l=  14 cons: SEQUENCE          
   53:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   58:d=5  hl=2 l=   7 prim: PRINTABLESTRING   :Root CA
   67:d=2  hl=2 l=  32 cons: SEQUENCE          
   69:d=3  hl=2 l=  13 prim: UTCTIME           :260227194746Z
   84:d=3  hl=2 l=  15 prim: GENERALIZEDTIME   :21260203194746Z
  101:d=2  hl=2 l=  18 cons: SEQUENCE          
  103:d=3  hl=2 l=  16 cons: SET               
  105:d=4  hl=2 l=  14 cons: SEQUENCE          
  107:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  112:d=5  hl=2 l=   7 prim: PRINTABLESTRING   :Root CA
  121:d=2  hl=2 l=  89 cons: SEQUENCE          
  123:d=3  hl=2 l=  19 cons: SEQUENCE          
  125:d=4  hl=2 l=   7 prim: OBJECT            :id-ecPublicKey
  134:d=4  hl=2 l=   8 prim: OBJECT            :prime256v1
  144:d=3  hl=2 l=  66 prim: BIT STRING        
  212:d=2  hl=2 l=  83 cons: cont [ 3 ]        
  214:d=3  hl=2 l=  81 cons: SEQUENCE          
  216:d=4  hl=2 l=  29 cons: SEQUENCE          
  218:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Subject Key Identifier
  223:d=5  hl=2 l=  22 prim: OCTET STRING      [HEX DUMP]:0414C0544C3D7684B36350EE3D5C4F127C04A7FF9524
  247:d=4  hl=2 l=  31 cons: SEQUENCE          
  249:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Authority Key Identifier
  254:d=5  hl=2 l=  24 prim: OCTET STRING      [HEX DUMP]:30168014C0544C3D7684B36350EE3D5C4F127C04A7FF9524
  280:d=4  hl=2 l=  15 cons: SEQUENCE          
  282:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Basic Constraints
  287:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  290:d=5  hl=2 l=   5 prim: OCTET STRING      [HEX DUMP]:30030101FF
  297:d=1  hl=2 l=  10 cons: SEQUENCE          
  299:d=2  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
  309:d=1  hl=2 l=  71 prim: BIT STRING

This still doesn’t explain why openssl verifies successfully with either CA certificate, while the Go program does not. To dig further, you can compile and step through the program with gdb, starting with a breakpoint on Verify().

gdb main -ex 'b crypto/x509.(*Certificate).Verify' -ex 'run'

Stepping through the function, you eventually arrive at the point where you are building the candidate certificate chains.

Add a breakpoint for this function using b crypto/x509.(*Certificate).buildChains.

	var candidateChains [][]*Certificate
	if opts.Roots.contains(c) {
		candidateChains = [][]*Certificate{{c}}
	} else {
		candidateChains, err = c.buildChains([]*Certificate{c}, nil, &opts)
		if err != nil {
			return nil, err
		}
	}

As part of evaluating whether the certificate pool provided has a candidate chain, findPotentialParents() is called on the Roots (it is also called on the Intermediates, but there are no intermediate certificates provided in this example).

	for _, root := range opts.Roots.findPotentialParents(c) {
		considerCandidate(rootCertificate, root)
	}

Finally, you arrive at the source of the failure. A potential parent for the leaf certificate should have a Subject that matches the Issuer of the leaf (i.e. the leaf should refer to it as the certificate that was used for signing).

	for _, c := range s.byName[string(cert.RawIssuer)] {
		candidate, constraint, err := s.cert(c)
		if err != nil {
			continue
		}
		kidMatch := bytes.Equal(candidate.SubjectKeyId, cert.AuthorityKeyId)
		switch {
		case kidMatch:
			matchingKeyID = append(matchingKeyID, potentialParent{candidate, constraint})
		case (len(candidate.SubjectKeyId) == 0 && len(cert.AuthorityKeyId) > 0) ||
			(len(candidate.SubjectKeyId) > 0 && len(cert.AuthorityKeyId) == 0):
			oneKeyID = append(oneKeyID, potentialParent{candidate, constraint})
		default:
			mismatchKeyID = append(mismatchKeyID, potentialParent{candidate, constraint})
		}
	}

The keys in the byName map of a CertPool contain the Subject of the CA certificates. When using the CA certificate that caused verification failure, stepping through the loop above you can see that there are zero iterations, or, that there are no CA certificates with a Subject that matches the leaf Issuer. How could that be? The key observation is that the raw Subject and Issuer, the literal bytes, are being used for comparison.

// CertPool is a set of certificates.
type CertPool struct {
	byName map[string][]int // cert.RawSubject => index into lazyCerts

	// lazyCerts contains funcs that return a certificate,
	// lazily parsing/decompressing it as needed.
	lazyCerts []lazyCert

	// haveSum maps from sum224(cert.Raw) to true. It's used only
	// for AddCert duplicate detection, to avoid CertPool.contains
	// calls in the AddCert path (because the contains method can
	// call getCert and otherwise negate savings from lazy getCert
	// funcs).
	haveSum map[sum224]bool

	// systemPool indicates whether this is a special pool derived from the
	// system roots. If it includes additional roots, it requires doing two
	// verifications, one using the roots provided by the caller, and one using
	// the system platform verifier.
	systemPool bool
}

We saw earlier that the two CA certificates differed in the ASN.1 data types used for their Subject. Expectedly, if you check the data type of the Issuer in the leaf certificate, you’ll see that it is a UTF8String, matching the CA certificate that was verified successfully.

openssl asn1parse -in leaf.crt.pem

    0:d=0  hl=4 l= 286 cons: SEQUENCE          
    4:d=1  hl=3 l= 196 cons: SEQUENCE          
    7:d=2  hl=2 l=  20 prim: INTEGER           :2C4DE1BE763153DD60F5CF47DFEB86092A978B83
   29:d=2  hl=2 l=  10 cons: SEQUENCE          
   31:d=3  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
   41:d=2  hl=2 l=  18 cons: SEQUENCE          
   43:d=3  hl=2 l=  16 cons: SET               
   45:d=4  hl=2 l=  14 cons: SEQUENCE          
   47:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   52:d=5  hl=2 l=   7 prim: UTF8STRING        :Root CA
   61:d=2  hl=2 l=  32 cons: SEQUENCE          
   63:d=3  hl=2 l=  13 prim: UTCTIME           :260227194746Z
   78:d=3  hl=2 l=  15 prim: GENERALIZEDTIME   :21260203194746Z
   95:d=2  hl=2 l=  15 cons: SEQUENCE          
   97:d=3  hl=2 l=  13 cons: SET               
   99:d=4  hl=2 l=  11 cons: SEQUENCE          
  101:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  106:d=5  hl=2 l=   4 prim: UTF8STRING        :leaf
  112:d=2  hl=2 l=  89 cons: SEQUENCE          
  114:d=3  hl=2 l=  19 cons: SEQUENCE          
  116:d=4  hl=2 l=   7 prim: OBJECT            :id-ecPublicKey
  125:d=4  hl=2 l=   8 prim: OBJECT            :prime256v1
  135:d=3  hl=2 l=  66 prim: BIT STRING        
  203:d=1  hl=2 l=  10 cons: SEQUENCE          
  205:d=2  hl=2 l=   8 prim: OBJECT            :ecdsa-with-SHA256
  215:d=1  hl=2 l=  73 prim: BIT STRING

Whether this is the correct behavior has been an ongoing debate in the Go project, and the matter is complicated by some tools, such as openssl as seen in this post, treating different ASN.1 data types for strings as equivalent when verifying certificates. Typically you’ll be using the same tooling or services for generating CA certificates and the leaf certificates they sign, so the encoding will likely be consistent. However, given that leaf certificates are typically much shorter lived than CA certificates, your tooling may evolve over time, potentially causing discrepancies in newly generated leaves.

Though Go’s handling of this scenario results in fail-closed behavior, it can still cause outages and downtime, making it import to be aware of how you are generating certificates and how they are expected to be verified.