Repository navigation
jwt_verify_cert intermittently returns -5 (JWT_VRFY_UNKNOWN_CERT) for valid tokens under multi-threaded load #3526
Description
Detailed Description of the Problem
With nbthread > 1 and concurrent requests, jwt_verify_cert(<alg>,<cert>) intermittently
returns -5 (JWT_VRFY_UNKNOWN_CERT) for valid tokens whose certificate is in the
certificate store with jwt on. jwt_verify(<alg>,<pubkey file>) with the same key and
tokens never fails. The rate grows with concurrency (~1% of fresh verifications at 64
concurrent connections, 2-4 threads). It never happens with one thread or with one
request in flight.
Expected Behavior
A valid token signed by a certificate loaded with jwt on always returns 1, regardless
of thread count, concurrency, or CLI activity.
Steps to Reproduce the Behavior
- Create key and certificate:
openssl ecparam -name prime256v1 -genkey -noout -out key.pem
openssl req -new -x509 -key key.pem -subj /CN=jwt -days 30 -out jwt.crt
openssl ec -in key.pem -pubout -out pub.pem- Start haproxy with the configuration below:
NBT=4 haproxy -f haproxy.cfg(or NBT=2). - Send 20,000 requests, each with a freshly signed ES256 token (so nothing is cached),
at 16-64 concurrent connections, and count thex-vresponse header values.
Client (Go, stdlib only):
package main
import (
"crypto/ecdsa"
"crypto/rand"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"encoding/pem"
"flag"
"fmt"
"log"
"net/http"
"os"
"sort"
"sync"
"sync/atomic"
)
func main() {
url := flag.String("url", "http://haproxy:8000/", "target")
keyFile := flag.String("key", "key.pem", "EC private key (PEM)")
n := flag.Int("n", 20000, "total requests")
c := flag.Int("c", 16, "concurrent connections")
flag.Parse()
b, _ := os.ReadFile(*keyFile)
blk, _ := pem.Decode(b)
key, err := x509.ParseECPrivateKey(blk.Bytes)
if err != nil {
log.Fatal(err)
}
enc := base64.RawURLEncoding
hdr := enc.EncodeToString([]byte(`{"alg":"ES256","typ":"JWT"}`))
var seq atomic.Int64
mint := func() string {
body := hdr + "." + enc.EncodeToString([]byte(fmt.Sprintf(`{"sub":"u","jti":"%d"}`, seq.Add(1))))
h := sha256.Sum256([]byte(body))
r, s, _ := ecdsa.Sign(rand.Reader, key, h[:])
sig := make([]byte, 64)
r.FillBytes(sig[:32])
s.FillBytes(sig[32:])
return body + "." + enc.EncodeToString(sig)
}
cl := &http.Client{Transport: &http.Transport{MaxIdleConnsPerHost: *c}}
var mu sync.Mutex
counts := map[string]int{}
var next atomic.Int64
var wg sync.WaitGroup
for i := 0; i < *c; i++ {
wg.Add(1)
go func() {
defer wg.Done()
local := map[string]int{}
for next.Add(1) <= int64(*n) {
req, _ := http.NewRequest("GET", *url, nil)
req.Header.Set("Authorization", "Bearer "+mint())
resp, err := cl.Do(req)
if err != nil {
local["err"]++
continue
}
resp.Body.Close()
local[resp.Header.Get("x-v")]++
}
mu.Lock()
for k, v := range local {
counts[k] += v
}
mu.Unlock()
}()
}
wg.Wait()
keys := make([]string, 0, len(counts))
for k := range counts {
keys = append(keys, k)
}
sort.Strings(keys)
fmt.Printf("n=%d c=%d", *n, *c)
for _, k := range keys {
fmt.Printf(" v[%s]=%d", k, counts[k])
}
fmt.Println()
}Run: client -url http://<host>:8000/ -key key.pem -n 20000 -c 64
(port 8001 for the jwt_verify comparison)
Results (-5 count per 20,000 requests; all other responses were 1)
Same behaviour when the certificate is loaded through the runtime API
(new/set/commit ssl cert, add ssl jwt) instead of crt-store. An immediate retry after
a -5 fails again ~3% of the time.
Do you have any idea what may have caused this?
3.4.6, src/jwt.c, jwt_jwsverify_rsa_ecdsa(): the x509 path (lines 417-431) takes
ckch_lock with HA_SPIN_TRYLOCK (line 419) and skips ckchs_lookup() when the lock is busy.
pubkey stays NULL, and line 448 returns JWT_VRFY_UNKNOWN_CERT (-5,
include/haproxy/jwt-t.h:82). The trylock dates from 522bca9 and was carried into
jwt_verify_cert by f5632fd. Current master has the same code.
ckch_lock is a plain HA_SPINLOCK_T (src/ssl_sock.c:676), so:
- every worker verifying a token takes it exclusively, and two concurrent
verifications are enough for one to fail (what the table above shows); - CLI commands hold it across their I/O handler and only release it in the release
callback (e.g. ssl_ckch.c:1704 forshow ssl sni, 2744 forshow ssl ca-file, and
the commit path around line 3170: "we don't unlock there, it will be unlock after
the IO handler, in the release handler"). While such a command runs, every
jwt_verify_cert call on every thread can return -5.
The key-file path (jwt_verify) uses jwt_cert_tree without locking and is unaffected.
History: 522bca9, which first added certificate support to jwt_verify, says the
ckch_store tree is locked only the first time a given key value is seen, and the
result is then cached in the jwt tree so later calls avoid the lookup. The current
jwt_verify_cert path (f5632fd onward) has no such cache: it calls ckchs_lookup()
under the trylock on every verification. That is presumably so runtime certificate
updates take effect, but it puts ckch_lock on the hot path.
Do you have an idea how to solve the issue?
Blocking on ckch_lock does not look acceptable, since CLI commands can hold it for
long periods. The same "trylock failure becomes an error" pattern is already flagged
in hlua.c (CertCache.set: "FIXME: this should not return an error but should come back
later"), but a converter cannot yield and come back.
Suggested direction: when a ckch_store is loaded or committed with jwt on, keep a
refcounted EVP_PKEY in a structure outside ckch_lock (e.g. in jwt_cert_tree, or a
per-store pointer swapped atomically on commit). Verification then only takes a
reference, under a read lock or RCU-style, and never touches ckch_lock. This would
also restore the original intent of 522bca9 (no ckch_store lookup per
verification) while keeping runtime updates working.
At minimum, a failed trylock should not be reported as "unknown certificate", and the
behaviour should be documented.
What is your configuration?
global
nbthread "${NBT}"
defaults
mode http
timeout client 10s
timeout connect 5s
timeout server 10s
crt-store
load crt "/w/jwt.crt" jwt on
frontend cert
bind :8000
http-request set-var(txn.v) req.hdr(authorization),word(2,' '),jwt_verify_cert("ES256","/w/jwt.crt")
http-request return status 200 hdr x-v %[var(txn.v)]
frontend keyfile
bind :8001
http-request set-var(txn.v) req.hdr(authorization),word(2,' '),jwt_verify("ES256","/w/pub.pem")
http-request return status 200 hdr x-v %[var(txn.v)]
Output of haproxy -vv
HAProxy version 3.4.6-56332c5 2026/09/28
Official Docker image haproxy:3.4
(haproxy@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26)
OpenSSL 3.5.7
Linux 6.8 aarch64 (Docker in a VM on Apple silicon); HAProxy pinned to 4 CPUs,
client on 3 other CPUs.
Last Outputs and Backtraces
n/a (no crash; incorrect return value)
Additional Information
Affected since 3.3.0 (jwt_verify_cert was added in f5632fd, first tag v3.3-dev10);
reproduced on 3.4.6, and current master has the same code.
Workarounds: use jwt_verify with a public-key file (no runtime key updates), or re-run
jwt_verify_cert when it returns -5. Note that matching -5 with -m int -- -5 does not
work as expected (it also matches 1); -m str -- -5 does.
Activity
CarterSmarty commentedon Oct 8, 2026
Full
haproxy -vvoutput:wlallemand commentedon Oct 9, 2026
Indeed that doesn't look right, it should be done the same way the SNI lookup is done, having only a read lock when accessing it.
rlebreton commentedon Oct 9, 2026
Even with a read/write lock, the CLI commands working on the ckch_stores would keep the write lock a long time so the read tries would be stuck? It might be possible to keep a read lock by default on the cli commands and promote the lock only around the
ckch_store_replacecall. In theory it looks like it should work but I might be missing something.wlallemand commentedon Oct 9, 2026
@rlebreton indeed, but doing it like the SNIs implies another tree, where the SSL_CTX count ref would be incremented. That would change a bit how it's done, and it would rollback a bit to how it was done with the cache tree.