Tests are now properly run against OTP, including this: "The test does not verify that ciphertext and shared secret correspond"

pick and choose:

PR Re-review: otp_crypto: Add ML-KEM-768 key encapsulation

Commit reviewed: 8dbf458969f8b2b994cc3e0b482fb041b784ccfe
Previous version reviewed: f337e64cd292422240f578c82796c4ac02e9a4b4
Recommendation: Request changes

The update resolves all three findings from the first review:

• ref-counted ML-KEM output allocations are now checked and failed heap descriptors are trimmed;
• static libsodium's private link dependencies and options are propagated;
• the test now generates a keypair and proves encapsulation/decapsulation agreement.

It also improves OTP compatibility by exposing the OTP 28.1 encapsulate_key/2 and
decapsulate_key/3 API. Two newly exposed edge cases remain.

Findings

[P2] Validate the embedded public-key hash before decapsulation

Location: src/libAtomVM/otp_crypto.c:3335-3355

The NIF validates only that the private key is 2400 bytes. An ML-KEM-768 private key contains the
embedded public key at bytes 1152–2335 and H(ek) at bytes 2336–2367. libsodium 1.0.22's
decapsulation implementation trusts the stored hash and always returns success; it does not validate
that the hash still matches the embedded public key.

This differs observably from OTP/OpenSSL. Flipping byte 2336 in a generated private key produces:

OTP 29:   raises "Can't create PKEY fromdata"
AtomVM:   returns a 32-byte secret different from the original secret

This is not ML-KEM's intended implicit rejection of an invalid ciphertext: the private key itself is
malformed. Validate H(ek) before calling crypto_kem_mlkem768_dec():

diff --git a/src/libAtomVM/otp_crypto.c b/src/libAtomVM/otp_crypto.c
--- a/src/libAtomVM/otp_crypto.c
+++ b/src/libAtomVM/otp_crypto.c
@@ -3348,7 +3348,25 @@ static term nif_crypto_decapsulate_key(Context *ctx, int argc, term argv[])
 
     unsigned char ss[crypto_kem_mlkem768_SHAREDSECRETBYTES];
 
     do_sodium_init();
+
+    const unsigned char *embedded_pk = sk + crypto_kem_mlkem768_SECRETKEYBYTES
+        - crypto_kem_mlkem768_PUBLICKEYBYTES - (2 * crypto_hash_sha3256_BYTES);
+    const unsigned char *stored_pk_hash
+        = sk + crypto_kem_mlkem768_SECRETKEYBYTES - (2 * crypto_hash_sha3256_BYTES);
+    unsigned char computed_pk_hash[crypto_hash_sha3256_BYTES];
+    if (UNLIKELY(crypto_hash_sha3256(computed_pk_hash, embedded_pk,
+                     crypto_kem_mlkem768_PUBLICKEYBYTES)
+            != 0)) {
+        sodium_memzero(computed_pk_hash, sizeof computed_pk_hash);
+        RAISE_ERROR(make_crypto_error(__FILE__, __LINE__, "ML-KEM-768 key validation failed", ctx));
+    }
+    if (UNLIKELY(sodium_memcmp(computed_pk_hash, stored_pk_hash, sizeof computed_pk_hash) != 0)) {
+        sodium_memzero(computed_pk_hash, sizeof computed_pk_hash);
+        RAISE_ERROR(BADARG_ATOM);
+    }
+    sodium_memzero(computed_pk_hash, sizeof computed_pk_hash);
+
     if (UNLIKELY(crypto_kem_mlkem768_dec(ss, ct, sk) != 0)) {
         sodium_memzero(ss, sizeof ss);
         RAISE_ERROR(make_crypto_error(__FILE__, __LINE__, "ML-KEM-768 decapsulation failed", ctx));

Add a regression that distinguishes malformed-key validation from ciphertext implicit rejection:

diff --git a/tests/erlang_tests/test_crypto_pk.erl b/tests/erlang_tests/test_crypto_pk.erl
--- a/tests/erlang_tests/test_crypto_pk.erl
+++ b/tests/erlang_tests/test_crypto_pk.erl
@@ -483,6 +483,10 @@ test_kem_round_trip() ->
     ok = expect_error(fun() -> crypto:decapsulate_key(mlkem512, Priv, Ct) end),
     ok = expect_error(fun() -> crypto:decapsulate_key(mlkem768, <<0:8>>, Ct) end),
     ok = expect_error(fun() -> crypto:decapsulate_key(mlkem768, Priv, <<0:8>>) end),
+    <<Prefix:2336/binary, HashByte, Tail/binary>> = Priv,
+    BadPriv = <<Prefix/binary, (HashByte bxor 1), Tail/binary>>,
+    ok = expect_error(fun() -> crypto:decapsulate_key(mlkem768, BadPriv, Ct) end),
     ok = expect_error(fun() -> crypto:generate_key(mlkem768, x25519) end),
     ok.

[P1] Include pkg-config's explicit libdir when locating libsodium.a

Location: src/platforms/generic_unix/lib/CMakeLists.txt:120

The revised NO_DEFAULT_PATH correctly prevents pairing pkg-config's headers with an unrelated
archive, but it searches only ${LIBSODIUM_LIBRARY_DIRS}. pkg-config intentionally omits system
directories such as /usr/lib from --libs-only-L; therefore that variable can be empty even when
the .pc file points to an installed /usr/lib/libsodium.a. With the project's supported CMake
3.13, pkg_check_modules() separately exposes the .pc file's libdir as
${LIBSODIUM_LIBDIR}.

This was reproduced with a valid .pc file whose libdir=/usr/lib: pkg-config --libs-only-L
returned nothing while pkg-config --variable=libdir returned /usr/lib.

diff --git a/src/platforms/generic_unix/lib/CMakeLists.txt b/src/platforms/generic_unix/lib/CMakeLists.txt
--- a/src/platforms/generic_unix/lib/CMakeLists.txt
+++ b/src/platforms/generic_unix/lib/CMakeLists.txt
@@ -117,7 +117,9 @@ if (AVM_USE_LIBSODIUM)
 
     target_include_directories(libAtomVM${PLATFORM_LIB_SUFFIX} PUBLIC ${LIBSODIUM_INCLUDE_DIRS})
     if (AVM_STATIC_LIBSODIUM)
-        find_library(LIBSODIUM_STATIC_LIB NAMES libsodium.a PATHS ${LIBSODIUM_LIBRARY_DIRS} NO_DEFAULT_PATH)
+        find_library(LIBSODIUM_STATIC_LIB NAMES libsodium.a
+            PATHS ${LIBSODIUM_LIBRARY_DIRS} ${LIBSODIUM_LIBDIR}
+            NO_DEFAULT_PATH)
         if (LIBSODIUM_STATIC_LIB STREQUAL "LIBSODIUM_STATIC_LIB-NOTFOUND")
             message(FATAL_ERROR "AVM_STATIC_LIBSODIUM=ON but libsodium.a was not found in ${LIBSODIUM_LIBRARY_DIRS}")
         endif()

The newly added static dependency and linker-option propagation should remain unchanged.

Verification performed

• Asked Oracle to review the fixes and expanded API; both findings above were independently reproduced.
• git diff --check HEAD^ HEAD — passed.
• clang-format --dry-run --Werror src/libAtomVM/otp_crypto.c — passed.
• erlc -Werror tests/erlang_tests/test_crypto_pk.erl — passed.
• Full macOS build with libsodium 1.0.22, AVM_USE_LIBSODIUM=ON, and
  AVM_STATIC_LIBSODIUM=ON — passed.
• otool -L confirmed the resulting AtomVM has no dynamic libsodium dependency.
• Updated test_crypto_pk.beam on that AtomVM — returned 0, including ML-KEM key generation,
  encapsulation, decapsulation, and implicit-rejection cases.
• The same Erlang test on OTP 29/OpenSSL 3.5 — returned 0.
• Native test-erlang suite — passed.