Post-Quantum Hybrid Key Exchange

Post-Quantum Cryptography

This document describes the post-quantum cryptography (PQC) support in GlaSSLess, including the standalone algorithms (ML-KEM, ML-DSA, SLH-DSA, LMS/HSS) and the hybrid key exchange schemes that combine classical and post-quantum cryptography.

Requirements

  • OpenSSL 3.5+ for lattice-based PQC algorithms (ML-KEM, ML-DSA, SLH-DSA)

  • OpenSSL 3.6+ for hash-based PQC signatures (LMS/HSS)

  • Java 25+ with --enable-native-access flag

Standalone PQC Algorithms

GlaSSLess supports the NIST-standardized post-quantum algorithms:

Algorithm Standard Type Variants
ML-KEM FIPS 203 Key Encapsulation ML-KEM-512, ML-KEM-768, ML-KEM-1024
ML-DSA FIPS 204 Digital Signature ML-DSA-44, ML-DSA-65, ML-DSA-87
SLH-DSA FIPS 205 Digital Signature 12 variants (SHA2/SHAKE, 128/192/256, s/f)
LMS/HSS SP 800-208 Digital Signature (verification only) RFC 8554

LMS/HSS Hash-Based Signatures

LMS (Leighton-Micali Signature) / HSS (Hierarchical Signature System) is a stateful hash-based signature scheme standardized in RFC 8554 and NIST SP 800-208. Unlike ML-DSA and SLH-DSA, LMS is stateful -- each private key can only produce a limited number of signatures, and state must be carefully managed to avoid key reuse.

GlaSSLess supports verification only . Signing and key generation are not supported because state management must be handled by the key owner.

  • Signature algorithm : LMS (aliases: HSS, id-alg-hss-lms-hashsig)

  • KeyFactory : LMS -- loads public keys from X.509/SubjectPublicKeyInfo encoding

  • Requires : OpenSSL 3.6+ with LMS support enabled

See Usage Guide for code examples.

Hybrid Key Encapsulation Mechanisms

Hybrid KEMs combine a classical key exchange algorithm (X25519 or X448) with ML-KEM to provide security against both classical and quantum computers. This "hybrid" approach ensures that the key exchange remains secure even if one of the underlying algorithms is broken.

Supported Hybrid KEMs

Algorithm Classical Component PQC Component Shared Secret Size Ciphertext Size
X25519MLKEM768 X25519 ML-KEM-768 64 bytes 1120 bytes
X448MLKEM1024 X448 ML-KEM-1024 64 bytes 1632 bytes

Unsupported Variants

The following hybrid KEMs are defined in standards but not currently supported in GlaSSLess:

Algorithm Classical Component PQC Component Status
SecP256r1MLKEM768 P-256 (secp256r1) ML-KEM-768 Blocked by OpenSSL
SecP384r1MLKEM1024 P-384 (secp384r1) ML-KEM-1024 Blocked by OpenSSL
Technical Details

OpenSSL 3.5.x supports these algorithms for in-memory operations but lacks key serialization support:

Feature X25519MLKEM768 SecP256r1MLKEM768
Key generation Yes Yes
Encapsulation Yes Yes
Decapsulation Yes Yes

EVP_PKEY_get_raw_public_key

Yes No

EVP_PKEY_get_raw_private_key

Yes No
DER/PEM encoding No No

EVP_PKEY_fromdata

Yes No

EVP_PKEY_dup

Yes No

Java's JCA model requires keys to be serializable via Key.getEncoded() and reconstructable via KeyFactory. Without OpenSSL serialization support, GlaSSLess cannot:

  1. Implement getEncoded() for public/private keys

  2. Support KeyFactory.generatePublic() / generatePrivate()

  3. Enable key persistence or transmission between JVMs

The XDH-based hybrids (X25519MLKEM768, X448MLKEM1024) work because OpenSSL provides raw key export functions for them.

Future Support

EC-based hybrid KEMs will be added to GlaSSLess when OpenSSL adds proper key serialization support, expected in a future 3.5.x patch or 3.6 release. Track progress at:

Usage Examples

Basic Hybrid Key Exchange
import javax.crypto.KEM;
import java.security.*;
import net.glassless.provider.GlaSSLessProvider;

Security.addProvider(new GlaSSLessProvider());

// Server: Generate hybrid key pair
KeyPairGenerator kpg = KeyPairGenerator.getInstance("X25519MLKEM768", PROVIDER_NAME);
KeyPair serverKeyPair = kpg.generateKeyPair();

// Server sends public key to client...
PublicKey serverPublicKey = serverKeyPair.getPublic();

// Client: Encapsulate a shared secret
KEM kem = KEM.getInstance("X25519MLKEM768", PROVIDER_NAME);
KEM.Encapsulator encapsulator = kem.newEncapsulator(serverPublicKey);
KEM.Encapsulated encapsulated = encapsulator.encapsulate();

// Client has: shared secret and ciphertext to send to server
SecretKey clientSharedSecret = encapsulated.key();
byte[] ciphertext = encapsulated.encapsulation();

// Client sends ciphertext to server...

// Server: Decapsulate to recover the same shared secret
KEM.Decapsulator decapsulator = kem.newDecapsulator(serverKeyPair.getPrivate());
SecretKey serverSharedSecret = decapsulator.decapsulate(ciphertext);

// Both parties now have the same 64-byte shared secret
assert Arrays.equals(clientSharedSecret.getEncoded(), serverSharedSecret.getEncoded());
Deriving AES Key from Shared Secret

The 64-byte shared secret should be processed through a KDF to derive application keys: 

import javax.crypto.KDF;
import javax.crypto.SecretKey;
import javax.crypto.spec.HKDFParameterSpec;

// After key exchange, derive an AES-256 key
KDF hkdf = KDF.getInstance("HKDF-SHA256", PROVIDER_NAME);

byte[] info = "my-application v1.0".getBytes();
HKDFParameterSpec params = HKDFParameterSpec
    .ofExtract()
    .addIKM(sharedSecret.getEncoded())
    .addSalt(new byte[32])  // Can use a fixed salt or random nonce
    .thenExpand(info, 32);  // 32 bytes for AES-256

SecretKey aesKey = hkdf.deriveKey("AES", params);
Partial Key Extraction

You can extract a specific portion of the shared secret directly during encapsulation: 

// Extract only the first 32 bytes as an AES key
KEM.Encapsulated encapsulated = encapsulator.encapsulate(0, 32, "AES");
SecretKey aesKey = encapsulated.key();  // 32-byte AES key
Checking Algorithm Availability
import net.glassless.provider.internal.OpenSSLCrypto;

// Check if hybrid KEMs are available
if (OpenSSLCrypto.isAlgorithmAvailable("KEYMGMT", "X25519MLKEM768")) {
    System.out.println("X25519MLKEM768 is available");
} else {
    System.out.println("X25519MLKEM768 requires OpenSSL 3.5+");
}
Key Serialization

Hybrid KEM keys use a custom encoding format (not standard ASN.1) due to OpenSSL limitations: 

// Get encoded key bytes
byte[] publicKeyBytes = serverKeyPair.getPublic().getEncoded();
byte[] privateKeyBytes = serverKeyPair.getPrivate().getEncoded();

// Reconstruct keys using KeyFactory
KeyFactory kf = KeyFactory.getInstance("X25519MLKEM768", PROVIDER_NAME);

// For public key
X509EncodedKeySpec pubSpec = new X509EncodedKeySpec(publicKeyBytes);
PublicKey reconstructedPublic = kf.generatePublic(pubSpec);

// For private key
PKCS8EncodedKeySpec privSpec = new PKCS8EncodedKeySpec(privateKeyBytes);
PrivateKey reconstructedPrivate = kf.generatePrivate(privSpec);

TLS 1.3 Integration

JDK Version Support

JDK Version Hybrid TLS Support GlaSSLess Role
JDK 24–26 None — JSSE has a hardcoded named group list with no hybrid groups

Application-layer hybrid KEM only (see Application-Layer Hybrid Key Exchange )
JDK 27+ (JEP 527)

Native — X25519MLKEM768 is the default TLS 1.3 named group

 FIPS compliance: routes ML-KEM operations through OpenSSL's FIPS-validated module

JDK 24–26: Why Hybrid TLS Is Not Possible

JDK 25/26 JSSE has three limitations that cannot be worked around by a JCA provider like GlaSSLess:

  1. Hardcoded named groups. The NamedGroup enum in sun.security.ssl is internal. There is no SPI to register new TLS named groups.

  2. No KEM-based key exchange. The TLS handshake code only knows KeyAgreement (interactive, both sides contribute). It does not use the javax.crypto.KEM API (asymmetric: encapsulate/decapsulate).

  3. No extensible handshake logic. The JSSE handshake state machine is in sun.security.ssl, which is not a public API. It cannot be subclassed or extended.

JDK 27+: JEP 527 Integration

JEP 527 modifies JSSE itself to support hybrid post-quantum key exchange. It builds on JEP 496 (ML-KEM), which was delivered in JDK 24.

What JEP 527 Changes

  • Adds hybrid named groups (X25519MLKEM768, SecP256r1MLKEM768, SecP384r1MLKEM1024) to JSSE's internal registry

  • Teaches the TLS handshake to use the javax.crypto.KEM SPI for key exchange — not just KeyAgreement

  • Makes X25519MLKEM768 the default preferred named group — applications get PQ protection without code changes

  • Allows configuration via SSLParameters.setNamedGroups() and the jdk.tls.namedGroups system property

Architecture: How JSSE Resolves Providers

JSSE uses a layered architecture for hybrid key exchange. Critically, the component algorithm lookups go through standard JCA provider resolution : 

JSSE TLS 1.3 Handshake
  └─ NamedGroup.X25519MLKEM768
       └─ HybridProvider (internal to JSSE, not in system provider list)
            └─ Hybrid.KEMImpl(left="ML-KEM-768", right="X25519")
                 ├─ KEM.getInstance("ML-KEM")              ← standard JCA lookup
                 └─ KEM.getInstance("DH", HybridProvider)  ← X25519-as-KEM wrapper
            └─ Hybrid.KeyPairGeneratorImpl
                 ├─ KeyPairGenerator.getInstance("ML-KEM-768")  ← standard JCA lookup
                 └─ KeyPairGenerator.getInstance("X25519")      ← standard JCA lookup

The JDK source explicitly notes: "This is done to work with 3rd-party providers that only have 'ML-KEM' KEM algorithm."

This means that when GlaSSLess is registered at a higher priority than SunJCE, the ML-KEM and X25519 operations within the TLS handshake are routed through GlaSSLess — and therefore through OpenSSL.

FIPS Compliance Use Case

JDK 27's SunJCE includes a pure-Java ML-KEM implementation that is performant but not FIPS-validated . For organizations that require FIPS 140-3 compliance, GlaSSLess provides the bridge between JSSE's TLS protocol handling and OpenSSL's FIPS-validated cryptographic module. 

import java.security.Security;
import net.glassless.provider.GlaSSLessProvider;

// Register GlaSSLess at highest priority
// ML-KEM and X25519 operations in TLS handshakes will now use
// OpenSSL's FIPS-validated provider instead of SunJCE
Security.insertProviderAt(new GlaSSLessProvider(), 1);

// Verify FIPS mode is active
GlaSSLessProvider provider = (GlaSSLessProvider) Security.getProvider(PROVIDER_NAME);
assert provider.isFIPSMode() : "OpenSSL FIPS provider is not enabled";

// From this point, all TLS 1.3 connections that negotiate X25519MLKEM768
// will use OpenSSL's FIPS-validated ML-KEM implementation
NOTE

When GlaSSLess is at position 1, it will also intercept PBE algorithms used for PKCS12 keystore operations. Load keystores before registering GlaSSLess, or register it at a position that only overrides the algorithms you need.
Performance Considerations

The full hybrid X25519MLKEM768 operation (keygen + encapsulate + decapsulate) was benchmarked across three approaches. Each iteration performs the complete key exchange that would occur during a TLS 1.3 handshake.

ML-KEM-768 in Isolation

When benchmarking ML-KEM-768 alone, JDK 27's pure-Java implementation (SunJCE) benefits from JIT compilation and inlining:

Provider ML-KEM-768 (keygen + encaps + decaps) Relative
SunJCE (Java) ~111 µs/op 1.0x (baseline)
GlaSSLess (OpenSSL via FFM) ~151 µs/op 0.74x

However, ML-KEM is only one component of the hybrid key exchange. The full picture is different.

Full Hybrid X25519MLKEM768
Approach avg (µs/op) p50 p99 vs baseline
SunJCE decomposed (JDK 27 default)
ML-KEM-768 via SunJCE + X25519 via SunEC
353.8 |346.6 |447.4 |1.00x
GlaSSLess decomposed (current behavior with GlaSSLess at pos 1)
ML-KEM-768 via GlaSSLess + X25519 via GlaSSLess
290.1 |287.3 |317.6 | 1.22x faster
GlaSSLess composite (single OpenSSL call)

X25519MLKEM768 as one native EVP_PKEY_encapsulate

 217.7 |216.3 |226.3 | 1.62x faster

Key findings:

  • GlaSSLess decomposed is already 22% faster than the JDK 27 default. While ML-KEM alone is slower via FFM, the X25519 component benefits significantly from OpenSSL's optimized native implementation, and the combined result favors GlaSSLess.

  • The composite approach is 62% faster than the JDK 27 default. OpenSSL performs the X25519 key agreement, ML-KEM encapsulation, and secret concatenation entirely in native code — a single FFM crossing instead of ~20 separate native calls across 3 Arena lifecycles.

  • Tail latency improves dramatically. The p99 drops from 447 µs (SunJCE) to 226 µs (composite) — fewer FFM crossings mean fewer opportunities for GC pauses or scheduling jitter to affect the operation.

Why the Composite Approach Is Faster

JDK 27's HybridProvider decomposes X25519MLKEM768 into separate ML-KEM-768 and X25519 JCA service calls. When GlaSSLess serves both components, each operation involves its own Arena lifecycle, key loading, and cleanup — roughly 20 FFM native calls total. GlaSSLess's composite X25519MLKEM768 KEM delegates to a single OpenSSL EVP_PKEY_encapsulate call (~7 FFM crossings, 1 Arena), letting OpenSSL handle the hybrid construction natively.

However, the composite approach is not currently used by JSSE because HybridProvider is pinned as the provider for hybrid named groups. JSSE resolves KEM.getInstance("X25519MLKEM768", HybridProvider.PROVIDER), bypassing standard JCA provider lookup at the hybrid level. Standard JCA lookup only occurs for the component algorithms (ML-KEM, X25519) inside HybridProvider.

Recommendation

Use GlaSSLess when FIPS-validated cryptography is required — it provides both compliance and a performance improvement over the JDK 27 default for the full hybrid operation. Even in the current decomposed mode, GlaSSLess at position 1 delivers a 22% speedup.

Why Hybrid Key Exchange Matters

Quantum computing poses an increasing threat to widely-deployed public-key encryption algorithms like RSA and ECDH. Even though large-scale quantum computers don't yet exist, adversaries could harvest encrypted data today and decrypt it later ("harvest now, decrypt later" attacks). Hybrid key exchange provides immediate protection by combining classical algorithms with quantum-resistant ones.

JDK 24–26 Workarounds

Until JDK 27, you can achieve post-quantum key exchange through:

  1. Application-layer hybrid KEM — perform a GlaSSLess hybrid KEM exchange over a classical TLS connection (see below)

  2. Reverse proxy — use nginx or haproxy compiled against OpenSSL 3.5+ for TLS termination with X25519MLKEM768

Application-Layer Hybrid Key Exchange

For applications on JDK 24–26 that need quantum-resistant key exchange today, perform the hybrid KEM exchange at the application layer after establishing a classical TLS connection: 

// Step 1: Establish classical TLS 1.3 connection
SSLSocket socket = (SSLSocket) sslContext.getSocketFactory()
    .createSocket("server.example.com", 443);
socket.startHandshake();

// Step 2: Perform hybrid key exchange over the TLS channel
ObjectOutputStream out = new ObjectOutputStream(socket.getOutputStream());
ObjectInputStream in = new ObjectInputStream(socket.getInputStream());

// Server generates and sends hybrid public key
KeyPairGenerator kpg = KeyPairGenerator.getInstance("X25519MLKEM768", PROVIDER_NAME);
KeyPair keyPair = kpg.generateKeyPair();
out.writeObject(keyPair.getPublic().getEncoded());

// Client encapsulates and sends ciphertext back
byte[] serverPubKeyBytes = (byte[]) in.readObject();
// ... reconstruct public key and encapsulate ...
out.writeObject(ciphertext);

// Both sides derive application keys from both:
// - TLS session keys (classical security)
// - Hybrid KEM shared secret (quantum resistance)

Security Considerations

Why Hybrid?

Hybrid key exchange provides defense-in-depth:

  1. If ML-KEM is broken : Classical X25519/X448 still provides security

  2. If X25519/X448 is broken by quantum computers : ML-KEM provides security

  3. Cryptographic agility : Easy to transition as standards evolve

Key Sizes

Algorithm Public Key Private Key Ciphertext Shared Secret
X25519MLKEM768 1216 bytes 2432 bytes 1120 bytes 64 bytes
X448MLKEM1024 1664 bytes 3168 bytes 1632 bytes 64 bytes

Performance

Hybrid KEMs are slower than classical key exchange due to the additional ML-KEM operations. Typical performance (varies by hardware):

Operation X25519MLKEM768 X448MLKEM1024
Key Generation ~0.1ms ~0.15ms
Encapsulation ~0.1ms ~0.15ms
Decapsulation ~0.1ms ~0.15ms

Standards Compliance

The hybrid KEMs follow the construction specified in:

See Also