ssh: reject RSA keys with excessively large moduli

Previously, the RSA key parser accepted keys with arbitrary modulus
sizes. Processing keys with extremely large moduli (e.g., > 8192 bits)
can consume excessive CPU resources during verification, potentially
leading to a Denial of Service (DoS).

This change introduces a limit of 8192 bits for the RSA modulus in
parseRSA, rejecting keys that exceed this size in line with the limit
enforced by crypto/tls.

This issue was found during a security audit by NCC Group Cryptography
Services, sponsored by Teleport.

Fixes golang/go#79565
Fixes CVE-2026-39829

Change-Id: Ibdddad1859a4d9db5c9f052d06c82f29bfc2e5e5
Reviewed-on: https://go-review.googlesource.com/c/crypto/+/781641
Reviewed-by: Neal Patel <nealpatel@google.com>
LUCI-TryBot-Result: golang-scoped@luci-project-accounts.iam.gserviceaccount.com <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>

ssh/keys.go

@@ -469,6 +469,12 @@ func parseRSA(in []byte) (out PublicKey, rest []byte, err error) {
 		return nil, nil, err
 	}
 
+	// 8192 bits is also the maximum RSA key size accepted by crypto/tls for
+	// signature verification:
+	// https://github.com/golang/go/blob/69801b25/src/crypto/tls/handshake_client.go#L1096
+	if w.N.BitLen() > 8192 {
+		return nil, nil, errors.New("ssh: rsa modulus too large")
+	}
 	if w.E.BitLen() > 24 {
 		return nil, nil, errors.New("ssh: exponent too large")
 	}