In cryptography, the RSA problem summarizes the task of performing an RSA private-key operation given only the public key. The RSA algorithm raises a message to an exponent, modulo a composite number N whose factors are not known. As such, the task can be neatly described as finding the eth roots of an arbitrary number, modulo N. For large RSA key sizes (in excess of 1024 bits), no efficient method for solving this problem is known; if an efficient method is ever developed, it would threaten the current or eventual security of RSA-based cryptosystems -- both for public-key encryption and digital signatures.

More specifically, the RSA problem is this: given an RSA public key (N, e), and a ciphertext CPe (mod N), to efficiently compute P. The structure of the RSA public key requires that N is a product of two large primes, 2 < e < N is coprime to φ(N), and 0 ≤ C < N. C is chosen randomly within that range; to specify the problem with complete precision, one must also specify how N and e are generated, which will depend on the precise means of RSA random keypair generation in use.

As of 2010 , the most efficient means known to solve the RSA problem is to first factor the modulus N, which is believed to be impractical if N is sufficiently large (see integer factorization). The RSA key setup routine already turns the public exponent e, with this prime factorization, into the private exponent d, and so the exact same algorithm allows anyone who factors N to obtain the private key. Any C can then be decrypted with the private key.

Just as there are no proofs that integer factorization is computationally difficult, there are also no proofs that the RSA problem is similarly difficult. By the above method, the RSA problem is at least as easy as factoring, but it might well be easier. Indeed, there is strong evidence[1] pointing to this conclusion: that a method to break the RSA method cannot be converted necessarily into a method for factoring large semiprimes. This is perhaps easiest to see by the sheer overkill of the factoring approach: the RSA problem asks us to decrypt one arbitrary ciphertext, whereas the factoring method reveals the private key: thus decrypting all arbitrary ciphertexts, and it also allows one to perform arbitrary RSA private-key encryptions. Along these same lines, finding the decryption exponent d indeed is computationally equivalent to factoring N, even though the RSA problem does not ask for d. An algorithm for this is, for example, given in [2].

In addition to the RSA problem, RSA also has a particular mathematical structure that can potentially be exploited without solving the RSA problem directly. To achieve the full strength of the RSA problem, an RSA-based cryptosystem must also use a padding scheme like OAEP, to protect against such structural problems in RSA.