MemoryErrors and OverflowErrors in SEXTEND

[sam-xif]

Summary of the problem

Hello manticore community,

I am working as part of a research team developing a code analysis tool for Python. One of the issues the tool discovered in manticore's codebase is that core.smtlib.operators.SEXTEND has the potential to throw uncaught MemoryError and OverflowError in its left shift operations. The ValueError is caused when a large value is given for the size_src and/or size_dest argument.

If you are interested in learning more about the tool and how it found this issue, let me know down in the comments, or you can contact me at [email protected]. We are primarily curious about whether you find that this issue is legitimate and worth reporting and fixing. If not, we would be interested in understanding why.

Thank you for your consideration!

-Sam

Manticore version

Latest master (commit hash: 8861005)

Python version

Python 3.8

OS / Environment

Linux (kernel version 5.10.218)

Dependencies

N/A

Step to reproduce the behavior

Call SEXTEND with a large value for the size_src and/or size_dest argument.

Expected behavior

Exception handling, or bound checks to ensure size_src and size_dest are within reasonable bounds.

Actual behavior

Tracebacks:

When size_src is too large:

Traceback (most recent call last):
  ...
  File ".../repos/manticore/manticore/core/smtlib/operators.py", line 139, in SEXTEND
    if x >= (1 << (size_src - 1)):
MemoryError

Traceback (most recent call last):
  ...
  File ".../repos/manticore/manticore/core/smtlib/operators.py", line 139, in SEXTEND
    if x >= (1 << (size_src - 1)):
OverflowError: too many digits in integer

When size_dest is too large:

Traceback (most recent call last):
  ...
  File ".../repos/manticore/manticore/core/smtlib/operators.py", line 141, in SEXTEND
    return x & ((1 << size_dest) - 1)
MemoryError

Traceback (most recent call last):
  ...
  File ".../repos/manticore/manticore/core/smtlib/operators.py", line 141, in SEXTEND
    return x & ((1 << size_dest) - 1)
OverflowError: too many digits in integer