Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple Injection Restriction Bypasses Fixed: Directory traversal, globbing, chaining commands via executables, nested Shell syntax, shell expansions #11

Merged
merged 14 commits into from
Mar 11, 2024

Conversation

LucasFaudman
Copy link
Contributor

api.py changes include:

  • Directory traversal and globbing solved by resolving paths and symlinks in the command before checking for banned executables and sensitive files.
  • Executables than can execute other commands are blocked.
  • Nested shell syntax handled by recursively shlex.split command before parsing and preforming checks.
  • Shell expansions that can be easily calculated are allowed but evaluated before preforming checks so that the expansion cannot be used to produce a banned value.

test_api.py changes include:

  • Tests now check that the raised exception's message startswith the expected message instead of ==. This is the same check since it verifies the expected exception was raised but it accommodates slightly different message endings in my version.

Lucas Faudman added 8 commits February 9, 2024 17:38
…entation by properly handling globbing, chained commands via other binaries, directory traversal and missing of banned binaries accessed in unconventional ways. Added checking for owner/group and uncommon path types that should not be in args. Added typeing
… use precomputed abs_path_strings for .endswith check
…age>) not == since my execptions have slighly different message format but the check is still the exact same
…d shell syntax with recursive_shlex_split. Compatible with Python3.8
@drdavella
Copy link
Member

@LucasFaudman thank you so much for your contribution! I will get to this review ASAP but I appreciate your patience as I have been a bit busy this week.

@LucasFaudman
Copy link
Contributor Author

LucasFaudman commented Feb 22, 2024

Absolutely, this has been a fun problem to work on, and no worries I assumed you guys would be busy with Dev week.

Its actually good you haven't had chance review it yet since I have a few more updates I am going to make after trying out some very weird payloads I used to bypass a difficult WAF in pen-test earlier this week:

I've fixed+tested the first 3 issues so far in my local repo. The 4th issue, I am just failing one test using nested shell arithmetic but I'll hopefully have time to solve that later today and I'll update my PR then.

…_EXPANSION_OPERATORS.

- Handle arithmetic expansion of parameters using -+ and/or nesting and shell variables.
- Track env vars set/modified by expansion and use them in the final command.
- Use expanuser to handle ~ and ~user tilde expansion when resolving paths.
- Correctly hand alphanumeric sequences expansions
- Block process substitution via input redirection.
- More tests, refactoring and comments.

My _shell_expand implementation is still in progress but is neccessary since there are no other viable solutions in Python.
The best I have seen is https://github.com/kojiromike/parameter-expansion
but it cannot be used in a security context because they say:
"All the standard shell expansions are supported, including some level of nested expansion, as long as this is not too complex or ambiguous."
and ambigous cases are exactly what needs to be handled.
@LucasFaudman
Copy link
Contributor Author

Changes include:

  • Complete Shell param/brace/sequence expansion for all ALLOWED_SHELL_EXPANSION_OPERATORS.
  • Handle arithmetic expansion of parameters using -+ and/or nesting and shell variables.
  • Track env vars set/modified by expansion and use them in the final command.
  • Use expanduser to handle ~ and ~user tilde expansion when resolving paths.
  • Correctly hand alphanumeric sequences expansions
  • Block process substitution via input redirection.
  • More tests, refactoring and comments.

_shell_expand implementation is still in progress but is necessary since there are no other viable solutions in Python.
The best I have seen is https://github.com/kojiromike/parameter-expansion
but it cannot be used in a security context because they say:
"All the standard shell expansions are supported, including some level of nested expansion, as long as this is not too complex or ambiguous."
and ambiguous cases are exactly what needs to be handled.

This implementation raises a SecurityException for all the expansion operators that are difficult to accurately evaluate without an actual shell. See list of banned expansions: https://github.com/LucasFaudman/python-security/blob/d22c6fb4fe59d19d92a221d563b87bc3d282dce8/tests/safe_command/test_injection.py#L276

These could be used in combination with slicing or other bash syntax to craft arbitrary strings that could be used for injection. Most people will/should not be using these expansions with user input, especially from Python so I think this will have a minimal impact on usability.

Of course we could expand the library to handle additional syntax securely but the trade-off of additional overhead is the limiting factor.

Copy link
Member

@drdavella drdavella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for this contribution. This is a really incredible amount of work and I'm impressed with the quality overall.

I've asked a few questions for clarification here but I appreciate your patience as we spend a bit more time digging into the implementations here.

BANNED_COMMAND_CHAINING_SEPARATORS = frozenset(("&", ";", "|", "\n"))
BANNED_COMMAND_AND_PROCESS_SUBSTITUTION_OPERATORS = frozenset(
("$(", "`", "<(", ">("))
BANNED_COMMAND_CHAINING_EXECUTABLES = frozenset((
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious but do you have a source for this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list of BANNED_COMMAND_CHAINING_EXECUTABLES is from a combination of lists provided by the SANS organization + a list of common shells/interpreters + a few suggestions from chatGPT + a few other binaries I know can be used for this purpose because I have used them myself. For example byobu-ugraph is my favorite Azure WAF workaround.

"script", "scriptreplay", "scriptlive" were suggested by GPT, and I have not verified them yet so they can probably be removed.

@@ -0,0 +1,530 @@
/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/{FILE}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the source of this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay awesome. It looks like it's a permissive license but we just need to be careful with attribution here.

@@ -0,0 +1,45 @@
{cmd}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the source of this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/fuzzdb-project/fuzzdb/blob/master/attack/os-cmd-execution/command-injection-template.txt

Note I removed the SSI and URL-encoded payloads since they are not what we are testing here.

from security.exceptions import SecurityException

with (Path(__file__).parent / "fuzzdb" / "command-injection-template.txt").open() as f:
FUZZDB_OS_COMMAND_INJECTION_PAYLOADS = [line.replace('\\n','\n').replace("\\'", "'")[:-1] for line in f] # Remove newline
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and below can you just use rstrip to remove the newline?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No because rstrip would remove characters from in the payload as well. I wrote it this way so the payload list can be modified with any payload ending with \n\t etc and only the final newline separating payloads is removed.

tests/safe_command/test_injection.py Outdated Show resolved Hide resolved
Lucas Faudman added 2 commits February 26, 2024 19:16
…d if shell expansion is needed.

- _shell expand is only used when shell=True or the executable is a shell. (list of shells derived from https://www.in-ulm.de/~mascheck/various/shells/) but reading /etc/shells could be used to get a more accurate/concise list on a per-system basis.
- The executable Popen kwarg is now handled correctly and takes precedence over the shell kwarg in determining if shell expansion is needed. See subprocess.py 1593-1596: When an executable is given and shell is True, the shell executable is replaced with the given executable.
- the initial venv state is a copy of the env Popen kwarg to not modify the original Popen kwargs.
- _resolve_executable_path now takes venv into account and uses the PATH env variable to find the executable if env is set in the Popen kwargs since this is how the subprocess module behaves.
- test_popen_kwargs added to test the Popen kwargs and the initial env state. Other tests adjusted accordingly and refactored EXCEPTIONS outside class so they can be used as pytest params.
@LucasFaudman
Copy link
Contributor Author

  • Removed redundant rmtree
  • check() now uses Popen kwargs to determine the initial env state and if shell expansion is needed.
  • _shell_expand is only used when shell=True or the executable is a shell. (list of shells derived from https://www.in-ulm.de/~mascheck/various/shells/) but reading /etc/shells could be used to get a more accurate/concise list on a per-system basis.
  • The executable Popen kwarg is now handled correctly and takes precedence over the shell kwarg in determining if shell expansion is needed. See subprocess.py 1593-1596: When an executable is given and shell is True, the shell executable is replaced with the given executable.
  • the initial venv state is a copy of the env Popen kwarg to not modify the original Popen kwargs.
  • _resolve_executable_path now takes venv into account and uses the PATH env variable to find the executable if env is set in the Popen kwargs since this is how the subprocess module behaves.
  • test_popen_kwargs added to test the Popen kwargs and the initial env state. Other tests adjusted accordingly and refactored EXCEPTIONS outside class so they can be used as pytest params.

Copy link
Member

@drdavella drdavella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LucasFaudman I appreciate your patience as we review this PR. I want to reiterate that I think this is a lot of great work and I intend to accept this PR, although we possibly may need to iterate on a few additional changes.

One notable point is that we need to add attribution for the test vectors you are using from fuzzdb. Their license is permissive but we need to include proper citation.

I've asked @matt- to provide an additional review from a security perspective but from a purely code perspective I am very impressed with the work here. Thanks again!


def run(original_func, command, *args, restrictions=DEFAULT_CHECKS, **kwargs):

def run(original_func: Callable, command: ValidCommand, *args, restrictions: ValidRestrictions = DEFAULT_CHECKS, **kwargs) -> Union[CompletedProcess, None]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we are substantially increasing the number/kind of restrictions here (which is good!) I wonder whether we should make it easier for users to allowlist certain known safe commands via the API? Doesn't necessarily need to be part of this PR but just something to think about.

@@ -0,0 +1,530 @@
/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/{FILE}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay awesome. It looks like it's a permissive license but we just need to be careful with attribution here.

@drdavella drdavella requested a review from matt- February 28, 2024 23:01
@LucasFaudman
Copy link
Contributor Author

@drdavella I definitely agree with you that the API needs to be updated to allow for more flexibility.

I think a way to do this is to make the current run method a private method _check_and_run. Then add new run and call methods which have subprocess.run and subprocess.call as the original_func arg to _check_and_run so that safe_command.run and safe_command.call would have same interface as the corresponding subprocess methods but with optional kwargs for modifying the restrictions and globals (BANNNED_*) used by the check methods.

Using BANNED_EXECUTABLES as an example, the global would be changed to DEFAULT_BANNED_EXECUTABLES and the check modifier kwargs could be in the form:

  • set_banned_executables: BANNED_EXECUTABLES = value
  • add_banned_executables: BANNED_EXECUTABLES = DEFAULT_BANNED_EXECUTABLES + value
  • remove_banned_executables: BANNED_EXECUTABLES = DEFAULT_BANNED_EXECUTABLES - value

In the _check_and_run method all the check modifier kwargs could be evaluated to create the sets to be passed as args to check and its underlying check_* methods.

Additionally this would make creating a SafeCommandRunner very simple.
A SafeCommandRunner could be initialized with the restrictions and check modifiers, and then the run and call methods could be used exactly like subprocess without having to pass the restrictions or check modifiers as args each time.

Lastly, I think we need to check subprocess._mswindows to see if the subprocess will be run on windows and raise a NotImplementedError until this module is updated to better support windows.

@LucasFaudman
Copy link
Contributor Author

@drdavella Would adding a copy of https://github.com/fuzzdb-project/fuzzdb/blob/master/_copyright.txt to the tests/safe_command/fuzzdb folder satisfy the attribution requirements?

@LucasFaudman
Copy link
Contributor Author

Hey @drdavella I just had a chance to implement the subprocess-like API I described in my previous comment. Check it out on this branch and let me know if you think I should merge it into this PR.

I think these changes will significantly improve the developer experience of using this module, but I imagine it will probably require editing the corresponding CodeMod, this should be easy though because subprocess just needs to be swapped for safe_command and optionally add the requirements or other config kwargs.

Commit changes include:

  • API is now subprocess-like. Popen, run, call, check_call, getstatusoutput, getoutput, check_output all callable directly from module or the SafeCommandRunner class.
  • restrictions and all config values can be modified with kwargs.
  • SafeCommandRunner class saves the current restrictions and config values, and can be used to run multiple commands with the same restrictions and config values.
  • Warning raised when imported on Windows.
  • Added fuzzDB _copyright file to fuzzdb tests dir

Only change to underlying implentation is that check now takes force_shell arg which is used to force shell expansion even when shell=True is not in the Popen kwargs. This is needed to wrap subprocess.getstatusoutput and subprocess.getoutput which both always run with shell=True but do not accept Popen kwargs

@drdavella
Copy link
Member

@LucasFaudman that's awesome!

Let's consider that as a separate change in a separate PR. The current PR is large enough as it is and I want to make sure we get it merged before moving on. You can feel free to open a new PR based on this one or just hold off until this is merged if you want.

@drdavella
Copy link
Member

@drdavella Would adding a copy of https://github.com/fuzzdb-project/fuzzdb/blob/master/_copyright.txt to the tests/safe_command/fuzzdb folder satisfy the attribution requirements?

Yes that should be sufficient. It might be a good idea to add a line to the top of the file that says: "These test vectors were taken from the fuzzdb project under the terms of the following license"

@drdavella
Copy link
Member

@LucasFaudman thanks for your patience with this review cadence. I'd like to get this one across the finish line and I think it's nearly ready to merge. No rush at all but please let me know whether you'd like me to go ahead and make the license updates.

@LucasFaudman
Copy link
Contributor Author

Hey @drdavella sorry for the delay, been very busy since the end of last week. I just updated my main branch with the FuzzDB _copyright.txt file and added: "These test vectors were taken from the fuzzdb project under the terms of the license in _copyright.txt" to the top of each test file.

I'll submit a second PR with the subprocess-like API branch once all the internals have been merged.

@matt- Since you are reviewing for security please let me know if there is anything you think I missed or that should be removed from the BANNED_COMMAND_CHAINING_EXECUTABLES.

@LucasFaudman
Copy link
Contributor Author

In my other branch I use the following snippet to warn users when importing this module on Windows. Do you think I should add this to this PR before its merged?

if subprocess._mswindows: # type: ignore
    from warnings import warn
    warn(RuntimeWarning("SafeCommand not yet fully supported on Windows. Only use if you know what you are doing."))

Copy link

sonarqubecloud bot commented Mar 5, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

@drdavella
Copy link
Member

Do you think I should add this to this PR before its merged?

Good question and good idea but let's save that for the next PR unless you feel strongly about it. I might choose slightly different wording:

"SafeCommand not yet fully supported on Windows. Use with caution."

@drdavella
Copy link
Member

sorry for the delay, been very busy since the end of last week.

Also: there is no need at all to apologize. I realize I have dragged this review out for quite awhile and I appreciate your patience and understanding. Your contributions are very much appreciated.

@LucasFaudman
Copy link
Contributor Author

Good question and good idea but let's save that for the next PR unless you feel strongly about it. I might choose slightly different wording:

I agree the warning can wait until the next PR since subprocess._mswindows is not in subprocess.__all__ and cannot be imported with a from statment, and the current implementation does not warrant fully importing subprocess. The subprocess-like API version does however fully import subprocess to wrap all its public methods so this warning can be added then without waste and I'll use your suggested message.

@LucasFaudman
Copy link
Contributor Author

Also: there is no need at all to apologize. I realize I have dragged this review out for quite awhile and I appreciate your patience and understanding. Your contributions are very much appreciated.

No worries at all on the long review, I also understand that you guys are busy and that I've definetly contributed to the delay by pushing additional commits before you've had a chance to review the previous ones.

@LucasFaudman
Copy link
Contributor Author

Since you guys seemed to be happy with the current check logic, I optimized the implementation without modifying the underlying check logic or the API.

I think these changes are important and you should consider this branch before merging. Please let me know what you think and I'll merge it with my main so the PR is updated.
https://github.com/LucasFaudman/python-security/tree/iterpaths

Changes in this branch:

  • Optimized space complexity of command parsing and path resolution by converting _resolve_path_in_parsed_command to a generator of Path objects that takes a single cmd_part and yields all the resolved paths for it.

  • Split check_* methods based on if they are checking the expanded_command, a single cmd_part, a single path_string, or single Path.

  • First the command is expanded and expanded_command is checked. Then if the expanded_command passes checks it is used to create a generator of cmd_parts.

  • Checks are then preformed on each part and its paths as they are generated, so all the parts/paths do not need to be stored in memory at once.

  • Reduces the space complexity from O(n) to O(1) where n is the number of cmd_parts or paths.

  • More loops over the BANNED_* are needed which does slightly increase the time complexity, but this trade off is worth it since:

    • This can actually reduce the overall check execution time by raising an exception sooner before fully resolving all the parts/paths.
    • The number of BANNED_* is relatively small and constant.
    • The number of cmd_parts and more importantly paths could be very large and unpredictable.
  • _recursive_resolve_symlinks correctly handles symlinks to symlinks, and/or symlinks to directories which may contain symlinks.

  • _path_is_executable now will not return True for dirs since dirs can have the x bit set but they are not executables.

  • max_resolved_paths, and rglob_dirs kwargs can be used to control this behavior.

  • SecurityException raised if the number of resolved paths exceeds max_resolved_paths.

  • updated test_resolve_paths_in_parsed_command and added test_max_resolved_paths to test this behavior.

  • Underlying check implmentation logic is unchanged.

@drdavella
Copy link
Member

@LucasFaudman I'll take a look: to make it a little easier could you create another PR from that branch and base it against this one?

Copy link
Member

@drdavella drdavella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LucasFaudman I am going to go ahead and merge these changes. I will not cut a new release until we include the work on your new branch but it will be easiest to continue this work with a basea of main.

Thanks for a lot of awesome work and congratulations on your first contribution to this repository!

@drdavella drdavella merged commit 780b236 into pixee:main Mar 11, 2024
1 check passed
@LucasFaudman
Copy link
Contributor Author

@drdavella Awesome, thank you! This evening after I finish my other work I'll submit another PR with the Paths generator mem optimized version.

In the future I also plan to expand the COMMON_EXPLOIT_EXECUTABLES and COMMAND_CHAINING_EXECUTABLES using:

  • GTFOBins "GTFOBins is a curated list of Unix binaries that can be used to bypass local security restrictions in misconfigured systems." and later LOLBAS when improving support for Windows down the line.
  • apropos-exec-sorted.txt A list of binaries I produced by running apropos exec on a variety of Linux distros and MacOS.

And expand the SENSITIVE_FILE_PATHS to include other common config files that contain secrets like ~/.aws/credentials, wp-config.php, etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants