-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple Injection Restriction Bypasses Fixed: Directory traversal, globbing, chaining commands via executables, nested Shell syntax, shell expansions #11
Conversation
…entation by properly handling globbing, chained commands via other binaries, directory traversal and missing of banned binaries accessed in unconventional ways. Added checking for owner/group and uncommon path types that should not be in args. Added typeing
… use precomputed abs_path_strings for .endswith check
…age>) not == since my execptions have slighly different message format but the check is still the exact same
…t and FuzzDB tests
…d shell syntax with recursive_shlex_split. Compatible with Python3.8
…t and FuzzDB tests
@LucasFaudman thank you so much for your contribution! I will get to this review ASAP but I appreciate your patience as I have been a bit busy this week. |
Absolutely, this has been a fun problem to work on, and no worries I assumed you guys would be busy with Dev week. Its actually good you haven't had chance review it yet since I have a few more updates I am going to make after trying out some very weird payloads I used to bypass a difficult WAF in pen-test earlier this week:
I've fixed+tested the first 3 issues so far in my local repo. The 4th issue, I am just failing one test using nested shell arithmetic but I'll hopefully have time to solve that later today and I'll update my PR then. |
…_EXPANSION_OPERATORS. - Handle arithmetic expansion of parameters using -+ and/or nesting and shell variables. - Track env vars set/modified by expansion and use them in the final command. - Use expanuser to handle ~ and ~user tilde expansion when resolving paths. - Correctly hand alphanumeric sequences expansions - Block process substitution via input redirection. - More tests, refactoring and comments. My _shell_expand implementation is still in progress but is neccessary since there are no other viable solutions in Python. The best I have seen is https://github.com/kojiromike/parameter-expansion but it cannot be used in a security context because they say: "All the standard shell expansions are supported, including some level of nested expansion, as long as this is not too complex or ambiguous." and ambigous cases are exactly what needs to be handled.
Changes include:
_shell_expand implementation is still in progress but is necessary since there are no other viable solutions in Python. This implementation raises a SecurityException for all the expansion operators that are difficult to accurately evaluate without an actual shell. See list of banned expansions: https://github.com/LucasFaudman/python-security/blob/d22c6fb4fe59d19d92a221d563b87bc3d282dce8/tests/safe_command/test_injection.py#L276 These could be used in combination with slicing or other bash syntax to craft arbitrary strings that could be used for injection. Most people will/should not be using these expansions with user input, especially from Python so I think this will have a minimal impact on usability. Of course we could expand the library to handle additional syntax securely but the trade-off of additional overhead is the limiting factor. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First of all, thanks for this contribution. This is a really incredible amount of work and I'm impressed with the quality overall.
I've asked a few questions for clarification here but I appreciate your patience as we spend a bit more time digging into the implementations here.
BANNED_COMMAND_CHAINING_SEPARATORS = frozenset(("&", ";", "|", "\n")) | ||
BANNED_COMMAND_AND_PROCESS_SUBSTITUTION_OPERATORS = frozenset( | ||
("$(", "`", "<(", ">(")) | ||
BANNED_COMMAND_CHAINING_EXECUTABLES = frozenset(( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just curious but do you have a source for this list?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The list of BANNED_COMMAND_CHAINING_EXECUTABLES
is from a combination of lists provided by the SANS organization + a list of common shells/interpreters + a few suggestions from chatGPT + a few other binaries I know can be used for this purpose because I have used them myself. For example byobu-ugraph
is my favorite Azure WAF workaround.
"script", "scriptreplay", "scriptlive"
were suggested by GPT, and I have not verified them yet so they can probably be removed.
@@ -0,0 +1,530 @@ | |||
/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/{FILE} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the source of this list?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay awesome. It looks like it's a permissive license but we just need to be careful with attribution here.
@@ -0,0 +1,45 @@ | |||
{cmd} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the source of this list?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note I removed the SSI and URL-encoded payloads since they are not what we are testing here.
tests/safe_command/test_injection.py
Outdated
from security.exceptions import SecurityException | ||
|
||
with (Path(__file__).parent / "fuzzdb" / "command-injection-template.txt").open() as f: | ||
FUZZDB_OS_COMMAND_INJECTION_PAYLOADS = [line.replace('\\n','\n').replace("\\'", "'")[:-1] for line in f] # Remove newline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here and below can you just use rstrip
to remove the newline?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No because rstrip
would remove characters from in the payload as well. I wrote it this way so the payload list can be modified with any payload ending with \n\t
etc and only the final newline separating payloads is removed.
…D_CHAINING_EXECUTABLES
…d if shell expansion is needed. - _shell expand is only used when shell=True or the executable is a shell. (list of shells derived from https://www.in-ulm.de/~mascheck/various/shells/) but reading /etc/shells could be used to get a more accurate/concise list on a per-system basis. - The executable Popen kwarg is now handled correctly and takes precedence over the shell kwarg in determining if shell expansion is needed. See subprocess.py 1593-1596: When an executable is given and shell is True, the shell executable is replaced with the given executable. - the initial venv state is a copy of the env Popen kwarg to not modify the original Popen kwargs. - _resolve_executable_path now takes venv into account and uses the PATH env variable to find the executable if env is set in the Popen kwargs since this is how the subprocess module behaves. - test_popen_kwargs added to test the Popen kwargs and the initial env state. Other tests adjusted accordingly and refactored EXCEPTIONS outside class so they can be used as pytest params.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LucasFaudman I appreciate your patience as we review this PR. I want to reiterate that I think this is a lot of great work and I intend to accept this PR, although we possibly may need to iterate on a few additional changes.
One notable point is that we need to add attribution for the test vectors you are using from fuzzdb
. Their license is permissive but we need to include proper citation.
I've asked @matt- to provide an additional review from a security perspective but from a purely code perspective I am very impressed with the work here. Thanks again!
|
||
def run(original_func, command, *args, restrictions=DEFAULT_CHECKS, **kwargs): | ||
|
||
def run(original_func: Callable, command: ValidCommand, *args, restrictions: ValidRestrictions = DEFAULT_CHECKS, **kwargs) -> Union[CompletedProcess, None]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that we are substantially increasing the number/kind of restrictions here (which is good!) I wonder whether we should make it easier for users to allowlist certain known safe commands via the API? Doesn't necessarily need to be part of this PR but just something to think about.
@@ -0,0 +1,530 @@ | |||
/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/0x2e0x2e/{FILE} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay awesome. It looks like it's a permissive license but we just need to be careful with attribution here.
@drdavella I definitely agree with you that the API needs to be updated to allow for more flexibility. I think a way to do this is to make the current Using BANNED_EXECUTABLES as an example, the global would be changed to DEFAULT_BANNED_EXECUTABLES and the check modifier kwargs could be in the form:
In the Additionally this would make creating a Lastly, I think we need to check |
@drdavella Would adding a copy of https://github.com/fuzzdb-project/fuzzdb/blob/master/_copyright.txt to the |
Hey @drdavella I just had a chance to implement the subprocess-like API I described in my previous comment. Check it out on this branch and let me know if you think I should merge it into this PR. I think these changes will significantly improve the developer experience of using this module, but I imagine it will probably require editing the corresponding CodeMod, this should be easy though because Commit changes include:
Only change to underlying implentation is that |
@LucasFaudman that's awesome! Let's consider that as a separate change in a separate PR. The current PR is large enough as it is and I want to make sure we get it merged before moving on. You can feel free to open a new PR based on this one or just hold off until this is merged if you want. |
Yes that should be sufficient. It might be a good idea to add a line to the top of the file that says: "These test vectors were taken from the |
@LucasFaudman thanks for your patience with this review cadence. I'd like to get this one across the finish line and I think it's nearly ready to merge. No rush at all but please let me know whether you'd like me to go ahead and make the license updates. |
Hey @drdavella sorry for the delay, been very busy since the end of last week. I just updated my main branch with the FuzzDB _copyright.txt file and added: "These test vectors were taken from the fuzzdb project under the terms of the license in _copyright.txt" to the top of each test file. I'll submit a second PR with the subprocess-like API branch once all the internals have been merged. @matt- Since you are reviewing for security please let me know if there is anything you think I missed or that should be removed from the |
In my other branch I use the following snippet to warn users when importing this module on Windows. Do you think I should add this to this PR before its merged? if subprocess._mswindows: # type: ignore
from warnings import warn
warn(RuntimeWarning("SafeCommand not yet fully supported on Windows. Only use if you know what you are doing.")) |
|
Good question and good idea but let's save that for the next PR unless you feel strongly about it. I might choose slightly different wording:
|
Also: there is no need at all to apologize. I realize I have dragged this review out for quite awhile and I appreciate your patience and understanding. Your contributions are very much appreciated. |
I agree the warning can wait until the next PR since |
No worries at all on the long review, I also understand that you guys are busy and that I've definetly contributed to the delay by pushing additional commits before you've had a chance to review the previous ones. |
Since you guys seemed to be happy with the current I think these changes are important and you should consider this branch before merging. Please let me know what you think and I'll merge it with my main so the PR is updated. Changes in this branch:
|
@LucasFaudman I'll take a look: to make it a little easier could you create another PR from that branch and base it against this one? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LucasFaudman I am going to go ahead and merge these changes. I will not cut a new release until we include the work on your new branch but it will be easiest to continue this work with a basea of main
.
Thanks for a lot of awesome work and congratulations on your first contribution to this repository!
@drdavella Awesome, thank you! This evening after I finish my other work I'll submit another PR with the Paths generator mem optimized version. In the future I also plan to expand the
And expand the |
api.py changes include:
shlex.split
command before parsing and preforming checks.test_api.py changes include: