13600950642

Committed 01 Mar 2025 04:10AM UTC coverage: 74.211% (+3.2%) from 71.055%

Build # 13600950642

Build Type

Pull #2546

github

Committed by

web-flow

Commit Message

Merge 77df40314 into 60cff2437

Pull Request Pull Request #2546: ssh: Allow passing `disabled_algorithms` keyword argument from `ssh` to paramiko

Run Details

3812 of 6380 branches covered (59.75%)

0 of 1 new or added line in 1 file covered. (0.0%)

1243 existing lines in 37 files now uncovered.

13352 of 17992 relevant lines covered (74.21%)

0.74 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

72.73

/pwnlib/util/sh_string.py

# -*- coding: utf-8 -*-
r"""
Routines here are for getting any NULL-terminated sequence of bytes evaluated
intact by any shell.  This includes all variants of quotes, whitespace, and
non-printable characters.

Supported Shells
----------------

The following shells have been evaluated:

- Ubuntu (dash/sh)
- MacOS (GNU Bash)
- Zsh
- FreeBSD (sh)
- OpenBSD (sh)
- NetBSD (sh)

Debian Almquist shell (Dash)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Ubuntu 14.04 and 16.04 use the Dash shell, and /bin/sh is actually just a
symlink to /bin/dash.  The feature set supported when invoked as "sh" instead
of "dash" is different, and we focus exclusively on the "/bin/sh" implementation.

From the `Ubuntu Man Pages`_, every character except for single-quote
can be wrapped in single-quotes, and a backslash can be used to escape unquoted
single-quotes.

::

    Quoting
      Quoting is used to remove the special meaning of certain characters or
      words to the shell, such as operators, whitespace, or keywords.  There
      are three types of quoting: matched single quotes, matched double quotes,
      and backslash.

    Backslash
      A backslash preserves the literal meaning of the following character,
      with the exception of ⟨newline⟩.  A backslash preceding a ⟨newline⟩ is
      treated as a line continuation.

    Single Quotes
      Enclosing characters in single quotes preserves the literal meaning of
      all the characters (except single quotes, making it impossible to put
      single-quotes in a single-quoted string).

    Double Quotes
      Enclosing characters within double quotes preserves the literal meaning
      of all characters except dollarsign ($), backquote (`), and backslash
      (\).  The backslash inside double quotes is historically weird, and
      serves to quote only the following characters:
            $ ` " \ <newline>.
      Otherwise it remains literal.

GNU Bash
~~~~~~~~

The Bash shell is default on many systems, though it is not generally the default
system-wide shell (i.e., the `system` syscall does not generally invoke it).

That said, its prevalence suggests that it also be addressed.

From the `GNU Bash Manual`_, every character except for single-quote
can be wrapped in single-quotes, and a backslash can be used to escape unquoted
single-quotes.

::

    3.1.2.1 Escape Character

    A non-quoted backslash ‘\’ is the Bash escape character. It preserves the
    literal value of the next character that follows, with the exception of
    newline. If a ``\\newline`` pair appears, and the backslash itself is not
    quoted, the ``\\newline`` is treated as a line continuation (that is, it
    is removed from the input stream and effectively ignored).

    3.1.2.2 Single Quotes

    Enclosing characters in single quotes (‘'’) preserves the literal value of
    each character within the quotes. A single quote may not occur between single
    uotes, even when preceded by a backslash.

    3.1.2.3 Double Quotes

    Enclosing characters in double quotes (‘"’) preserves the literal value of a
    ll characters within the quotes, with the exception of ‘$’, ‘`’, ‘\’, and,
    when history expansion is enabled, ‘!’. The characters ‘$’ and ‘`’ retain their
    pecial meaning within double quotes (see Shell Expansions). The backslash retains
    its special meaning only when followed by one of the following characters:
    ‘$’, ‘`’, ‘"’, ‘\’, or newline. Within double quotes, backslashes that are
    followed by one of these characters are removed. Backslashes preceding
    characters without a special meaning are left unmodified. A double quote may
    be quoted within double quotes by preceding it with a backslash. If enabled,
    history expansion will be performed unless an ‘!’ appearing in double quotes
    is escaped using a backslash. The backslash preceding the ‘!’ is not removed.

    The special parameters ‘*’ and ‘@’ have special meaning when in double quotes
    see Shell Parameter Expansion).

Z Shell
~~~~~~~

The Z shell is also a relatively common user shell, even though it's not generally
the default system-wide shell.

From the `Z Shell Manual`_, every character except for single-quote
can be wrapped in single-quotes, and a backslash can be used to escape unquoted
single-quotes.

::

    A character may be quoted (that is, made to stand for itself) by preceding
    it with a ‘\’. ‘\’ followed by a newline is ignored.

    A string enclosed between ‘$'’ and ‘'’ is processed the same way as the
    string arguments of the print builtin, and the resulting string is considered
    o be entirely quoted. A literal ‘'’ character can be included in the string
    by using the ‘\\'’ escape.

    All characters enclosed between a pair of single quotes ('') that is not
    preceded by a ‘$’ are quoted. A single quote cannot appear within single
    quotes unless the option RC_QUOTES is set, in which case a pair of single
    quotes are turned into a single quote. For example,

    print ''''
    outputs nothing apart from a newline if RC_QUOTES is not set, but one single
    quote if it is set.

    Inside double quotes (""), parameter and command substitution occur, and
    ‘\’ quotes the characters ‘\’, ‘`’, ‘"’, and ‘$’.

FreeBSD Shell
~~~~~~~~~~~~~

Compatibility with the FreeBSD shell is included for completeness.

From the `FreeBSD man pages`_, every character except for single-quote
can be wrapped in single-quotes, and a backslash can be used to escape unquoted
single-quotes.

::

     Quoting is used to remove the special meaning of certain characters or
     words to the shell, such as operators, whitespace, keywords, or alias
     names.

     There are four types of quoting: matched single quotes, dollar-single
     quotes, matched double quotes, and backslash.

     Single Quotes
         Enclosing characters in single quotes preserves the literal mean-
         ing of all the characters (except single quotes, making it impos-
         sible to put single-quotes in a single-quoted string).

     Dollar-Single Quotes
         Enclosing characters between $' and ' preserves the literal mean-
         ing of all characters except backslashes and single quotes.  A
         backslash introduces a C-style escape sequence:

         ...

     Double Quotes
         Enclosing characters within double quotes preserves the literal
         meaning of all characters except dollar sign (`$'), backquote
         (``'), and backslash (`\\').  The backslash inside double quotes
         is historically weird.  It remains literal unless it precedes the
         following characters, which it serves to quote:

           $     `     "     \     \\n

     Backslash
         A backslash preserves the literal meaning of the following char-
         acter, with the exception of the newline character (`\\n').  A
         backslash preceding a newline is treated as a line continuation.

OpenBSD Shell
~~~~~~~~~~~~~

From the `OpenBSD Man Pages`_, every character except for single-quote
can be wrapped in single-quotes, and a backslash can be used to escape unquoted
single-quotes.

::

    A backslash (\) can be used to quote any character except a newline.
    If a newline follows a backslash the shell removes them both, effectively
    making the following line part of the current one.

    A group of characters can be enclosed within single quotes (') to quote
    every character within the quotes.

    A group of characters can be enclosed within double quotes (") to quote
    every character within the quotes except a backquote (`) or a dollar
    sign ($), both of which retain their special meaning. A backslash (\)
    within double quotes retains its special meaning, but only when followed
    by a backquote, dollar sign, double quote, or another backslash.
    An at sign (@) within double quotes has a special meaning
    (see SPECIAL PARAMETERS, below).

NetBSD Shell
~~~~~~~~~~~~

The NetBSD shell's documentation is identical to the Dash documentation.

Android Shells
~~~~~~~~~~~~~~

Android has gone through some number of shells.

- Mksh, a Korn shell, was used with Toolbox releases (5.0 and prior)
- Toybox, also derived from the Almquist Shell (6.0 and newer)

Notably, the Toolbox implementation is not POSIX compliant
as it lacks a "printf" builtin (e.g. Android 5.0 emulator images).

Toybox Shell
~~~~~~~~~~~~

Android 6.0 (and possibly other versions) use a shell based on ``toybox``.

While it does not include a ``printf`` builtin, ``toybox`` itself includes
a POSIX-compliant ``printf`` binary.

The Ash shells should be feature-compatible with ``dash``.

BusyBox Shell
~~~~~~~~~~~~~

`BusyBox's Wikipedia page`_ claims to use an ``ash``-compliant shell,
and should therefore be compatible with ``dash``.


.. _Ubuntu Man Pages: https://manpages.ubuntu.com/manpages/trusty/man1/dash.1.html
.. _GNU Bash Manual: https://www.gnu.org/software/bash/manual/bash.html#Quoting
.. _Z Shell Manual: https://zsh.sourceforge.io/Doc/Release/Shell-Grammar.html#Quoting
.. _FreeBSD man pages: https://www.freebsd.org/cgi/man.cgi?query=sh
.. _OpenBSD Man Pages: https://man.openbsd.org/sh#SHELL_GRAMMAR
.. _BusyBox's Wikipedia page: https://en.wikipedia.org/wiki/BusyBox#Features
"""
from __future__ import absolute_import
from __future__ import division

import string
import subprocess

from pwnlib.context import context
from pwnlib.log import getLogger
from pwnlib.tubes.process import process
from pwnlib.util import fiddling
from pwnlib.util.misc import which, normalize_argv_env

log = getLogger(__name__)

def test_all():
    test('a') ##
    test('ab') ##
    test('a b') ##
    test(r"a\'b") ##
    everything_1 = bytes(range(1,256))
    for s in everything_1:
        test(s)
        test(s*4)
        test(s * 2 + b'X')
        test(b'X' + s * 2)
        test((s*2 + b'X') * 2)
        test(s + b'X' + s)
        test(s*2 + b'X' + s*2)
        test(b'X' + s*2 + b'X')
    test(everything_1)
    test(everything_1 * 2)
    test(everything_1 * 4)
    everything_2 = b''.join(bytes([c,c]) for c in range(1,256)) ##
    test(everything_2)

    test(randoms(1000, everything_1))


def test(original):
    r"""Tests the output provided by a shell interpreting a string

    .. doctest::
        :options: +POSIX

        >>> test(b'foobar')
        >>> test(b'foo bar')
        >>> test(b'foo bar\n')
        >>> test(b"foo'bar")
        >>> test(b"foo\\\\bar")
        >>> test(b"foo\\\\'bar")
        >>> test(b"foo\\x01'bar")
        >>> test(b'\n')
        >>> test(b'\xff')
        >>> test(os.urandom(16 * 1024).replace(b'\x00', b''))
    """
    input = sh_string(original)

    if isinstance(input, str):
        input = input.encode()

    cmdstr = b'/bin/echo %s' % input

    SUPPORTED_SHELLS = [
        ['ash', '-c', cmdstr],
        ['bash', '-c', cmdstr],
        ['bash', '-o', 'posix', '-c', cmdstr],
        ['ksh', '-c', cmdstr],
        ['busybox', 'ash', '-c', cmdstr],
        ['busybox', 'sh', '-c', cmdstr],
        ['zsh', '-c', cmdstr],
        ['posh', '-c', cmdstr],
        ['dash', '-c', cmdstr],
        ['mksh', '-c', cmdstr],
        ['sh', '-c', cmdstr],
        # ['adb', 'exec-out', cmdstr]
    ]

    for shell in SUPPORTED_SHELLS:
        binary = shell[0]

        if not which(binary):
            log.warn_once('Shell %r is not available' % binary)
            continue

        progress = log.progress('%s: %r' % (binary, original))

        with context.quiet:
            with process(shell) as p:
                data = p.recvall(timeout=2)
                p.kill()

        # Remove exactly one trailing newline added by echo
        # We cannot assume "echo -n" exists.
        data = data[:-1]

        if data != original:
            for i,(a,b) in enumerate(zip(data, original)):
                if a == b:
                    continue
                log.error(('Shell %r failed\n' +
                          'Expect %r\n' +
                          'Sent   %r\n' +
                          'Output %r\n' +
                          'Mismatch @ %i: %r vs %r') \
                        % (binary, original, input, data, i, a, b))

        progress.success()



SINGLE_QUOTE = "'" ##
ESCAPED_SINGLE_QUOTE = r"\'" ##

ESCAPED = {
    # The single quote itself must be escaped, outside of single quotes.
    "'": "\\'", ##

    # Slashes must themselves be escaped
    #
    # Additionally, some shells coalesce any number N>1 of '\' into
    # a single backslash literal.
    # '\\': '"\\\\\\\\"'
}

def sh_string(s):
    r"""Outputs a string in a format that will be understood by /bin/sh.

    If the string does not contain any bad characters, it will simply be
    returned, possibly with quotes. If it contains bad characters, it will
    be escaped in a way which is compatible with most known systems.

    Warning:
        This does not play along well with the shell's built-in "echo".
        It works exactly as expected to set environment variables and
        arguments, **unless** it's the shell-builtin echo.

    Argument:
        s(str): String to escape.

    Examples:

        >>> sh_string('foobar')
        'foobar'
        >>> sh_string('foo bar')
        "'foo bar'"
        >>> sh_string("foo'bar")
        "'foo'\\''bar'"
        >>> sh_string("foo\\\\bar")
        "'foo\\\\bar'"
        >>> sh_string("foo\\\\'bar")
        "'foo\\\\'\\''bar'"
        >>> sh_string("foo\\x01'bar")
        "'foo\\x01'\\''bar'"
    """
    orig_s = s
    if isinstance(s, (bytes, bytearray)):
        s = s.decode('latin1')
    if '\x00' in s: ##
        log.error("sh_string(): Cannot create a null-byte")

    if not s:
        quoted_string = "''" ##
        if isinstance(orig_s, (bytes, bytearray)):
            quoted_string = quoted_string.encode('latin1')
        return quoted_string

    chars = set(s)
    very_good = set(string.ascii_letters + string.digits + "_+.,/-") ##

    # Alphanumeric can always just be used verbatim.
    if chars <= very_good:
        return orig_s

    # If there are no single-quotes, the entire thing can be single-quoted
    if not (chars & set(ESCAPED)):
        quoted_string = "'%s'" % s ##
        if isinstance(orig_s, (bytes, bytearray)):
            quoted_string = quoted_string.encode('latin1')
        return quoted_string

    # If there are single-quotes, we can single-quote around them, and simply
    # escape the single-quotes.
    quoted_string = '' ##
    quoted = False
    for char in s: ##
        if char not in ESCAPED:
            if not quoted:
                quoted_string += SINGLE_QUOTE
                quoted = True
            quoted_string += char ##
        else:
            if quoted:
                quoted = False
                quoted_string += SINGLE_QUOTE
            quoted_string += ESCAPED[char]

    if quoted:
        quoted_string += SINGLE_QUOTE

    if isinstance(orig_s, (bytes, bytearray)):
        quoted_string = quoted_string.encode('latin1')
    return quoted_string

def sh_prepare(variables, export = False):
    r"""Outputs a posix compliant shell command that will put the data specified
    by the dictionary into the environment.

    It is assumed that the keys in the dictionary are valid variable names that
    does not need any escaping.

    Arguments:
      variables(dict): The variables to set.
      export(bool): Should the variables be exported or only stored in the shell environment?
      output(str): A valid posix shell command that will set the given variables.

    It is assumed that `var` is a valid name for a variable in the shell.

    Examples:

        >>> sh_prepare({'X': 'foobar'})
        b'X=foobar'
        >>> r = sh_prepare({'X': 'foobar', 'Y': 'cookies'})
        >>> r == b'X=foobar;Y=cookies' or r == b'Y=cookies;X=foobar' or r
        True
        >>> sh_prepare({'X': 'foo bar'})
        b"X='foo bar'"
        >>> sh_prepare({'X': "foo'bar"})
        b"X='foo'\\''bar'"
        >>> sh_prepare({'X': "foo\\\\bar"})
        b"X='foo\\\\bar'"
        >>> sh_prepare({'X': "foo\\\\'bar"})
        b"X='foo\\\\'\\''bar'"
        >>> sh_prepare({'X': "foo\\x01'bar"})
        b"X='foo\\x01'\\''bar'"
        >>> sh_prepare({'X': "foo\\x01'bar"}, export = True)
        b"export X='foo\\x01'\\''bar'"
        >>> sh_prepare({'X': "foo\\x01'bar\\n"})
        b"X='foo\\x01'\\''bar\\n'"
        >>> sh_prepare({'X': "foo\\x01'bar\\n"})
        b"X='foo\\x01'\\''bar\\n'"
        >>> sh_prepare({'X': "foo\\x01'bar\\n"}, export = True)
        b"export X='foo\\x01'\\''bar\\n'"
    """

    out = []
    export = b'export ' if export else b''

    _, variables = normalize_argv_env([], variables, log)

    for k, v in variables:
        out.append(b'%s%s=%s' % (export, k, sh_string(v)))

    return b';'.join(out)

def sh_command_with(f, *args):
    r"""sh_command_with(f, arg0, ..., argN) -> command

    Returns a command create by evaluating `f(new_arg0, ..., new_argN)`
    whenever `f` is a function and `f % (new_arg0, ..., new_argN)` otherwise.

    If the arguments are purely alphanumeric, then they are simply passed to
    function. If they are simple to escape, they will be escaped and passed to
    the function.

    If the arguments contain trailing newlines, then it is hard to use them
    directly because of a limitation in the posix shell. In this case the
    output from `f` is prepended with a bit of code to create the variables.

    Examples:

        >>> sh_command_with(lambda: "echo hello")
        'echo hello'
        >>> sh_command_with(lambda x: "echo " + x, "hello")
        'echo hello'
        >>> sh_command_with(lambda x: "/bin/echo " + x, "\\x01")
        "/bin/echo '\\x01'"
        >>> sh_command_with(lambda x: "/bin/echo " + x, "\\x01\\n")
        "/bin/echo '\\x01\\n'"
        >>> sh_command_with("/bin/echo %s", "\\x01\\n")
        "/bin/echo '\\x01\\n'"
    """

    args = list(args)
    out = []

    for n in range(len(args)):
        args[n] = sh_string(args[n])
    if hasattr(f, '__call__'):
        out.append(f(*args))
    else:
        out.append(f % tuple(args))
    return ';'.join(out)

1	# -- coding: utf-8 --
2	r"""
3	Routines here are for getting any NULL-terminated sequence of bytes evaluated
4	intact by any shell. This includes all variants of quotes, whitespace, and
5	non-printable characters.
6
7	Supported Shells
8	----------------
9
10	The following shells have been evaluated:
11
12	- Ubuntu (dash/sh)
13	- MacOS (GNU Bash)
14	- Zsh
15	- FreeBSD (sh)
16	- OpenBSD (sh)
17	- NetBSD (sh)
18
19	Debian Almquist shell (Dash)
20	~~~~~~~~~~~~~~~~~~~~~~~~~~~~
21
22	Ubuntu 14.04 and 16.04 use the Dash shell, and /bin/sh is actually just a
23	symlink to /bin/dash. The feature set supported when invoked as "sh" instead
24	of "dash" is different, and we focus exclusively on the "/bin/sh" implementation.
25
26	From the `Ubuntu Man Pages`_, every character except for single-quote
27	can be wrapped in single-quotes, and a backslash can be used to escape unquoted
28	single-quotes.
29
30	::
31
32	Quoting
33	Quoting is used to remove the special meaning of certain characters or
34	words to the shell, such as operators, whitespace, or keywords. There
35	are three types of quoting: matched single quotes, matched double quotes,
36	and backslash.
37
38	Backslash
39	A backslash preserves the literal meaning of the following character,
40	with the exception of ⟨newline⟩. A backslash preceding a ⟨newline⟩ is
41	treated as a line continuation.
42
43	Single Quotes
44	Enclosing characters in single quotes preserves the literal meaning of
45	all the characters (except single quotes, making it impossible to put
46	single-quotes in a single-quoted string).
47
48	Double Quotes
49	Enclosing characters within double quotes preserves the literal meaning
50	of all characters except dollarsign ($), backquote (`), and backslash
51	(\). The backslash inside double quotes is historically weird, and
52	serves to quote only the following characters:
53	$ ` " \ <newline>.
54	Otherwise it remains literal.
55
56	GNU Bash
57	~~~~~~~~
58
59	The Bash shell is default on many systems, though it is not generally the default
60	system-wide shell (i.e., the `system` syscall does not generally invoke it).
61
62	That said, its prevalence suggests that it also be addressed.
63
64	From the `GNU Bash Manual`_, every character except for single-quote
65	can be wrapped in single-quotes, and a backslash can be used to escape unquoted
66	single-quotes.
67
68	::
69
70	3.1.2.1 Escape Character
71
72	A non-quoted backslash ‘\’ is the Bash escape character. It preserves the
73	literal value of the next character that follows, with the exception of
74	newline. If a ``\\newline`` pair appears, and the backslash itself is not
75	quoted, the ``\\newline`` is treated as a line continuation (that is, it
76	is removed from the input stream and effectively ignored).
77
78	3.1.2.2 Single Quotes
79
80	Enclosing characters in single quotes (‘'’) preserves the literal value of
81	each character within the quotes. A single quote may not occur between single
82	uotes, even when preceded by a backslash.
83
84	3.1.2.3 Double Quotes
85
86	Enclosing characters in double quotes (‘"’) preserves the literal value of a
87	ll characters within the quotes, with the exception of ‘$’, ‘`’, ‘\’, and,
88	when history expansion is enabled, ‘!’. The characters ‘$’ and ‘`’ retain their
89	pecial meaning within double quotes (see Shell Expansions). The backslash retains
90	its special meaning only when followed by one of the following characters:
91	‘$’, ‘`’, ‘"’, ‘\’, or newline. Within double quotes, backslashes that are
92	followed by one of these characters are removed. Backslashes preceding
93	characters without a special meaning are left unmodified. A double quote may
94	be quoted within double quotes by preceding it with a backslash. If enabled,
95	history expansion will be performed unless an ‘!’ appearing in double quotes
96	is escaped using a backslash. The backslash preceding the ‘!’ is not removed.
97
98	The special parameters ‘*’ and ‘@’ have special meaning when in double quotes
99	see Shell Parameter Expansion).
100
101	Z Shell
102	~~~~~~~
103
104	The Z shell is also a relatively common user shell, even though it's not generally
105	the default system-wide shell.
106
107	From the `Z Shell Manual`_, every character except for single-quote
108	can be wrapped in single-quotes, and a backslash can be used to escape unquoted
109	single-quotes.
110
111	::
112
113	A character may be quoted (that is, made to stand for itself) by preceding
114	it with a ‘\’. ‘\’ followed by a newline is ignored.
115
116	A string enclosed between ‘$'’ and ‘'’ is processed the same way as the
117	string arguments of the print builtin, and the resulting string is considered
118	o be entirely quoted. A literal ‘'’ character can be included in the string
119	by using the ‘\\'’ escape.
120
121	All characters enclosed between a pair of single quotes ('') that is not
122	preceded by a ‘$’ are quoted. A single quote cannot appear within single
123	quotes unless the option RC_QUOTES is set, in which case a pair of single
124	quotes are turned into a single quote. For example,
125
126	print ''''
127	outputs nothing apart from a newline if RC_QUOTES is not set, but one single
128	quote if it is set.
129
130	Inside double quotes (""), parameter and command substitution occur, and
131	‘\’ quotes the characters ‘\’, ‘`’, ‘"’, and ‘$’.
132
133	FreeBSD Shell
134	~~~~~~~~~~~~~
135
136	Compatibility with the FreeBSD shell is included for completeness.
137
138	From the `FreeBSD man pages`_, every character except for single-quote
139	can be wrapped in single-quotes, and a backslash can be used to escape unquoted
140	single-quotes.
141
142	::
143
144	Quoting is used to remove the special meaning of certain characters or
145	words to the shell, such as operators, whitespace, keywords, or alias
146	names.
147
148	There are four types of quoting: matched single quotes, dollar-single
149	quotes, matched double quotes, and backslash.
150
151	Single Quotes
152	Enclosing characters in single quotes preserves the literal mean-
153	ing of all the characters (except single quotes, making it impos-
154	sible to put single-quotes in a single-quoted string).
155
156	Dollar-Single Quotes
157	Enclosing characters between $' and ' preserves the literal mean-
158	ing of all characters except backslashes and single quotes. A
159	backslash introduces a C-style escape sequence:
160
161	...
162
163	Double Quotes
164	Enclosing characters within double quotes preserves the literal
165	meaning of all characters except dollar sign (`$'), backquote
166	(``'), and backslash (`\\'). The backslash inside double quotes
167	is historically weird. It remains literal unless it precedes the
168	following characters, which it serves to quote:
169
170	$ ` " \ \\n
171
172	Backslash
173	A backslash preserves the literal meaning of the following char-
174	acter, with the exception of the newline character (`\\n'). A
175	backslash preceding a newline is treated as a line continuation.
176
177	OpenBSD Shell
178	~~~~~~~~~~~~~
179
180	From the `OpenBSD Man Pages`_, every character except for single-quote
181	can be wrapped in single-quotes, and a backslash can be used to escape unquoted
182	single-quotes.
183
184	::
185
186	A backslash (\) can be used to quote any character except a newline.
187	If a newline follows a backslash the shell removes them both, effectively
188	making the following line part of the current one.
189
190	A group of characters can be enclosed within single quotes (') to quote
191	every character within the quotes.
192
193	A group of characters can be enclosed within double quotes (") to quote
194	every character within the quotes except a backquote (`) or a dollar
195	sign ($), both of which retain their special meaning. A backslash (\)
196	within double quotes retains its special meaning, but only when followed
197	by a backquote, dollar sign, double quote, or another backslash.
198	An at sign (@) within double quotes has a special meaning
199	(see SPECIAL PARAMETERS, below).
200
201	NetBSD Shell
202	~~~~~~~~~~~~
203
204	The NetBSD shell's documentation is identical to the Dash documentation.
205
206	Android Shells
207	~~~~~~~~~~~~~~
208
209	Android has gone through some number of shells.
210
211	- Mksh, a Korn shell, was used with Toolbox releases (5.0 and prior)
212	- Toybox, also derived from the Almquist Shell (6.0 and newer)
213
214	Notably, the Toolbox implementation is not POSIX compliant
215	as it lacks a "printf" builtin (e.g. Android 5.0 emulator images).
216
217	Toybox Shell
218	~~~~~~~~~~~~
219
220	Android 6.0 (and possibly other versions) use a shell based on ``toybox``.
221
222	While it does not include a ``printf`` builtin, ``toybox`` itself includes
223	a POSIX-compliant ``printf`` binary.
224
225	The Ash shells should be feature-compatible with ``dash``.
226
227	BusyBox Shell
228	~~~~~~~~~~~~~
229
230	`BusyBox's Wikipedia page`_ claims to use an ``ash``-compliant shell,
231	and should therefore be compatible with ``dash``.
232
233
234	.. _Ubuntu Man Pages: https://manpages.ubuntu.com/manpages/trusty/man1/dash.1.html
235	.. _GNU Bash Manual: https://www.gnu.org/software/bash/manual/bash.html#Quoting
236	.. _Z Shell Manual: https://zsh.sourceforge.io/Doc/Release/Shell-Grammar.html#Quoting
237	.. _FreeBSD man pages: https://www.freebsd.org/cgi/man.cgi?query=sh
238	.. _OpenBSD Man Pages: https://man.openbsd.org/sh#SHELL_GRAMMAR
239	.. _BusyBox's Wikipedia page: https://en.wikipedia.org/wiki/BusyBox#Features
240	"""
241	from __future__ import absolute_import	1✔
242	from __future__ import division	1✔
243
244	import string	1✔
245	import subprocess	1✔
246
247	from pwnlib.context import context	1✔
248	from pwnlib.log import getLogger	1✔
249	from pwnlib.tubes.process import process	1✔
250	from pwnlib.util import fiddling	1✔
251	from pwnlib.util.misc import which, normalize_argv_env	1✔
252
253	log = getLogger(__name__)	1✔
254
255	def test_all():	1✔
UNCOV 256	test('a') ##	×
257	test('ab') ##	×
258	test('a b') ##	×
259	test(r"a\'b") ##	×
260	everything_1 = bytes(range(1,256))	×
261	for s in everything_1:	×
262	test(s)	×
263	test(s*4)	×
264	test(s * 2 + b'X')	×
265	test(b'X' + s * 2)	×
266	test((s2 + b'X') 2)	×
267	test(s + b'X' + s)	×
268	test(s2 + b'X' + s2)	×
269	test(b'X' + s*2 + b'X')	×
270	test(everything_1)	×
271	test(everything_1 * 2)	×
272	test(everything_1 * 4)	×
273	everything_2 = b''.join(bytes([c,c]) for c in range(1,256)) ##	×
274	test(everything_2)	×
275
UNCOV 276	test(randoms(1000, everything_1))	×
277
278
279	def test(original):	1✔
280	r"""Tests the output provided by a shell interpreting a string
281
282	.. doctest::
283	:options: +POSIX
284
285	>>> test(b'foobar')
286	>>> test(b'foo bar')
287	>>> test(b'foo bar\n')
288	>>> test(b"foo'bar")
289	>>> test(b"foo\\\\bar")
290	>>> test(b"foo\\\\'bar")
291	>>> test(b"foo\\x01'bar")
292	>>> test(b'\n')
293	>>> test(b'\xff')
294	>>> test(os.urandom(16 * 1024).replace(b'\x00', b''))
295	"""
296	input = sh_string(original)	1✔
297
298	if isinstance(input, str):	1!
UNCOV 299	input = input.encode()	×
300
301	cmdstr = b'/bin/echo %s' % input	1✔
302
303	SUPPORTED_SHELLS = [	1✔
304	['ash', '-c', cmdstr],
305	['bash', '-c', cmdstr],
306	['bash', '-o', 'posix', '-c', cmdstr],
307	['ksh', '-c', cmdstr],
308	['busybox', 'ash', '-c', cmdstr],
309	['busybox', 'sh', '-c', cmdstr],
310	['zsh', '-c', cmdstr],
311	['posh', '-c', cmdstr],
312	['dash', '-c', cmdstr],
313	['mksh', '-c', cmdstr],
314	['sh', '-c', cmdstr],
315	# ['adb', 'exec-out', cmdstr]
316	]
317
318	for shell in SUPPORTED_SHELLS:	1✔
319	binary = shell[0]	1✔
320
321	if not which(binary):	1✔
322	log.warn_once('Shell %r is not available' % binary)	1✔
323	continue	1✔
324
325	progress = log.progress('%s: %r' % (binary, original))	1✔
326
327	with context.quiet:	1✔
328	with process(shell) as p:	1✔
329	data = p.recvall(timeout=2)	1✔
330	p.kill()	1✔
331
332	# Remove exactly one trailing newline added by echo
333	# We cannot assume "echo -n" exists.
334	data = data[:-1]	1✔
335
336	if data != original:	1!
UNCOV 337	for i,(a,b) in enumerate(zip(data, original)):	×
338	if a == b:	×
339	continue	×
340	log.error(('Shell %r failed\n' +	×
341	'Expect %r\n' +
342	'Sent %r\n' +
343	'Output %r\n' +
344	'Mismatch @ %i: %r vs %r') \
345	% (binary, original, input, data, i, a, b))
346
347	progress.success()	1✔
348
349
350
351	SINGLE_QUOTE = "'" ##	1✔
352	ESCAPED_SINGLE_QUOTE = r"\'" ##	1✔
353
354	ESCAPED = {	1✔
355	# The single quote itself must be escaped, outside of single quotes.
356	"'": "\\'", ##
357
358	# Slashes must themselves be escaped
359	#
360	# Additionally, some shells coalesce any number N>1 of '\' into
361	# a single backslash literal.
362	# '\\': '"\\\\\\\\"'
363	}
364
365	def sh_string(s):	1✔
366	r"""Outputs a string in a format that will be understood by /bin/sh.
367
368	If the string does not contain any bad characters, it will simply be
369	returned, possibly with quotes. If it contains bad characters, it will
370	be escaped in a way which is compatible with most known systems.
371
372	Warning:
373	This does not play along well with the shell's built-in "echo".
374	It works exactly as expected to set environment variables and
375	arguments, unless it's the shell-builtin echo.
376
377	Argument:
378	s(str): String to escape.
379
380	Examples:
381
382	>>> sh_string('foobar')
383	'foobar'
384	>>> sh_string('foo bar')
385	"'foo bar'"
386	>>> sh_string("foo'bar")
387	"'foo'\\''bar'"
388	>>> sh_string("foo\\\\bar")
389	"'foo\\\\bar'"
390	>>> sh_string("foo\\\\'bar")
391	"'foo\\\\'\\''bar'"
392	>>> sh_string("foo\\x01'bar")
393	"'foo\\x01'\\''bar'"
394	"""
395	orig_s = s	1✔
396	if isinstance(s, (bytes, bytearray)):	1✔
397	s = s.decode('latin1')	1✔
398	if '\x00' in s: ##	1!
UNCOV 399	log.error("sh_string(): Cannot create a null-byte")	×
400
401	if not s:	1!
UNCOV 402	quoted_string = "''" ##	×
403	if isinstance(orig_s, (bytes, bytearray)):	×
404	quoted_string = quoted_string.encode('latin1')	×
405	return quoted_string	×
406
407	chars = set(s)	1✔
408	very_good = set(string.ascii_letters + string.digits + "_+.,/-") ##	1✔
409
410	# Alphanumeric can always just be used verbatim.
411	if chars <= very_good:	1✔
412	return orig_s	1✔
413
414	# If there are no single-quotes, the entire thing can be single-quoted
415	if not (chars & set(ESCAPED)):	1✔
416	quoted_string = "'%s'" % s ##	1✔
417	if isinstance(orig_s, (bytes, bytearray)):	1✔
418	quoted_string = quoted_string.encode('latin1')	1✔
419	return quoted_string	1✔
420
421	# If there are single-quotes, we can single-quote around them, and simply
422	# escape the single-quotes.
423	quoted_string = '' ##	1✔
424	quoted = False	1✔
425	for char in s: ##	1✔
426	if char not in ESCAPED:	1✔
427	if not quoted:	1✔
428	quoted_string += SINGLE_QUOTE	1✔
429	quoted = True	1✔
430	quoted_string += char ##	1✔
431	else:
432	if quoted:	1✔
433	quoted = False	1✔
434	quoted_string += SINGLE_QUOTE	1✔
435	quoted_string += ESCAPED[char]	1✔
436
437	if quoted:	1!
438	quoted_string += SINGLE_QUOTE	1✔
439
440	if isinstance(orig_s, (bytes, bytearray)):	1✔
441	quoted_string = quoted_string.encode('latin1')	1✔
442	return quoted_string	1✔
443
444	def sh_prepare(variables, export = False):	1✔
445	r"""Outputs a posix compliant shell command that will put the data specified
446	by the dictionary into the environment.
447
448	It is assumed that the keys in the dictionary are valid variable names that
449	does not need any escaping.
450
451	Arguments:
452	variables(dict): The variables to set.
453	export(bool): Should the variables be exported or only stored in the shell environment?
454	output(str): A valid posix shell command that will set the given variables.
455
456	It is assumed that `var` is a valid name for a variable in the shell.
457
458	Examples:
459
460	>>> sh_prepare({'X': 'foobar'})
461	b'X=foobar'
462	>>> r = sh_prepare({'X': 'foobar', 'Y': 'cookies'})
463	>>> r == b'X=foobar;Y=cookies' or r == b'Y=cookies;X=foobar' or r
464	True
465	>>> sh_prepare({'X': 'foo bar'})
466	b"X='foo bar'"
467	>>> sh_prepare({'X': "foo'bar"})
468	b"X='foo'\\''bar'"
469	>>> sh_prepare({'X': "foo\\\\bar"})
470	b"X='foo\\\\bar'"
471	>>> sh_prepare({'X': "foo\\\\'bar"})
472	b"X='foo\\\\'\\''bar'"
473	>>> sh_prepare({'X': "foo\\x01'bar"})
474	b"X='foo\\x01'\\''bar'"
475	>>> sh_prepare({'X': "foo\\x01'bar"}, export = True)
476	b"export X='foo\\x01'\\''bar'"
477	>>> sh_prepare({'X': "foo\\x01'bar\\n"})
478	b"X='foo\\x01'\\''bar\\n'"
479	>>> sh_prepare({'X': "foo\\x01'bar\\n"})
480	b"X='foo\\x01'\\''bar\\n'"
481	>>> sh_prepare({'X': "foo\\x01'bar\\n"}, export = True)
482	b"export X='foo\\x01'\\''bar\\n'"
483	"""
484
485	out = []	1✔
486	export = b'export ' if export else b''	1✔
487
488	_, variables = normalize_argv_env([], variables, log)	1✔
489
490	for k, v in variables:	1✔
491	out.append(b'%s%s=%s' % (export, k, sh_string(v)))	1✔
492
493	return b';'.join(out)	1✔
494
495	def sh_command_with(f, *args):	1✔
496	r"""sh_command_with(f, arg0, ..., argN) -> command
497
498	Returns a command create by evaluating `f(new_arg0, ..., new_argN)`
499	whenever `f` is a function and `f % (new_arg0, ..., new_argN)` otherwise.
500
501	If the arguments are purely alphanumeric, then they are simply passed to
502	function. If they are simple to escape, they will be escaped and passed to
503	the function.
504
505	If the arguments contain trailing newlines, then it is hard to use them
506	directly because of a limitation in the posix shell. In this case the
507	output from `f` is prepended with a bit of code to create the variables.
508
509	Examples:
510
511	>>> sh_command_with(lambda: "echo hello")
512	'echo hello'
513	>>> sh_command_with(lambda x: "echo " + x, "hello")
514	'echo hello'
515	>>> sh_command_with(lambda x: "/bin/echo " + x, "\\x01")
516	"/bin/echo '\\x01'"
517	>>> sh_command_with(lambda x: "/bin/echo " + x, "\\x01\\n")
518	"/bin/echo '\\x01\\n'"
519	>>> sh_command_with("/bin/echo %s", "\\x01\\n")
520	"/bin/echo '\\x01\\n'"
521	"""
522
523	args = list(args)	1✔
524	out = []	1✔
525
526	for n in range(len(args)):	1✔
527	args[n] = sh_string(args[n])	1✔
528	if hasattr(f, '__call__'):	1✔
529	out.append(f(*args))	1✔
530	else:
531	out.append(f % tuple(args))	1✔
532	return ';'.join(out)	1✔

Gallopsled / pwntools / 13600950642

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous