#38002

Committed 06 Feb 2025 06:14AM UTC coverage: 20.322% (-2.4%) from 22.722%

Build # #38002

Build Type

push

local

Committed by

web-flow

Commit Message

bpart: Fully switch to partitioned semantics (#57253)

This is the final PR in the binding partitions series (modulo bugs and
tweaks), i.e. it closes #54654 and thus closes #40399, which was the
original design sketch.

This thus activates the full designed semantics for binding partitions,
in particular allowing safe replacement of const bindings. It in
particular allows struct redefinitions. This thus closes
timholy/Revise.jl#18 and also closes #38584.

The biggest semantic change here is probably that this gets rid of the
notion of "resolvedness" of a binding. Previously, a lot of the behavior
of our implementation depended on when bindings were "resolved", which
could happen at basically an arbitrary point (in the compiler, in REPL
completion, in a different thread), making a lot of the semantics around
bindings ill- or at least implementation-defined. There are several
related issues in the bugtracker, so this closes #14055 closes #44604
closes #46354 closes #30277

It is also the last step to close #24569.
It also supports bindings for undef->defined transitions and thus closes
#53958 closes #54733 - however, this is not activated yet for
performance reasons and may need some further optimization.

Since resolvedness no longer exists, we need to replace it with some
hopefully more well-defined semantics. I will describe the semantics
below, but before I do I will make two notes:

1. There are a number of cases where these semantics will behave
slightly differently than the old semantics absent some other task going
around resolving random bindings.
2. The new behavior (except for the replacement stuff) was generally
permissible under the old semantics if the bindings happened to be
resolved at the right time.

With all that said, there are essentially three "strengths" of bindings:

1. Implicit Bindings: Anything implicitly obtained from `using Mod`, "no
binding", plus slightly more exotic corner cases around conflicts

2. Weakly declared bindin... (continued)

Run Details

11 of 111 new or added lines in 7 files covered. (9.91%)

1273 existing lines in 68 files now uncovered.

9908 of 48755 relevant lines covered (20.32%)

105126.48 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

53.89

/base/cmd.jl

# This file is a part of Julia. License is MIT: https://julialang.org/license

abstract type AbstractCmd end

# libuv process option flags
const UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS = UInt32(1 << 2)
const UV_PROCESS_DETACHED = UInt32(1 << 3)
const UV_PROCESS_WINDOWS_HIDE = UInt32(1 << 4)
const UV_PROCESS_WINDOWS_DISABLE_EXACT_NAME = UInt32(1 << 7)

struct Cmd <: AbstractCmd
    exec::Vector{String}
    ignorestatus::Bool
    flags::UInt32 # libuv process flags
    env::Union{Vector{String},Nothing}
    dir::String
    cpus::Union{Nothing,Vector{UInt16}}
    Cmd(exec::Vector{<:AbstractString}) =
        new(exec, false, 0x00, nothing, "", nothing)
    Cmd(cmd::Cmd, ignorestatus, flags, env, dir, cpus = nothing) =
        new(cmd.exec, ignorestatus, flags, env,
            dir === cmd.dir ? dir : cstr(dir), cpus)
    function Cmd(cmd::Cmd; ignorestatus::Bool=cmd.ignorestatus, env=cmd.env, dir::AbstractString=cmd.dir,
                 cpus::Union{Nothing,Vector{UInt16}} = cmd.cpus,
                 detach::Bool = 0 != cmd.flags & UV_PROCESS_DETACHED,
                 windows_verbatim::Bool = 0 != cmd.flags & UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS,
                 windows_hide::Bool = 0 != cmd.flags & UV_PROCESS_WINDOWS_HIDE)
        flags = detach * UV_PROCESS_DETACHED |
                windows_verbatim * UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS |
                windows_hide * UV_PROCESS_WINDOWS_HIDE
        new(cmd.exec, ignorestatus, flags, byteenv(env),
            dir === cmd.dir ? dir : cstr(dir), cpus)
    end
end

has_nondefault_cmd_flags(c::Cmd) =
    c.ignorestatus ||
    c.flags != 0x00 ||
    c.env !== nothing ||
    c.dir !== "" ||
    c.cpus !== nothing

"""
    Cmd(cmd::Cmd; ignorestatus, detach, windows_verbatim, windows_hide, env, dir)
    Cmd(exec::Vector{String})

Construct a new `Cmd` object, representing an external program and arguments, from `cmd`,
while changing the settings of the optional keyword arguments:

* `ignorestatus::Bool`: If `true` (defaults to `false`), then the `Cmd` will not throw an
  error if the return code is nonzero.
* `detach::Bool`: If `true` (defaults to `false`), then the `Cmd` will be run in a new
  process group, allowing it to outlive the `julia` process and not have Ctrl-C passed to
  it.
* `windows_verbatim::Bool`: If `true` (defaults to `false`), then on Windows the `Cmd` will
  send a command-line string to the process with no quoting or escaping of arguments, even
  arguments containing spaces. (On Windows, arguments are sent to a program as a single
  "command-line" string, and programs are responsible for parsing it into arguments. By
  default, empty arguments and arguments with spaces or tabs are quoted with double quotes
  `"` in the command line, and `\\` or `"` are preceded by backslashes.
  `windows_verbatim=true` is useful for launching programs that parse their command line in
  nonstandard ways.) Has no effect on non-Windows systems.
* `windows_hide::Bool`: If `true` (defaults to `false`), then on Windows no new console
  window is displayed when the `Cmd` is executed. This has no effect if a console is
  already open or on non-Windows systems.
* `env`: Set environment variables to use when running the `Cmd`. `env` is either a
  dictionary mapping strings to strings, an array of strings of the form `"var=val"`, an
  array or tuple of `"var"=>val` pairs. In order to modify (rather than replace) the
  existing environment, initialize `env` with `copy(ENV)` and then set `env["var"]=val` as
  desired.  To add to an environment block within a `Cmd` object without replacing all
  elements, use [`addenv()`](@ref) which will return a `Cmd` object with the updated environment.
* `dir::AbstractString`: Specify a working directory for the command (instead
  of the current directory).

For any keywords that are not specified, the current settings from `cmd` are used.

Note that the `Cmd(exec)` constructor does not create a copy of `exec`. Any subsequent changes to `exec` will be reflected in the `Cmd` object.

The most common way to construct a `Cmd` object is with command literals (backticks), e.g.

    `ls -l`

This can then be passed to the `Cmd` constructor to modify its settings, e.g.

    Cmd(`echo "Hello world"`, ignorestatus=true, detach=false)
"""
Cmd

hash(x::Cmd, h::UInt) = hash(x.exec, hash(x.env, hash(x.ignorestatus, hash(x.dir, hash(x.flags, h)))))
==(x::Cmd, y::Cmd) = x.exec == y.exec && x.env == y.env && x.ignorestatus == y.ignorestatus &&
                     x.dir == y.dir && isequal(x.flags, y.flags)

struct OrCmds <: AbstractCmd
    a::AbstractCmd
    b::AbstractCmd
    OrCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)
end

struct ErrOrCmds <: AbstractCmd
    a::AbstractCmd
    b::AbstractCmd
    ErrOrCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)
end

struct AndCmds <: AbstractCmd
    a::AbstractCmd
    b::AbstractCmd
    AndCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)
end

hash(x::AndCmds, h::UInt) = hash(x.a, hash(x.b, h))
==(x::AndCmds, y::AndCmds) = x.a == y.a && x.b == y.b

shell_escape(cmd::Cmd; special::AbstractString="") =
    shell_escape(cmd.exec..., special=special)
shell_escape_posixly(cmd::Cmd) =
    shell_escape_posixly(cmd.exec...)
shell_escape_csh(cmd::Cmd) =
    shell_escape_csh(cmd.exec...)
escape_microsoft_c_args(cmd::Cmd) =
    escape_microsoft_c_args(cmd.exec...)
escape_microsoft_c_args(io::IO, cmd::Cmd) =
    escape_microsoft_c_args(io::IO, cmd.exec...)

function show(io::IO, cmd::Cmd)
    print_env = cmd.env !== nothing
    print_dir = !isempty(cmd.dir)
    (print_env || print_dir) && print(io, "setenv(")
    print_cpus = cmd.cpus !== nothing
    print_cpus && print(io, "setcpuaffinity(")
    print(io, '`')
    join(io, map(cmd.exec) do arg
        replace(sprint(context=io) do io
            with_output_color(:underline, io) do io
                print_shell_word(io, arg, shell_special)
            end
        end, '`' => "\\`")
    end, ' ')
    print(io, '`')
    if print_cpus
        print(io, ", ")
        show(io, collect(Int, something(cmd.cpus)))
        print(io, ")")
    end
    print_env && (print(io, ","); show(io, cmd.env))
    print_dir && (print(io, "; dir="); show(io, cmd.dir))
    (print_dir || print_env) && print(io, ")")
    nothing
end

function show(io::IO, cmds::Union{OrCmds,ErrOrCmds})
    print(io, "pipeline(")
    show(io, cmds.a)
    print(io, ", ")
    print(io, isa(cmds, ErrOrCmds) ? "stderr=" : "stdout=")
    show(io, cmds.b)
    print(io, ")")
end

function show(io::IO, cmds::AndCmds)
    show(io, cmds.a)
    print(io, " & ")
    show(io, cmds.b)
end

const STDIN_NO  = 0
const STDOUT_NO = 1
const STDERR_NO = 2

struct FileRedirect
    filename::String
    append::Bool
    FileRedirect(filename::AbstractString, append::Bool) = FileRedirect(convert(String, filename), append)
    function FileRedirect(filename::String, append::Bool)
        if lowercase(filename) == (@static Sys.iswindows() ? "nul" : "/dev/null")
            @warn "For portability use devnull instead of a file redirect" maxlog=1
        end
        return new(filename, append)
    end
end

# setup_stdio ≈ cconvert
# rawhandle ≈ unsafe_convert
rawhandle(::DevNull) = C_NULL
rawhandle(x::OS_HANDLE) = x
if OS_HANDLE !== RawFD
    rawhandle(x::RawFD) = Libc._get_osfhandle(x)
end
setup_stdio(stdio::Union{DevNull,OS_HANDLE,RawFD}, ::Bool) = (stdio, false)

const Redirectable = Union{IO, FileRedirect, RawFD, OS_HANDLE}
const StdIOSet = NTuple{3, Redirectable}

struct CmdRedirect <: AbstractCmd
    cmd::AbstractCmd
    handle::Redirectable
    stream_no::Int
    readable::Bool
end
CmdRedirect(cmd, handle, stream_no) = CmdRedirect(cmd, handle, stream_no, stream_no == STDIN_NO)

function show(io::IO, cr::CmdRedirect)
    print(io, "pipeline(")
    show(io, cr.cmd)
    print(io, ", ")
    if cr.stream_no == STDOUT_NO
        print(io, "stdout")
    elseif cr.stream_no == STDERR_NO
        print(io, "stderr")
    elseif cr.stream_no == STDIN_NO
        print(io, "stdin")
    else
        print(io, cr.stream_no)
    end
    print(io, cr.readable ? "<" : ">")
    show(io, cr.handle)
    print(io, ")")
end

"""
    ignorestatus(command)

Mark a command object so that running it will not throw an error if the result code is non-zero.
"""
ignorestatus(cmd::Cmd) = Cmd(cmd, ignorestatus=true)
ignorestatus(cmd::Union{OrCmds,AndCmds}) =
    typeof(cmd)(ignorestatus(cmd.a), ignorestatus(cmd.b))

"""
    detach(command)

Mark a command object so that it will be run in a new process group, allowing it to outlive the julia process, and not have Ctrl-C interrupts passed to it.
"""
detach(cmd::Cmd) = Cmd(cmd; detach=true)

# like String(s), but throw an error if s contains NUL, since
# libuv requires NUL-terminated strings
function cstr(s)
    if Base.containsnul(s)
        throw(ArgumentError("strings containing NUL cannot be passed to spawned processes"))
    end
    return String(s)::String
end

# convert various env representations into an array of "key=val" strings
byteenv(env::AbstractArray{<:AbstractString}) =
    String[cstr(x) for x in env]
byteenv(env::AbstractDict) =
    String[cstr(string(k)*"="*string(v)) for (k,v) in env]
byteenv(env::Nothing) = nothing
byteenv(env::Union{AbstractVector{Pair{T,V}}, Tuple{Vararg{Pair{T,V}}}}) where {T<:AbstractString,V} =
    String[cstr(k*"="*string(v)) for (k,v) in env]

"""
    setenv(command::Cmd, env; dir)

Set environment variables to use when running the given `command`. `env` is either a
dictionary mapping strings to strings, an array of strings of the form `"var=val"`, or
zero or more `"var"=>val` pair arguments. In order to modify (rather than replace) the
existing environment, create `env` through `copy(ENV)` and then setting `env["var"]=val`
as desired, or use [`addenv`](@ref).

The `dir` keyword argument can be used to specify a working directory for the command.
`dir` defaults to the currently set `dir` for `command` (which is the current working
directory if not specified already).

See also [`Cmd`](@ref), [`addenv`](@ref), [`ENV`](@ref), [`pwd`](@ref).
"""
setenv(cmd::Cmd, env; dir=cmd.dir) = Cmd(cmd; env=byteenv(env), dir=dir)
setenv(cmd::Cmd, env::Pair{<:AbstractString}...; dir=cmd.dir) =
    setenv(cmd, env; dir=dir)
setenv(cmd::Cmd; dir=cmd.dir) = Cmd(cmd; dir=dir)

# split environment entry string into before and after first `=` (key and value)
function splitenv(e::String)
    i = findnext('=', e, 2)
    if i === nothing
        throw(ArgumentError("malformed environment entry"))
    end
    e[1:prevind(e, i)], e[nextind(e, i):end]
end

"""
    addenv(command::Cmd, env...; inherit::Bool = true)

Merge new environment mappings into the given [`Cmd`](@ref) object, returning a new `Cmd` object.
Duplicate keys are replaced.  If `command` does not contain any environment values set already,
it inherits the current environment at time of `addenv()` call if `inherit` is `true`.
Keys with value `nothing` are deleted from the env.

See also [`Cmd`](@ref), [`setenv`](@ref), [`ENV`](@ref).

!!! compat "Julia 1.6"
    This function requires Julia 1.6 or later.
"""
function addenv(cmd::Cmd, env::Dict; inherit::Bool = true)
    new_env = Dict{String,String}()
    if cmd.env === nothing
        if inherit
            merge!(new_env, ENV)
        end
    else
        for (k, v) in splitenv.(cmd.env)
            new_env[string(k)::String] = string(v)::String
        end
    end
    for (k, v) in env
        if v === nothing
            delete!(new_env, string(k)::String)
        else
            new_env[string(k)::String] = string(v)::String
        end
    end
    return setenv(cmd, new_env)
end

function addenv(cmd::Cmd, pairs::Pair{<:AbstractString}...; inherit::Bool = true)
    return addenv(cmd, Dict(k => v for (k, v) in pairs); inherit)
end

function addenv(cmd::Cmd, env::Vector{<:AbstractString}; inherit::Bool = true)
    return addenv(cmd, Dict(k => v for (k, v) in splitenv.(env)); inherit)
end

"""
    setcpuaffinity(original_command::Cmd, cpus) -> command::Cmd

Set the CPU affinity of the `command` by a list of CPU IDs (1-based) `cpus`.  Passing
`cpus = nothing` means to unset the CPU affinity if the `original_command` has any.

This function is supported only in Linux and Windows.  It is not supported in macOS because
libuv does not support affinity setting.

!!! compat "Julia 1.8"
    This function requires at least Julia 1.8.

# Examples

In Linux, the `taskset` command line program can be used to see how `setcpuaffinity` works.

```julia
julia> run(setcpuaffinity(`sh -c 'taskset -p \$\$'`, [1, 2, 5]));
pid 2273's current affinity mask: 13
```

Note that the mask value `13` reflects that the first, second, and the fifth bits (counting
from the least significant position) are turned on:

```julia
julia> 0b010011
0x13
```
"""
function setcpuaffinity end
setcpuaffinity(cmd::Cmd, ::Nothing) = Cmd(cmd; cpus = nothing)
setcpuaffinity(cmd::Cmd, cpus) = Cmd(cmd; cpus = collect(UInt16, cpus))

(&)(left::AbstractCmd, right::AbstractCmd) = AndCmds(left, right)
redir_out(src::AbstractCmd, dest::AbstractCmd) = OrCmds(src, dest)
redir_err(src::AbstractCmd, dest::AbstractCmd) = ErrOrCmds(src, dest)

# Stream Redirects
redir_out(dest::Redirectable, src::AbstractCmd) = CmdRedirect(src, dest, STDIN_NO)
redir_out(src::AbstractCmd, dest::Redirectable) = CmdRedirect(src, dest, STDOUT_NO)
redir_err(src::AbstractCmd, dest::Redirectable) = CmdRedirect(src, dest, STDERR_NO)

# File redirects
redir_out(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, false), STDOUT_NO)
redir_out(src::AbstractString, dest::AbstractCmd) = CmdRedirect(dest, FileRedirect(src, false), STDIN_NO)
redir_err(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, false), STDERR_NO)
redir_out_append(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, true), STDOUT_NO)
redir_err_append(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, true), STDERR_NO)

"""
    pipeline(command; stdin, stdout, stderr, append=false)

Redirect I/O to or from the given `command`. Keyword arguments specify which of the
command's streams should be redirected. `append` controls whether file output appends to the
file. This is a more general version of the 2-argument `pipeline` function.
`pipeline(from, to)` is equivalent to `pipeline(from, stdout=to)` when `from` is a command,
and to `pipeline(to, stdin=from)` when `from` is another kind of data source.

**Examples**:

```julia
run(pipeline(`dothings`, stdout="out.txt", stderr="errs.txt"))
run(pipeline(`update`, stdout="log.txt", append=true))
```
"""
function pipeline(cmd::AbstractCmd; stdin=nothing, stdout=nothing, stderr=nothing, append::Bool=false)
    if append && stdout === nothing && stderr === nothing
        throw(ArgumentError("append set to true, but no output redirections specified"))
    end
    if stdin !== nothing
        cmd = redir_out(stdin, cmd)
    end
    if stdout !== nothing
        cmd = append ? redir_out_append(cmd, stdout) : redir_out(cmd, stdout)
    end
    if stderr !== nothing
        cmd = append ? redir_err_append(cmd, stderr) : redir_err(cmd, stderr)
    end
    return cmd
end

pipeline(cmd::AbstractCmd, dest) = pipeline(cmd, stdout=dest)
pipeline(src::Union{Redirectable,AbstractString}, cmd::AbstractCmd) = pipeline(cmd, stdin=src)

"""
    pipeline(from, to, ...)

Create a pipeline from a data source to a destination. The source and destination can be
commands, I/O streams, strings, or results of other `pipeline` calls. At least one argument
must be a command. Strings refer to filenames. When called with more than two arguments,
they are chained together from left to right. For example, `pipeline(a,b,c)` is equivalent to
`pipeline(pipeline(a,b),c)`. This provides a more concise way to specify multi-stage
pipelines.

**Examples**:

```julia
run(pipeline(`ls`, `grep xyz`))
run(pipeline(`ls`, "out.txt"))
run(pipeline("out.txt", `grep xyz`))
```
"""
pipeline(a, b, c, d...) = pipeline(pipeline(a, b), c, d...)


## implementation of `cmd` syntax ##

cmd_interpolate(xs...) = cstr(string(map(cmd_interpolate1, xs)...))
cmd_interpolate1(x) = x
cmd_interpolate1(::Nothing) = throw(ArgumentError("`nothing` can not be interpolated into commands (`Cmd`)"))

arg_gen() = String[]
arg_gen(x::AbstractString) = String[cstr(x)]
function arg_gen(cmd::Cmd)
    if has_nondefault_cmd_flags(cmd)
        throw(ArgumentError("Non-default environment behavior is only permitted for the first interpolant."))
    end
    cmd.exec
end

function arg_gen(head)
    if isiterable(typeof(head))
        vals = String[]
        for x in head
            push!(vals, cmd_interpolate(x))
        end
        return vals
    else
        return String[cmd_interpolate(head)]
    end
end

function arg_gen(head, tail...)
    head = arg_gen(head)
    tail = arg_gen(tail...)
    vals = String[]
    for h = head, t = tail
        push!(vals, cmd_interpolate(h,t))
    end
    return vals
end

function cmd_gen(parsed)
    args = String[]
    if length(parsed) >= 1 && isa(parsed[1], Tuple{Cmd})
        cmd = (parsed[1]::Tuple{Cmd})[1]
        (ignorestatus, flags, env, dir) = (cmd.ignorestatus, cmd.flags, cmd.env, cmd.dir)
        append!(args, cmd.exec)
        for arg in tail(parsed)
            append!(args, Base.invokelatest(arg_gen, arg...)::Vector{String})
        end
        return Cmd(Cmd(args), ignorestatus, flags, env, dir)
    else
        for arg in parsed
            append!(args, arg_gen(arg...)::Vector{String})
        end
        return Cmd(args)
    end
end

@assume_effects :foldable !:consistent function cmd_gen(
    parsed::Tuple{Vararg{Tuple{Vararg{Union{String, SubString{String}}}}}}
)
    return @invoke cmd_gen(parsed::Any)
end

"""
    @cmd str

Similar to ``` `str` ```, generate a `Cmd` from the `str` string which represents the shell command(s) to be executed.
The [`Cmd`](@ref) object can be run as a process and can outlive the spawning julia process (see `Cmd` for more).

# Examples
```jldoctest
julia> cm = @cmd " echo 1 "
`echo 1`

julia> run(cm)
1
Process(`echo 1`, ProcessExited(0))
```
"""
macro cmd(str)
    cmd_ex = shell_parse(str, special=shell_special, filename=String(__source__.file))[1]
    return :(cmd_gen($(esc(cmd_ex))))
end

1	# This file is a part of Julia. License is MIT: https://julialang.org/license
2
3	abstract type AbstractCmd end
4
5	# libuv process option flags
6	const UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS = UInt32(1 << 2)
7	const UV_PROCESS_DETACHED = UInt32(1 << 3)
8	const UV_PROCESS_WINDOWS_HIDE = UInt32(1 << 4)
9	const UV_PROCESS_WINDOWS_DISABLE_EXACT_NAME = UInt32(1 << 7)
10
11	struct Cmd <: AbstractCmd
12	exec::Vector{String}
13	ignorestatus::Bool
14	flags::UInt32 # libuv process flags
15	env::Union{Vector{String},Nothing}
16	dir::String
17	cpus::Union{Nothing,Vector{UInt16}}
18	Cmd(exec::Vector{<:AbstractString}) =	70✔
19	new(exec, false, 0x00, nothing, "", nothing)
20	Cmd(cmd::Cmd, ignorestatus, flags, env, dir, cpus = nothing) =	742✔
21	new(cmd.exec, ignorestatus, flags, env,
22	dir === cmd.dir ? dir : cstr(dir), cpus)
23	function Cmd(cmd::Cmd; ignorestatus::Bool=cmd.ignorestatus, env=cmd.env, dir::AbstractString=cmd.dir,	15✔
24	cpus::Union{Nothing,Vector{UInt16}} = cmd.cpus,
25	detach::Bool = 0 != cmd.flags & UV_PROCESS_DETACHED,
26	windows_verbatim::Bool = 0 != cmd.flags & UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS,
27	windows_hide::Bool = 0 != cmd.flags & UV_PROCESS_WINDOWS_HIDE)
28	flags = detach * UV_PROCESS_DETACHED \|	15✔
29	windows_verbatim * UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS \|
30	windows_hide * UV_PROCESS_WINDOWS_HIDE
31	new(cmd.exec, ignorestatus, flags, byteenv(env),	15✔
32	dir === cmd.dir ? dir : cstr(dir), cpus)
33	end
34	end
35
UNCOV 36	has_nondefault_cmd_flags(c::Cmd) =	×
37	c.ignorestatus \|\|
38	c.flags != 0x00 \|\|
39	c.env !== nothing \|\|
40	c.dir !== "" \|\|
41	c.cpus !== nothing
42
43	"""
44	Cmd(cmd::Cmd; ignorestatus, detach, windows_verbatim, windows_hide, env, dir)
45	Cmd(exec::Vector{String})
46
47	Construct a new `Cmd` object, representing an external program and arguments, from `cmd`,
48	while changing the settings of the optional keyword arguments:
49
50	* `ignorestatus::Bool`: If `true` (defaults to `false`), then the `Cmd` will not throw an
51	error if the return code is nonzero.
52	* `detach::Bool`: If `true` (defaults to `false`), then the `Cmd` will be run in a new
53	process group, allowing it to outlive the `julia` process and not have Ctrl-C passed to
54	it.
55	* `windows_verbatim::Bool`: If `true` (defaults to `false`), then on Windows the `Cmd` will
56	send a command-line string to the process with no quoting or escaping of arguments, even
57	arguments containing spaces. (On Windows, arguments are sent to a program as a single
58	"command-line" string, and programs are responsible for parsing it into arguments. By
59	default, empty arguments and arguments with spaces or tabs are quoted with double quotes
60	`"` in the command line, and `\\` or `"` are preceded by backslashes.
61	`windows_verbatim=true` is useful for launching programs that parse their command line in
62	nonstandard ways.) Has no effect on non-Windows systems.
63	* `windows_hide::Bool`: If `true` (defaults to `false`), then on Windows no new console
64	window is displayed when the `Cmd` is executed. This has no effect if a console is
65	already open or on non-Windows systems.
66	* `env`: Set environment variables to use when running the `Cmd`. `env` is either a
67	dictionary mapping strings to strings, an array of strings of the form `"var=val"`, an
68	array or tuple of `"var"=>val` pairs. In order to modify (rather than replace) the
69	existing environment, initialize `env` with `copy(ENV)` and then set `env["var"]=val` as
70	desired. To add to an environment block within a `Cmd` object without replacing all
71	elements, use [`addenv()`](@ref) which will return a `Cmd` object with the updated environment.
72	* `dir::AbstractString`: Specify a working directory for the command (instead
73	of the current directory).
74
75	For any keywords that are not specified, the current settings from `cmd` are used.
76
77	Note that the `Cmd(exec)` constructor does not create a copy of `exec`. Any subsequent changes to `exec` will be reflected in the `Cmd` object.
78
79	The most common way to construct a `Cmd` object is with command literals (backticks), e.g.
80
81	`ls -l`
82
83	This can then be passed to the `Cmd` constructor to modify its settings, e.g.
84
85	Cmd(`echo "Hello world"`, ignorestatus=true, detach=false)
86	"""
87	Cmd
88
89	hash(x::Cmd, h::UInt) = hash(x.exec, hash(x.env, hash(x.ignorestatus, hash(x.dir, hash(x.flags, h)))))	×
90	==(x::Cmd, y::Cmd) = x.exec == y.exec && x.env == y.env && x.ignorestatus == y.ignorestatus &&	×
91	x.dir == y.dir && isequal(x.flags, y.flags)
92
93	struct OrCmds <: AbstractCmd
94	a::AbstractCmd
95	b::AbstractCmd
96	OrCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)	×
97	end
98
99	struct ErrOrCmds <: AbstractCmd
100	a::AbstractCmd
101	b::AbstractCmd
102	ErrOrCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)	×
103	end
104
105	struct AndCmds <: AbstractCmd
106	a::AbstractCmd
107	b::AbstractCmd
108	AndCmds(a::AbstractCmd, b::AbstractCmd) = new(a, b)	×
109	end
110
111	hash(x::AndCmds, h::UInt) = hash(x.a, hash(x.b, h))	×
112	==(x::AndCmds, y::AndCmds) = x.a == y.a && x.b == y.b	×
113
114	shell_escape(cmd::Cmd; special::AbstractString="") =	×
115	shell_escape(cmd.exec..., special=special)
116	shell_escape_posixly(cmd::Cmd) =	×
117	shell_escape_posixly(cmd.exec...)
118	shell_escape_csh(cmd::Cmd) =	×
119	shell_escape_csh(cmd.exec...)
120	escape_microsoft_c_args(cmd::Cmd) =	×
121	escape_microsoft_c_args(cmd.exec...)
122	escape_microsoft_c_args(io::IO, cmd::Cmd) =	×
123	escape_microsoft_c_args(io::IO, cmd.exec...)
124
125	function show(io::IO, cmd::Cmd)	5✔
126	print_env = cmd.env !== nothing	5✔
127	print_dir = !isempty(cmd.dir)	5✔
128	(print_env \|\| print_dir) && print(io, "setenv(")	9✔
129	print_cpus = cmd.cpus !== nothing	5✔
130	print_cpus && print(io, "setcpuaffinity(")	5✔
131	print(io, '`')	5✔
132	join(io, map(cmd.exec) do arg	5✔
133	replace(sprint(context=io) do io	50✔
134	with_output_color(:underline, io) do io	316✔
135	print_shell_word(io, arg, shell_special)	316✔
136	end
137	end, '`' => "\\`")
138	end, ' ')
139	print(io, '`')	5✔
140	if print_cpus	5✔
141	print(io, ", ")	×
142	show(io, collect(Int, something(cmd.cpus)))	×
143	print(io, ")")	×
144	end
145	print_env && (print(io, ","); show(io, cmd.env))	6✔
146	print_dir && (print(io, "; dir="); show(io, cmd.dir))	5✔
147	(print_dir \|\| print_env) && print(io, ")")	10✔
148	nothing	5✔
149	end
150
151	function show(io::IO, cmds::Union{OrCmds,ErrOrCmds})	×
152	print(io, "pipeline(")	×
153	show(io, cmds.a)	×
154	print(io, ", ")	×
155	print(io, isa(cmds, ErrOrCmds) ? "stderr=" : "stdout=")	×
156	show(io, cmds.b)	×
157	print(io, ")")	×
158	end
159
160	function show(io::IO, cmds::AndCmds)	×
161	show(io, cmds.a)	×
162	print(io, " & ")	×
163	show(io, cmds.b)	×
164	end
165
166	const STDIN_NO = 0
167	const STDOUT_NO = 1
168	const STDERR_NO = 2
169
170	struct FileRedirect
171	filename::String
172	append::Bool
173	FileRedirect(filename::AbstractString, append::Bool) = FileRedirect(convert(String, filename), append)	×
174	function FileRedirect(filename::String, append::Bool)	×
175	if lowercase(filename) == (@static Sys.iswindows() ? "nul" : "/dev/null")	×
176	@warn "For portability use devnull instead of a file redirect" maxlog=1	×
177	end
178	return new(filename, append)	×
179	end
180	end
181
182	# setup_stdio ≈ cconvert
183	# rawhandle ≈ unsafe_convert
184	rawhandle(::DevNull) = C_NULL	×
185	rawhandle(x::OS_HANDLE) = x	×
186	if OS_HANDLE !== RawFD
187	rawhandle(x::RawFD) = Libc._get_osfhandle(x)	×
188	end
UNCOV 189	setup_stdio(stdio::Union{DevNull,OS_HANDLE,RawFD}, ::Bool) = (stdio, false)	×
190
191	const Redirectable = Union{IO, FileRedirect, RawFD, OS_HANDLE}
192	const StdIOSet = NTuple{3, Redirectable}
193
194	struct CmdRedirect <: AbstractCmd
195	cmd::AbstractCmd	183✔
196	handle::Redirectable
197	stream_no::Int
198	readable::Bool
199	end
200	CmdRedirect(cmd, handle, stream_no) = CmdRedirect(cmd, handle, stream_no, stream_no == STDIN_NO)	183✔
201
202	function show(io::IO, cr::CmdRedirect)	×
203	print(io, "pipeline(")	×
204	show(io, cr.cmd)	×
205	print(io, ", ")	×
206	if cr.stream_no == STDOUT_NO	×
207	print(io, "stdout")	×
208	elseif cr.stream_no == STDERR_NO	×
209	print(io, "stderr")	×
210	elseif cr.stream_no == STDIN_NO	×
211	print(io, "stdin")	×
212	else
213	print(io, cr.stream_no)	×
214	end
215	print(io, cr.readable ? "<" : ">")	×
216	show(io, cr.handle)	×
217	print(io, ")")	×
218	end
219
220	"""
221	ignorestatus(command)
222
223	Mark a command object so that running it will not throw an error if the result code is non-zero.
224	"""
225	ignorestatus(cmd::Cmd) = Cmd(cmd, ignorestatus=true)	43✔
226	ignorestatus(cmd::Union{OrCmds,AndCmds}) =	×
227	typeof(cmd)(ignorestatus(cmd.a), ignorestatus(cmd.b))
228
229	"""
230	detach(command)
231
232	Mark a command object so that it will be run in a new process group, allowing it to outlive the julia process, and not have Ctrl-C interrupts passed to it.
233	"""
234	detach(cmd::Cmd) = Cmd(cmd; detach=true)	4✔
235
236	# like String(s), but throw an error if s contains NUL, since
237	# libuv requires NUL-terminated strings
238	function cstr(s)
239	if Base.containsnul(s)	44✔
240	throw(ArgumentError("strings containing NUL cannot be passed to spawned processes"))	×
241	end
242	return String(s)::String	44✔
243	end
244
245	# convert various env representations into an array of "key=val" strings
UNCOV 246	byteenv(env::AbstractArray{<:AbstractString}) =	×
247	String[cstr(x) for x in env]
UNCOV 248	byteenv(env::AbstractDict) =	×
249	String[cstr(string(k)"="string(v)) for (k,v) in env]
250	byteenv(env::Nothing) = nothing	×
251	byteenv(env::Union{AbstractVector{Pair{T,V}}, Tuple{Vararg{Pair{T,V}}}}) where {T<:AbstractString,V} =	16✔
252	String[cstr(k"="string(v)) for (k,v) in env]
253
254	"""
255	setenv(command::Cmd, env; dir)
256
257	Set environment variables to use when running the given `command`. `env` is either a
258	dictionary mapping strings to strings, an array of strings of the form `"var=val"`, or
259	zero or more `"var"=>val` pair arguments. In order to modify (rather than replace) the
260	existing environment, create `env` through `copy(ENV)` and then setting `env["var"]=val`
261	as desired, or use [`addenv`](@ref).
262
263	The `dir` keyword argument can be used to specify a working directory for the command.
264	`dir` defaults to the currently set `dir` for `command` (which is the current working
265	directory if not specified already).
266
267	See also [`Cmd`](@ref), [`addenv`](@ref), [`ENV`](@ref), [`pwd`](@ref).
268	"""
269	setenv(cmd::Cmd, env; dir=cmd.dir) = Cmd(cmd; env=byteenv(env), dir=dir)	78✔
270	setenv(cmd::Cmd, env::Pair{<:AbstractString}...; dir=cmd.dir) =	32✔
271	setenv(cmd, env; dir=dir)
272	setenv(cmd::Cmd; dir=cmd.dir) = Cmd(cmd; dir=dir)	8✔
273
274	# split environment entry string into before and after first `=` (key and value)
275	function splitenv(e::String)	×
276	i = findnext('=', e, 2)	×
277	if i === nothing	×
278	throw(ArgumentError("malformed environment entry"))	×
279	end
280	e[1:prevind(e, i)], e[nextind(e, i):end]	×
281	end
282
283	"""
284	addenv(command::Cmd, env...; inherit::Bool = true)
285
286	Merge new environment mappings into the given [`Cmd`](@ref) object, returning a new `Cmd` object.
287	Duplicate keys are replaced. If `command` does not contain any environment values set already,
288	it inherits the current environment at time of `addenv()` call if `inherit` is `true`.
289	Keys with value `nothing` are deleted from the env.
290
291	See also [`Cmd`](@ref), [`setenv`](@ref), [`ENV`](@ref).
292
293	!!! compat "Julia 1.6"
294	This function requires Julia 1.6 or later.
295	"""
296	function addenv(cmd::Cmd, env::Dict; inherit::Bool = true)	44✔
297	new_env = Dict{String,String}()	22✔
298	if cmd.env === nothing	22✔
299	if inherit	22✔
300	merge!(new_env, ENV)	22✔
301	end
302	else
303	for (k, v) in splitenv.(cmd.env)	×
304	new_env[string(k)::String] = string(v)::String	×
305	end	×
306	end
307	for (k, v) in env	44✔
308	if v === nothing	48✔
309	delete!(new_env, string(k)::String)	18✔
310	else
311	new_env[string(k)::String] = string(v)::String	30✔
312	end
313	end	74✔
314	return setenv(cmd, new_env)	22✔
315	end
316
317	function addenv(cmd::Cmd, pairs::Pair{<:AbstractString}...; inherit::Bool = true)	18✔
318	return addenv(cmd, Dict(k => v for (k, v) in pairs); inherit)	18✔
319	end
320
321	function addenv(cmd::Cmd, env::Vector{<:AbstractString}; inherit::Bool = true)	×
322	return addenv(cmd, Dict(k => v for (k, v) in splitenv.(env)); inherit)	×
323	end
324
325	"""
326	setcpuaffinity(original_command::Cmd, cpus) -> command::Cmd
327
328	Set the CPU affinity of the `command` by a list of CPU IDs (1-based) `cpus`. Passing
329	`cpus = nothing` means to unset the CPU affinity if the `original_command` has any.
330
331	This function is supported only in Linux and Windows. It is not supported in macOS because
332	libuv does not support affinity setting.
333
334	!!! compat "Julia 1.8"
335	This function requires at least Julia 1.8.
336
337	# Examples
338
339	In Linux, the `taskset` command line program can be used to see how `setcpuaffinity` works.
340
341	```julia
342	julia> run(setcpuaffinity(`sh -c 'taskset -p \$\$'`, [1, 2, 5]));
343	pid 2273's current affinity mask: 13
344	```
345
346	Note that the mask value `13` reflects that the first, second, and the fifth bits (counting
347	from the least significant position) are turned on:
348
349	```julia
350	julia> 0b010011
351	0x13
352	```
353	"""
354	function setcpuaffinity end
355	setcpuaffinity(cmd::Cmd, ::Nothing) = Cmd(cmd; cpus = nothing)	×
356	setcpuaffinity(cmd::Cmd, cpus) = Cmd(cmd; cpus = collect(UInt16, cpus))	×
357
358	(&)(left::AbstractCmd, right::AbstractCmd) = AndCmds(left, right)	×
359	redir_out(src::AbstractCmd, dest::AbstractCmd) = OrCmds(src, dest)	×
360	redir_err(src::AbstractCmd, dest::AbstractCmd) = ErrOrCmds(src, dest)	×
361
362	# Stream Redirects
363	redir_out(dest::Redirectable, src::AbstractCmd) = CmdRedirect(src, dest, STDIN_NO)	2✔
364	redir_out(src::AbstractCmd, dest::Redirectable) = CmdRedirect(src, dest, STDOUT_NO)	84✔
365	redir_err(src::AbstractCmd, dest::Redirectable) = CmdRedirect(src, dest, STDERR_NO)	92✔
366
367	# File redirects
368	redir_out(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, false), STDOUT_NO)	×
369	redir_out(src::AbstractString, dest::AbstractCmd) = CmdRedirect(dest, FileRedirect(src, false), STDIN_NO)	5✔
370	redir_err(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, false), STDERR_NO)	×
371	redir_out_append(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, true), STDOUT_NO)	×
372	redir_err_append(src::AbstractCmd, dest::AbstractString) = CmdRedirect(src, FileRedirect(dest, true), STDERR_NO)	×
373
374	"""
375	pipeline(command; stdin, stdout, stderr, append=false)
376
377	Redirect I/O to or from the given `command`. Keyword arguments specify which of the
378	command's streams should be redirected. `append` controls whether file output appends to the
379	file. This is a more general version of the 2-argument `pipeline` function.
380	`pipeline(from, to)` is equivalent to `pipeline(from, stdout=to)` when `from` is a command,
381	and to `pipeline(to, stdin=from)` when `from` is another kind of data source.
382
383	Examples:
384
385	```julia
386	run(pipeline(`dothings`, stdout="out.txt", stderr="errs.txt"))
387	run(pipeline(`update`, stdout="log.txt", append=true))
388	```
389	"""
390	function pipeline(cmd::AbstractCmd; stdin=nothing, stdout=nothing, stderr=nothing, append::Bool=false)	108✔
391	if append && stdout === nothing && stderr === nothing	108✔
392	throw(ArgumentError("append set to true, but no output redirections specified"))	×
393	end
394	if stdin !== nothing	108✔
395	cmd = redir_out(stdin, cmd)	7✔
396	end
397	if stdout !== nothing	108✔
398	cmd = append ? redir_out_append(cmd, stdout) : redir_out(cmd, stdout)	84✔
399	end
400	if stderr !== nothing	108✔
401	cmd = append ? redir_err_append(cmd, stderr) : redir_err(cmd, stderr)	92✔
402	end
403	return cmd	108✔
404	end
405
406	pipeline(cmd::AbstractCmd, dest) = pipeline(cmd, stdout=dest)	1✔
407	pipeline(src::Union{Redirectable,AbstractString}, cmd::AbstractCmd) = pipeline(cmd, stdin=src)	1✔
408
409	"""
410	pipeline(from, to, ...)
411
412	Create a pipeline from a data source to a destination. The source and destination can be
413	commands, I/O streams, strings, or results of other `pipeline` calls. At least one argument
414	must be a command. Strings refer to filenames. When called with more than two arguments,
415	they are chained together from left to right. For example, `pipeline(a,b,c)` is equivalent to
416	`pipeline(pipeline(a,b),c)`. This provides a more concise way to specify multi-stage
417	pipelines.
418
419	Examples:
420
421	```julia
422	run(pipeline(`ls`, `grep xyz`))
423	run(pipeline(`ls`, "out.txt"))
424	run(pipeline("out.txt", `grep xyz`))
425	```
426	"""
427	pipeline(a, b, c, d...) = pipeline(pipeline(a, b), c, d...)	1✔
428
429
430	## implementation of `cmd` syntax ##
431
432	cmd_interpolate(xs...) = cstr(string(map(cmd_interpolate1, xs)...))	10✔
433	cmd_interpolate1(x) = x	2✔
434	cmd_interpolate1(::Nothing) = throw(ArgumentError("`nothing` can not be interpolated into commands (`Cmd`)"))	×
435
UNCOV 436	arg_gen() = String[]	×
UNCOV 437	arg_gen(x::AbstractString) = String[cstr(x)]	×
UNCOV 438	function arg_gen(cmd::Cmd)	×
UNCOV 439	if has_nondefault_cmd_flags(cmd)	×
440	throw(ArgumentError("Non-default environment behavior is only permitted for the first interpolant."))	×
441	end
UNCOV 442	cmd.exec	×
443	end
444
445	function arg_gen(head)	3✔
446	if isiterable(typeof(head))	3✔
447	vals = String[]	3✔
448	for x in head	3✔
449	push!(vals, cmd_interpolate(x))	2✔
450	end	2✔
451	return vals	3✔
452	else
453	return String[cmd_interpolate(head)]	×
454	end
455	end
456
457	function arg_gen(head, tail...)	8✔
458	head = arg_gen(head)	8✔
459	tail = arg_gen(tail...)	8✔
460	vals = String[]	8✔
461	for h = head, t = tail	8✔
462	push!(vals, cmd_interpolate(h,t))	8✔
463	end	8✔
464	return vals	8✔
465	end
466
467	function cmd_gen(parsed)	380✔
468	args = String[]	381✔
469	if length(parsed) >= 1 && isa(parsed[1], Tuple{Cmd})	381✔
470	cmd = (parsed[1]::Tuple{Cmd})[1]	371✔
471	(ignorestatus, flags, env, dir) = (cmd.ignorestatus, cmd.flags, cmd.env, cmd.dir)	371✔
472	append!(args, cmd.exec)	740✔
473	for arg in tail(parsed)	371✔
474	append!(args, Base.invokelatest(arg_gen, arg...)::Vector{String})	2,201✔
475	end	1,437✔
476	return Cmd(Cmd(args), ignorestatus, flags, env, dir)	371✔
477	else
478	for arg in parsed	10✔
479	append!(args, arg_gen(arg...)::Vector{String})	56✔
480	end	37✔
481	return Cmd(args)	10✔
482	end
483	end
484
485	@assume_effects :foldable !:consistent function cmd_gen(	9✔
486	parsed::Tuple{Vararg{Tuple{Vararg{Union{String, SubString{String}}}}}}
487	)
488	return @invoke cmd_gen(parsed::Any)	30✔
489	end
490
491	"""
492	@cmd str
493
494	Similar to ``` `str` ```, generate a `Cmd` from the `str` string which represents the shell command(s) to be executed.
495	The [`Cmd`](@ref) object can be run as a process and can outlive the spawning julia process (see `Cmd` for more).
496
497	# Examples
498	```jldoctest
499	julia> cm = @cmd " echo 1 "
500	`echo 1`
501
502	julia> run(cm)
503	1
504	Process(`echo 1`, ProcessExited(0))
505	```
506	"""
507	macro cmd(str)	225✔
508	cmd_ex = shell_parse(str, special=shell_special, filename=String(__source__.file))[1]	225✔
509	return :(cmd_gen($(esc(cmd_ex))))	225✔
510	end

JuliaLang / julia / #38002

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous