Shell or Suspicious Process Spawned by Ray Worker

Description

Detects a Ray worker (ray/_private/workers/default_worker.py) spawning a shell, an inline Python command (python -c / python3 -c), or a discovery or download utility. Ray runs job entrypoints from that worker through a shell, so a Python reverse shell shows up as sh -c or bash -c whose command line contains python -c and the program it starts, such as /bin/sh -i. On an exposed cluster this is exploitation of the public-facing Ray jobs API (CVE-2023-48022). Ray commonly hosts LLM training, fine-tuning, and serving, so this can be an attack on that infrastructure: theft of model weights, datasets, or API and cloud tokens, tampering with a served model, or use of the GPUs for cryptomining.

Query · esql

FROM logs-endpoint.events.process-* METADATA _id, _version, _index
| WHERE KQL("""event.category:process and event.type:start""")
  AND TO_LOWER(process.parent.name) LIKE "python*"
  AND (
    TO_LOWER(process.parent.command_line) LIKE """*\\ray\\_private\\workers\\default_worker.py*"""
    OR TO_LOWER(process.parent.command_line) LIKE "*/ray/_private/workers/default_worker.py*"
  )
  AND (
    TO_LOWER(process.name) IN (
      "cmd.exe", "powershell.exe", "pwsh.exe",
      "bash", "bash.exe", "sh", "zsh", "dash",
      "curl", "curl.exe", "wget", "wget.exe",
      "whoami", "whoami.exe", "id", "uname", "hostname", "hostname.exe"
    )
    // Direct python -c child of the worker.
    OR (
      TO_LOWER(process.name) LIKE "python*"
      AND MV_CONTAINS(process.args, "-c")
    )
  )
| KEEP
    _id,
    _version,
    _index,
    data_stream.namespace,
    @timestamp,
    host.name,
    host.id,
    host.os.type,
    agent.id,
    user.name,
    user.domain,
    process.name,
    process.executable,
    process.command_line,
    process.args,
    process.parent.name,
    process.parent.executable,
    process.parent.command_line,
    process.parent.entity_id

Investigation fields

Pivot points the source recommends for triage.

  • host.name
  • user.name
  • process.parent.name
  • process.parent.executable
  • process.parent.command_line
  • process.name
  • process.executable
  • process.command_line

Implementation guide

This rule is designed for data generated by Elastic Defend, which provides native endpoint detection and response, along with event enrichments designed to work with our detection rules.

Setup instructions: https://ela.st/install-elastic-defend

Analyst notes

Investigating Shell or Suspicious Process Spawned by Ray Worker

A Python process running ray/_private/workers/default_worker.py started cmd, PowerShell, a Unix shell, inline python -c / python3 -c, curl, wget, whoami, id, uname, or hostname. Ray's job supervisor is that worker. It runs the entrypoint with a shell, so the direct child is usually sh -c or cmd.exe /c and the entrypoint text is on that command line. A python3 -c reverse shell that then runs /bin/sh -i is visible on that shell command line. The python3 -c process and the interactive /bin/sh are children of the shell, not of the worker, so they are separate events.

Possible investigation steps

  • Review process.command_line for the entrypoint. Downloaders, whoami, id, encoded commands, and reverse shells are higher priority than a known training or serving command.
  • Confirm process.parent.command_line contains ray/_private/workers/default_worker.py.
  • On the same host, review the child and grandchild processes and any outbound connections from that tree.
  • Check whether the Ray dashboard (port 8265 by default) is reachable from outside the cluster, and who submitted the job. The submitter is not the parent process.
  • Ask the workload owner whether this entrypoint is an approved Ray job.

False positive analysis

  • Approved Ray jobs start their entrypoint from this same worker, usually through cmd.exe /c on Windows or sh -c on Linux. Expect this rule on clusters that run jobs.
  • Tune with the entrypoint command line, the Ray working directory, the host, or the user. Do not exclude python or default_worker.py broadly, or the worker respawn noise returns and the entrypoint shell is missed.

Response and remediation

  • If the entrypoint is not an approved job, stop the child process and isolate the host if it made outbound connections or read credentials.
  • Remove unapproved files dropped by the job and rotate credentials present in the Ray environment.
  • Stop exposing the dashboard. Bind it to a trusted interface and require authentication in front of it.
Raw source Shell or Suspicious Process Spawned by Ray Worker · Elastic TOML
Esc
Published by elastic/detection-rules ↗, licensed under Elastic License 2.0 ↗. Reproduced here unmodified. 
[metadata]
creation_date = "2026/10/29"
integration = ["endpoint"]
maturity = "production"
min_stack_comments = "MV_CONTAINS was added to ES|QL in 9.2.0 and is not available on 8.19; 9.3.0 is the lowest supported 9.x release."
min_stack_version = "9.3.0"
updated_date = "2026/10/29"

[rule]
author = ["Elastic"]
description = """
Detects a Ray worker (`ray/_private/workers/default_worker.py`) spawning a shell, an inline Python
command (`python -c` / `python3 -c`), or a discovery or download utility. Ray runs job entrypoints from
that worker through a shell, so a Python reverse shell shows up as `sh -c` or `bash -c` whose command
line contains `python -c` and the program it starts, such as `/bin/sh -i`. On an exposed cluster this is
exploitation of the public-facing Ray jobs API (CVE-2023-48022). Ray commonly hosts LLM
training, fine-tuning, and serving, so this can be an attack on that infrastructure: theft of model
weights, datasets, or API and cloud tokens, tampering with a served model, or use of the GPUs for
cryptomining.
"""
from = "now-9m"
language = "esql"
license = "Elastic License v2"
name = "Shell or Suspicious Process Spawned by Ray Worker"
note = """## Triage and analysis

### Investigating Shell or Suspicious Process Spawned by Ray Worker

A Python process running `ray/_private/workers/default_worker.py` started `cmd`, PowerShell, a Unix shell,
inline `python -c` / `python3 -c`, `curl`, `wget`, `whoami`, `id`, `uname`, or `hostname`. Ray's job
supervisor is that worker. It runs the entrypoint with a shell, so the direct child is usually `sh -c` or
`cmd.exe /c` and the entrypoint text is on that command line. A `python3 -c` reverse shell that then runs
`/bin/sh -i` is visible on that shell command line. The `python3 -c` process and the interactive `/bin/sh`
are children of the shell, not of the worker, so they are separate events.

### Possible investigation steps

- Review `process.command_line` for the entrypoint. Downloaders, `whoami`, `id`, encoded commands, and
  reverse shells are higher priority than a known training or serving command.
- Confirm `process.parent.command_line` contains `ray/_private/workers/default_worker.py`.
- On the same host, review the child and grandchild processes and any outbound connections from that tree.
- Check whether the Ray dashboard (port 8265 by default) is reachable from outside the cluster, and who
  submitted the job. The submitter is not the parent process.
- Ask the workload owner whether this entrypoint is an approved Ray job.

### False positive analysis

- Approved Ray jobs start their entrypoint from this same worker, usually through `cmd.exe /c` on Windows or
  `sh -c` on Linux. Expect this rule on clusters that run jobs.
- Tune with the entrypoint command line, the Ray working directory, the host, or the user. Do not exclude
  `python` or `default_worker.py` broadly, or the worker respawn noise returns and the entrypoint shell is missed.

### Response and remediation

- If the entrypoint is not an approved job, stop the child process and isolate the host if it made outbound
  connections or read credentials.
- Remove unapproved files dropped by the job and rotate credentials present in the Ray environment.
- Stop exposing the dashboard. Bind it to a trusted interface and require authentication in front of it.
"""
references = [
    "https://www.oligo.security/blog/shadowray-attack-ai-workloads-actively-exploited-in-the-wild",
    "https://nvd.nist.gov/vuln/detail/CVE-2023-48022",
    "https://docs.ray.io/en/latest/ray-security/index.html",
]
risk_score = 47
rule_id = "fc7b9873-a9ed-43af-a9a2-c37660c4a211"
setup = """## Setup

This rule is designed for data generated by [Elastic Defend](https://www.elastic.co/security/endpoint-security), which provides native endpoint detection and response, along with event enrichments designed to work with our detection rules.

Setup instructions: https://ela.st/install-elastic-defend
"""
severity = "medium"
tags = [
    "Domain: Endpoint",
    "OS: Windows",
    "OS: Linux",
    "OS: macOS",
    "Use Case: Threat Detection",
    "Tactic: Initial Access",
    "Tactic: Execution",
    "Tactic: Discovery",
    "Data Source: Elastic Defend",
    "Resources: Investigation Guide",
    "Noise: High",
    "Threat: Vulnerability Exploit",
    "Performance: Normal",
    "Rule Type: ES|QL",
    "Platform: Windows",
    "Platform: Linux",
    "Platform: macOS",
    "Vuln: CVE-2023-48022",
]
timestamp_override = "event.ingested"
type = "esql"

query = '''
FROM logs-endpoint.events.process-* METADATA _id, _version, _index
| WHERE KQL("""event.category:process and event.type:start""")
  AND TO_LOWER(process.parent.name) LIKE "python*"
  AND (
    TO_LOWER(process.parent.command_line) LIKE """*\\ray\\_private\\workers\\default_worker.py*"""
    OR TO_LOWER(process.parent.command_line) LIKE "*/ray/_private/workers/default_worker.py*"
  )
  AND (
    TO_LOWER(process.name) IN (
      "cmd.exe", "powershell.exe", "pwsh.exe",
      "bash", "bash.exe", "sh", "zsh", "dash",
      "curl", "curl.exe", "wget", "wget.exe",
      "whoami", "whoami.exe", "id", "uname", "hostname", "hostname.exe"
    )
    // Direct python -c child of the worker.
    OR (
      TO_LOWER(process.name) LIKE "python*"
      AND MV_CONTAINS(process.args, "-c")
    )
  )
| KEEP
    _id,
    _version,
    _index,
    data_stream.namespace,
    @timestamp,
    host.name,
    host.id,
    host.os.type,
    agent.id,
    user.name,
    user.domain,
    process.name,
    process.executable,
    process.command_line,
    process.args,
    process.parent.name,
    process.parent.executable,
    process.parent.command_line,
    process.parent.entity_id
'''

[rule.investigation_fields]
field_names = [
    "host.name",
	"user.name",
    "process.parent.name",
    "process.parent.executable",
    "process.parent.command_line",
    "process.name",
    "process.executable",
    "process.command_line",
]

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1190"
name = "Exploit Public-Facing Application"
reference = "https://attack.mitre.org/techniques/T1190/"

[rule.threat.tactic]
id = "TA0001"
name = "Initial Access"
reference = "https://attack.mitre.org/tactics/TA0001/"

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1059"
name = "Command and Scripting Interpreter"
reference = "https://attack.mitre.org/techniques/T1059/"

[[rule.threat.technique.subtechnique]]
id = "T1059.001"
name = "PowerShell"
reference = "https://attack.mitre.org/techniques/T1059/001/"

[[rule.threat.technique.subtechnique]]
id = "T1059.003"
name = "Windows Command Shell"
reference = "https://attack.mitre.org/techniques/T1059/003/"

[[rule.threat.technique.subtechnique]]
id = "T1059.004"
name = "Unix Shell"
reference = "https://attack.mitre.org/techniques/T1059/004/"

[[rule.threat.technique.subtechnique]]
id = "T1059.006"
name = "Python"
reference = "https://attack.mitre.org/techniques/T1059/006/"

[rule.threat.tactic]
id = "TA0002"
name = "Execution"
reference = "https://attack.mitre.org/tactics/TA0002/"

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1033"
name = "System Owner/User Discovery"
reference = "https://attack.mitre.org/techniques/T1033/"

[[rule.threat.technique]]
id = "T1082"
name = "System Information Discovery"
reference = "https://attack.mitre.org/techniques/T1082/"

[rule.threat.tactic]
id = "TA0007"
name = "Discovery"
reference = "https://attack.mitre.org/tactics/TA0007/"

[[rule.threat]]
framework = "MITRE ATT&CK"

[[rule.threat.technique]]
id = "T1105"
name = "Ingress Tool Transfer"
reference = "https://attack.mitre.org/techniques/T1105/"

[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"

Detection rules belong to the projects that publish them and remain under their own licenses. This site indexes and links to them; it claims no rights in them.