Kubernetes Abuse of Secret by Unusual Location

name: Kubernetes Abuse of Secret by Unusual Location
id: 40a064c1-4ec1-4381-9e35-61192ba8ef82
version: 10
creation_date: '2023-12-20'
modification_date: '2026-05-13'
author: Patrick Bareiss, Splunk
status: production
type: Anomaly
description: The following analytic detects unauthorized access or misuse of Kubernetes Secrets from unusual locations. It leverages Kubernetes Audit logs to identify anomalies in access patterns by analyzing the source of requests by country. This activity is significant for a SOC as Kubernetes Secrets store sensitive information like passwords, OAuth tokens, and SSH keys, making them critical assets. If confirmed malicious, this behavior could indicate an attacker attempting to exfiltrate or misuse these secrets, potentially leading to unauthorized access to sensitive systems or data.
data_source:
    - Kubernetes Audit
search: |-
    `kube_audit` objectRef.resource=secrets verb=get
      | iplocation sourceIPs{}
      | fillnull
      | search NOT `kube_allowed_locations`
      | stats count
        BY objectRef.name objectRef.namespace objectRef.resource
           requestReceivedTimestamp requestURI responseStatus.code
           sourceIPs{} stage user.groups{}
           user.uid user.username userAgent
           verb City Country
      | rename sourceIPs{} as src_ip, user.username as user
      | `kubernetes_abuse_of_secret_by_unusual_location_filter`
how_to_implement: The detection is based on data that originates from Kubernetes Audit logs. Ensure that audit logging is enabled in your Kubernetes cluster. Kubernetes audit logs provide a record of the requests made to the Kubernetes API server, which is crucial for monitoring and detecting suspicious activities. Configure the audit policy in Kubernetes to determine what kind of activities are logged. This is done by creating an Audit Policy and providing it to the API server. Use the Splunk OpenTelemetry Collector for Kubernetes to collect the logs. This doc will describe how to collect the audit log file https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/migration-from-sck.md. When you want to use this detection with AWS EKS, you need to enable EKS control plane logging https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html. Then you can collect the logs from Cloudwatch using the AWS TA https://splunk.github.io/splunk-add-on-for-amazon-web-services/CloudWatchLogs/.
known_false_positives: No false positives have been identified at this time.
references:
    - https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
drilldown_searches:
    - name: View the detection results for - "$user$"
      search: '%original_detection_search% | search  user = "$user$"'
      earliest_offset: $info_min_time$
      latest_offset: $info_max_time$
    - name: View risk events for the last 7 days for - "$user$"
      search: '| from datamodel Risk.All_Risk | search normalized_risk_object IN ("$user$") | stats count min(_time) as firstTime max(_time) as lastTime values(search_name) as "Search Name" values(risk_message) as "Risk Message" values(analyticstories) as "Analytic Stories" values(annotations._all) as "Annotations" values(annotations.mitre_attack.mitre_tactic) as "ATT&CK Tactics" by normalized_risk_object | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`'
      earliest_offset: 7d
      latest_offset: "0"
intermediate_findings:
    entities:
        - field: user
          type: user
          score: 20
          message: Access of Kubernetes secret $objectRef.name$ from unusual location $Country$ by $user$
threat_objects:
    - field: src_ip
      type: ip_address
analytic_story:
    - Kubernetes Security
asset_type: Kubernetes
mitre_attack_id:
    - T1552.007
product:
    - Splunk Enterprise
    - Splunk Enterprise Security
    - Splunk Cloud
category: cloud
security_domain: network
tests:
    - name: True Positive Test
      attack_data:
        - data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1552.007/kube_audit_get_secret/kube_audit_get_secret.json
          sourcetype: _json
          source: kubernetes
      test_type: unit