pwru (packet, where are you?)

pwru is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities. It allows fine-grained introspection of kernel state to facilitate debugging network connectivity issues.

The following example shows where the packets of a curl request are dropped after installing an IP tables rule:

Running

Requirements

pwru requires >= 5.3 kernel to run. For --output-skb >= 5.9 kernel is required. For --backend=kprobe-multi >= 5.18 kernel is required.

pwru optionally requires debugfs. It has to be mounted in /sys/kernel/debug. In case the folder is empty, it can be mounted with:

mount -t debugfs none /sys/kernel/debug

The following kernel configuration is required.

Option	Backend	Note
CONFIG_DEBUG_INFO_BTF=y	both	available since >= 5.3
CONFIG_KPROBES=y	both
CONFIG_PERF_EVENTS=y	both
CONFIG_BPF=y	both
CONFIG_BPF_SYSCALL=y	both
CONFIG_FUNCTION_TRACER=y	kprobe-multi	/sys/kernel/debug/tracing/available_filter_functions
CONFIG_FPROBE=y	kprobe-multi	available since >= 5.18

You can use zgrep $OPTION /proc/config.gz to validate whether option is enabled.

Downloading

You can download the statically linked executable for x86_64 and arm64 from the release page.

Usage

$ ./pwru --help

Usage: ./pwru [options] [pcap-filter]
    Available pcap-filter: see "man 7 pcap-filter"
    Available options:
      --all-kmods                      attach to all available kernel modules
      --backend string                 Tracing backend('kprobe', 'kprobe-multi'). Will auto-detect if not specified.
      --filter-func string             filter kernel functions to be probed by name (exact match, supports RE2 regular expression)
      --filter-ifname string           filter skb ifname in --filter-netns (if not specified, use current netns)
      --filter-kprobe-batch uint       batch size for kprobe attaching/detaching (default 10)
      --filter-mark mark[/mask]        filter skb mark (format: mark[/mask], e.g., 0xa00/0xf00) (default 0x0)
      --filter-netns string            filter netns ("/proc/<pid>/ns/net", "inode:<inode>")
      --filter-non-skb-funcs strings   filter non-skb kernel functions to be probed (--filter-track-skb-by-stackid will be enabled)
      --filter-trace-tc                trace TC bpf progs
      --filter-trace-xdp               trace XDP bpf progs
      --filter-track-bpf-helpers       trace BPF helper functions
      --filter-track-skb               trace a packet even if it does not match given filters (e.g., after NAT or tunnel decapsulation)
      --filter-track-skb-by-stackid    trace a packet even after it is kfreed (e.g., traffic going through bridge)
      --filter-tunnel-pcap-l2 string   pcap expression for vxlan/geneve tunnel (l2)
      --filter-tunnel-pcap-l3 string   pcap expression for vxlan/geneve tunnel (l3)
  -h, --help                           display this message and exit
      --kernel-btf string              specify kernel BTF file
      --kmods strings                  list of kernel modules names to attach to
      --output-caller                  print caller function name
      --output-file string             write traces to file (rotates at 100MB by default)
      --output-file-max-size int       max size in MB per file before rotation (default 100)
      --output-file-max-age int        max age in days to keep rotated files (default 0, no limit)
      --output-file-max-backups int    max number of rotated files to keep (default 0, no limit)
      --output-file-compress           compress rotated files with gzip (default false)
      --output-json                    output traces in JSON format
      --output-limit-lines uint        exit the program after the number of events has been received/printed
      --output-meta                    print skb metadata (default true)
      --output-skb                     print skb
      --output-skb-cb                  print skb->cb
      --output-skb-metadata strings    print skb metadata (e.g., "skb->mark", "skb->hash"), 4 at most
      --output-skb-shared-info         print skb shared info
      --output-stack                   print stack
      --output-tcp-flags               print TCP flags
      --output-tunnel                  print encapsulated tunnel header data
      --output-tuple                   print L4 tuple (default true)
      --output-xdp-metadata strings    print xdp metadata (e.g., "xdp->rxq->queue_index"), 4 at most
      --set-percpu-buf uint32          set the size of buffers to print skb data (used by --output-skb and --output-skb-shared-info) (default 4096)
      --timestamp string               print timestamp per skb ("current", "relative", "absolute", "none") (default "none")
      --version                        show pwru version and exit

The --filter-func switch does an exact match on function names i.e. --filter-func=foo only matches foo(); for a wildcarded match, try --filter-func=".*foo.*" instead.

Running with Docker

Docker images for pwru are published at https://hub.docker.com/r/cilium/pwru.

An example how to run pwru with Docker:

docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/:/sys/kernel/debug/ cilium/pwru pwru --output-tuple 'host 1.1.1.1'

Running on Kubernetes

The following example shows how to run pwru on a given node:

#!/usr/bin/env bash
NODE=kind-control-plane
PWRU_ARGS="--output-tuple 'host 1.1.1.1'"

trap " kubectl delete --wait=false pod pwru " EXIT

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: pwru
spec:
  nodeSelector:
    kubernetes.io/hostname: ${NODE}
  containers:
  - image: docker.io/cilium/pwru:latest
    name: pwru
    volumeMounts:
    - mountPath: /sys/kernel/debug
      name: sys-kernel-debug
    securityContext:
      privileged: true
    command: ["/bin/sh"]
    args: ["-c", "pwru ${PWRU_ARGS}"]
  volumes:
  - name: sys-kernel-debug
    hostPath:
      path: /sys/kernel/debug
      type: DirectoryOrCreate
  hostNetwork: true
  hostPID: true
EOF

kubectl wait pod pwru --for condition=Ready --timeout=90s
kubectl logs -f pwru

Also see contrib/scripts/pwru-on-k8s-node for a reusable version of this script.

Distro packages

pwru is available in some distro packages. GH#89 tracks any packaging work and progress.

Running on Vagrant

See docs/vagrant.md

Developing

Dependencies

• Go >= 1.16
• LLVM/clang >= 12
• Bison
• Lex/Flex >= 2.5.31
• libpcap(-dev)

Building

make

Alternatively, you can build in the Docker container:

make release

Sign-off

Ensure that all commits have Developer Certificate of Origin by adding a Signed-off-by line to your commit messages.

Community

Join the #pwru Slack channel to chat with developers, maintainers, and other users. This is a good first stop to ask questions and share your experiences.

Logo Credits

The detective gopher is based on the Go gopher designed by Renee French.