SystemTap is a GPLv2 license
system wide tool that allows you to gather
tracing and profiling data from a running Linux system.
Understanding
systemtap
Systemtap connects to the Linux kernel and monitors for
available events,
which are exposed through the
kprobes kernel facility
Based on an event, the kernel can run a handler
which is executed as a sub routine
The event and handler together are refers as probe
Stap is doing its work by running scripts. To work, these scripts you need to compile the
stap scripts in the kernel and start it as a kernel module. The kernel module next will do its work and after doing so it will unload
The
stap command will run the scripts and do the compilation
https://www.golinuxcloud.com/systemtap-tutorial-linux-example/
- Linux introspection and SystemTap
SystemTap is a dynamic method of monitoring and tracing the operation of a running Linux kernel.
An interface and language for dynamic kernel analysis
Kernel tracing
SystemTap is
similar to an older technology called
DTrace, which originated in the Sun Solaris operating system. Within
DTrace, developers can write scripts in the D programming language (a subset of the C language but
modified to support trace-specific behaviors). A
DTrace script contains
a number of probes and associated actions that occur when the probe "fires." For example, a probe can represent something as simple as invoking a system call or more complicated interactions such as a particular line of code being executed.
https://www.ibm.com/developerworks/linux/library/l-systemtap/
- SystemTap provides a command line interface and a scripting language to examine the activities of a running Linux system, particularly the kernel, in fine detail. SystemTap scripts are written in the SystemTap scripting language, are then compiled to C-code kernel modules and inserted into the kernel.
https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.tuning.systemtap.html
OProfile is a low overhead, system-wide performance monitoring tool provided by the
oprofile package.
It uses the performance monitoring hardware on the processor to retrieve information about the kernel and executables on the system, such as when memory
is referenced, the number of second-level cache requests, and the number of hardware interrupts received.
OProfile is also able to profile applications that run in a Java Virtual Machine (JVM)
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/oprofile
- Learn eBPF Tracing: Tutorial and Examples
It can be used for many things: network performance, firewalls, security, tracing, and device drivers.
The term tracing refers to performance analysis and observability tools that can produce per-event info.
What is
eBPF,
bcc,
bpftrace, and
iovisor?
eBPF does to Linux what JavaScript does to HTML. (Sort of.) So instead of a static HTML website, JavaScript lets you define mini programs that run on events like mouse clicks,
which are run in a safe virtual machine in the browser. And with
eBPF, instead of a fixed kernel, you can now write mini programs that run on events like
disk I/O, which
are run in a safe virtual machine in the kernel
http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
- 5.4. Performance Counters for Linux (PCL) Tools and perf
Performance Counters for Linux (PCL) is a new kernel-based subsystem that provides a framework for collecting and analyzing performance data.
The PCL subsystem can be used to measure hardware events, including retired instructions and processor clock cycles. It can also measure software events, including major page faults and context switches. For example, PCL counters can compute the Instructions Per Clock (IPC) from a process's counts of instructions retired and processor clock cycles. A low IPC ratio
indicates the code
makes poor use of the CPU.
Other hardware events can also be used to diagnose poor CPU performance.
Both
OProfile and Performance Counters for Linux (PCL) use the same hardware Performance Monitoring Unit (PMU).
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/perf
- Perf is a profiler tool for Linux 2.6+ based systems that abstracts away CPU hardware differences in Linux performance measurements and presents a simple commandline interface.
https://www.dedoimedo.com/computers/linux-perf.html
the perf Linux profiler
Performance Counters for Linux (PCL)
perf_events is an event-oriented observability
tool,which can help you solve advanced performance and troubleshooting functions
perf_events is part of the Linux kernel
While it uses many Linux tracing features, some
are not yet exposed via the perf command, and need to
be used via the
ftrace interface instead
Why is the kernel on-CPU so much? What code-paths?
Which code-paths are causing CPU level 2 cache misses?
Are the
CPUs stalled on memory I/O?
Which code-paths are allocating memory, and how much?
What is triggering TCP retransmits?
Is a certain kernel function being called, and how often?
What reasons are threads leaving the CPU?
http://www.brendangregg.com/perf.html#Tracepoints
Valgrind is an instrumentation framework for building dynamic analysis tools that can
be used to profile applications
in detail. The default installation
alrready provides five standard tools.
Valgrind tools are
generally used to investigate memory management and threading problems.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/profiling#valgrind
The
ftrace framework provides users with several tracing capabilities, accessible through an interface much simpler than
SystemTap's. This framework uses a set of virtual files in the
debugfs file system; these files enable specific tracers. The
ftrace function
tracer outputs each function called in the kernel in real time; other tracers within the
ftrace framework can also
be used to analyze
wakeup latency, task switches, kernel events
, and the like.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/ftrace
