Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Thursday, June 6, 2019

Linux Introspection & Profiling


  • SystemTap

SystemTap is a GPLv2 license system wide tool that allows you to gather tracing and profiling data from a running Linux system.
Understanding systemtap

    Systemtap connects to the Linux kernel and monitors for available events, which are exposed through the kprobes kernel facility
    Based on an event, the kernel can run a handler which is executed as a sub routine
    The event and handler together are refers as probe
    Stap is doing its work by running scripts. To work, these scripts you need to compile the stap scripts in the kernel and start it as a kernel module. The kernel module next will do its work and after doing so it will unload
    The stap command will run the scripts and do the compilation

https://www.golinuxcloud.com/systemtap-tutorial-linux-example/


  • Linux introspection and SystemTap


SystemTap is a dynamic method of monitoring and tracing the operation of a running Linux kernel.
An interface and language for dynamic kernel analysis

Kernel tracing
SystemTap is similar to an older technology called DTrace, which originated in the Sun Solaris operating system. Within DTrace, developers can write scripts in the D programming language (a subset of the C language but modified to support trace-specific behaviors). A DTrace script contains a number of probes and associated actions that occur when the probe "fires." For example, a probe can represent something as simple as invoking a system call or more complicated interactions such as a particular line of code being executed.

https://www.ibm.com/developerworks/linux/library/l-systemtap/

  • SystemTap provides a command line interface and a scripting language to examine the activities of a running Linux system, particularly the kernel, in fine detail. SystemTap scripts are written in the SystemTap scripting language, are then compiled to C-code kernel modules and inserted into the kernel.

https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.tuning.systemtap.html

  • OProfile

OProfile is a low overhead, system-wide performance monitoring tool provided by the oprofile package.
It uses the performance monitoring hardware on the processor to retrieve information about the kernel and executables on the system, such as when memory is referenced, the number of second-level cache requests, and the number of hardware interrupts received. OProfile is also able to profile applications that run in a Java Virtual Machine (JVM)
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/oprofile

  • Learn eBPF Tracing: Tutorial and Examples

It can be used for many things: network performance, firewalls, security, tracing, and device drivers.
The term tracing refers to performance analysis and observability tools that can produce per-event info.

What is eBPF, bcc, bpftrace, and iovisor?
eBPF does to Linux what JavaScript does to HTML. (Sort of.) So instead of a static HTML website, JavaScript lets you define mini programs that run on events like mouse clicks, which are run in a safe virtual machine in the browser. And with eBPF, instead of a fixed kernel, you can now write mini programs that run on events like disk I/O, which are run in a safe virtual machine in the kernel
http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html

  • 5.4. Performance Counters for Linux (PCL) Tools and perf

Performance Counters for Linux (PCL) is a new kernel-based subsystem that provides a framework for collecting and analyzing performance data.

The PCL subsystem can be used to measure hardware events, including retired instructions and processor clock cycles. It can also measure software events, including major page faults and context switches. For example, PCL counters can compute the Instructions Per Clock (IPC) from a process's counts of instructions retired and processor clock cycles. A low IPC ratio indicates the code makes poor use of the CPU. Other hardware events can also be used to diagnose poor CPU performance.

Both OProfile and Performance Counters for Linux (PCL) use the same hardware Performance Monitoring Unit (PMU).

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/perf

  • Perf is a profiler tool for Linux 2.6+ based systems that abstracts away CPU hardware differences in Linux performance measurements and presents a simple commandline interface.

https://www.dedoimedo.com/computers/linux-perf.html

  • perf Examples

the perf Linux profiler
Performance Counters for Linux (PCL)
perf_events is an event-oriented observability tool,which can help you solve advanced performance and troubleshooting functions
perf_events is part of the Linux kernel
While it uses many Linux tracing features, some are not yet exposed via the perf command, and need to be used via the ftrace interface instead

    Why is the kernel on-CPU so much? What code-paths?
    Which code-paths are causing CPU level 2 cache misses?
    Are the CPUs stalled on memory I/O?
    Which code-paths are allocating memory, and how much?
    What is triggering TCP retransmits?
    Is a certain kernel function being called, and how often?
    What reasons are threads leaving the CPU?


http://www.brendangregg.com/perf.html#Tracepoints

  • Valgrind

Valgrind is an instrumentation framework for building dynamic analysis tools that can be used to profile applications in detail. The default installation alrready provides five standard tools. Valgrind tools are generally used to investigate memory management and threading problems.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/profiling#valgrind


  • FTRACE

The ftrace framework provides users with several tracing capabilities, accessible through an interface much simpler than SystemTap's. This framework uses a set of virtual files in the debugfs file system; these files enable specific tracers. The ftrace function tracer outputs each function called in the kernel in real time; other tracers within the ftrace framework can also be used to analyze wakeup latency, task switches, kernel events, and the like.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/developer_guide/ftrace

Tuesday, May 28, 2019

tty

  • In Linux, everything is a file
any hardware device connected to the system is represented as a special file.
a terminal is also represented as a file
The tty command basically prints the file name of the terminal connected to standard input.

What is tty?
tty is short for teletype
popularly known as terminal
basically a device
allows you to interact with the system by passing on the data (you input) to the system, and displaying the output produced by the system.
ttys can be of different types
graphical consoles that you can access with the Ctrl+Alt+Fn key combination
terminal emulators like Gnome terminal that run inside an X session

 the meaning of each exit status:
    0 if standard input is a terminal
    1 if standard input is not a terminal
    2 if given incorrect arguments
    3 if a write error occurs

In unix terminology, the short answer is that

    terminal = tty = text input/output environment
    console = physical terminal
    shell = command line interpreter

https://www.howtoforge.com/linux-tty-command/

Wednesday, March 6, 2019

the directories dot and dot-dot

Verify on your system that the directories dot and dot-dot are not the same, except in the root directory

the root directory
vagrant@control01:/$ ls -ail
total 97
      2 drwxr-xr-x  24 root    root     4096 Feb 21 20:28 .
      2 drwxr-xr-x  24 root    root     4096 Feb 21 20:28 ..


the directories dot and dot-dot are not the same
vagrant@control01:~$ ls -ail
total 64
3145730 drwxr-xr-x 5 vagrant vagrant 4096 Feb 22 05:08 .
3145729 drwxr-xr-x 3 root    root    4096 Aug 24 08:48 ..

3145730
The first is the directory itself (so it contain some data about directory permissions)
3145729
the second is the parent directory


The UNIX System guarantees that every process has a unique numeric identifier called the process ID. The process ID is always a non-negative integer


Directories cannot have hard links
Unix permits you to give files many names ("links"),but, not directories
You are not allowed to create a hard link to a directory
Each directory inode is allowed to appear once in exactly one parent directory and no more
every sub-directory only has one parent directory, and that means the special name ".." (dot dot) in a sub-directory always refers unambiguously to its unique parent directory.

Each subdirectory adds one to the link count

A directory may have sub-directories. Since the special name ".." (dot dot) in every one of those sub-directories is a link to the inode number of the parent directory, the link count of the parent directory is increased by one for every sub-directory the parent contains. (Remember - the link count counts how many name-inode maps point to this inode, and that includes the special "." (dot) and ".." (dot dot) name-inode maps!) A directory with five sub-directories will show a link count of 2+5=7.


  • What is Sticky Bit?

The sticky bit is used to indicate special permissions for files and directories.
If a directory with sticky bit enabled will restrict deletion of the file inside it.
It can be removed by root, owner of the file or who have to write permission on it.
This is useful for publically accessible directories like /tmp.

What is SUID (setuid)?
If SUID bit is set on a file and a user executed it. The process will have the same rights as the owner of the file being executed.
For example: passwd command have SUID bit enabled. When a normal user changes his password this script update few system files like /etc/passwd and /etc/shadow which can’t be updated by non-root account. So that passwd command process always run with root user rights.

$ ls -lrt /usr/bin/passwd
-rwsr-xr-x 1 root root 54256 May 16  2017 /usr/bin/passwd
$ ls -lrt /etc/shadow
-rw-r----- 1 root shadow 994 Aug 24  2018 /etc/shadow
$ ls -lrt /bin/su
-rwsr-xr-x 1 root root 40128 May 16  2017 /bin/su

https://tecadmin.net/understanding-sticky-bit-suid-and-sgid-in-linux/

Monday, May 21, 2018

Linux Security

  • Linux server security: Three steps to secure each system

Auditing
To audit a Linux system we use our open source tool Lynis and perform a system scan. It runs on almost all Unix and Linux based systems and only requires a shell and root permissions. After the scan all findings will be reported and additional information will be stored in the log files (/var/log/lynis.log). Also a hardening index will be displayed, to give the auditor a first impression on how well the system is hardened.

Hardening
After running Lynis it’s time to deal with the findings.

Compliance
maintain the effect or earlier security efforts, it’s important to keep measuring your security level and compare them with baselines.Every system should be checked on a regular base and deviations to your standard should be detected as soon as possible. By determining the risk level of this deviations, it will be much easier to take an appropriate action or implement different security measures.
https://linux-audit.com/linux-server-security-three-steps-to-secure-each-system/


  • the su command requires you to enter the password of the target user.

There's another way to switch to the root user: run the 'su -' command:

Now, what's the difference between 'su' and 'su -' ? Well, the former keeps the environment of the old/original user even after the switch to root has been made, while the latter creates a new environment (as dictated by the ~/.bashrc of the root user), similar to the case when you explicitly log in as root user from the log-in screen.
The optional argument - may be used to provide an environment similar to what the user would expect had the user logged in directly.

So, you'll agree that logging in with 'su -' makes more sense. But as the 'su' command also exists, one might wonder when that's useful.
the benefits and pitfalls of the 'su' command

It sometimes can be advantageous for a system administrator to use the shell account of an ordinary user rather than its own. In particular, occasionally the most efficient way to solve a user's problem is to log into that user's account in order to reproduce or debug the problem.
However, in many situations it is not desirable, or it can even be dangerous, for the root user to be operating from an ordinary user's shell account and with that account's environmental variables rather than from its own. While inadvertently using an ordinary user's shell account, root could install a program or make other changes to the system that would not have the same result as if they were made while using the root account. For instance, a program could be installed that could give the ordinary user power to accidentally damage the system or gain unauthorized access to certain data.

-, -l, --login
Provide an environment similar to what the user would expect had the user logged in directly.

When - is used, it must be specified as the last su option. The other forms (-l and --login) do not have this restriction.

su -c
It lets you provide a command that you want to run after switching to the target user.


Sudo vs Su
The primary difference between the two is the password they require: while 'sudo' requires current user's password, 'su' requires you to enter the root user password
Quite clearly, 'sudo' is a better alternative between the two as far as security is concerned.
For example, consider the case of computer being used by multiple users who also require root access. Using 'su' in such a scenario means sharing the root password with all of them

Moreover, in case you want to revoke the superuser/root access of a particular user, the only way is to change the root password and then redistribute the new root password among all the other users.

With Sudo, on the other hand, you can handle both these scenarios effortlessly. Given that 'sudo' requires users to enter their own password, you don't need to share the root password will all the users in the first place. And to stop a particular user from accessing root privileges, all you have to do is to tweak the corresponding entry in the 'sudoers' file.

The other difference between the two commands is in their default behavior. While 'sudo' only allows you to run a single command with elevated privileges, the 'su' command launches a new shell, allowing you to run as many commands as you want with root privileges until you explicitly exit that sell.

So the default behavior of the 'su' command is potentially dangerous given the possibility that the user can forget the fact that they are working as root

Logging
Although commands run through 'sudo' are executed as the target user (which is 'root' by default), they are tagged with the sudoer's user-name. But in case of 'su', it's not possible to directly trace what a user did after they su'd to the root account.

Flexibility
The 'sudo' command is far more flexible in that you can even limit the commands that you want the sudo-ers to have access to. In other words, users with access to 'sudo' can only be given access to commands that are required for their job. However, with 'su' that's not possible - either you have the privilege to do everything or nothing.

Sudo su
some Linux distributions - like Ubuntu - disable the root user account by default.

In case you want to enable the root account on your system
sudo passwd root

https://www.howtoforge.com/tutorial/sudo-vs-su/

  • If sudo has been configured to allow the user to run root's shell, the user can run sudo -s or sudo -i to mimic su or su -l, respectively, and supply his own password or no password rather than root's password. Similarly, sudo -u john -i mimics su -l john if you are allowed to run john's shell

The default behavior of su is to remain within the current directory and to maintain the environmental variables of the original user (rather than switch to those of the new user).

it is advisable that administrative users, as well as any other users that are authorized to use su (and it is suggested that there be very few, if any) acquire the habit of always running the su command with the -l/--login option. It has two effects:
     switches from the current directory to the home directory of the new user (e.g., to /root in the case of the root user) by logging in as that user
    changes the environmental variables to those of the new user as dictated by their ~/.bashrc. That is, the current directory and environment will be changed to what would be expected if the new user had actually logged on to a new session (rather than just taking over an existing session).

su and wheel
BSD su allows only members of the "wheel" group to assume root's identity by default. This is not the default behavior of GNU su, but this behavior can be mimicked using PAM. Uncomment the appropriate line in /etc/pam.d/su and /etc/pam.d/su-l:

https://wiki.archlinux.org/index.php/Su


  •  Configuring and Using Chroot Jails

 A chroot operation changes the apparent root directory for a running process and its children. It allows you to run a program with a root directory other than /. The program cannot see or access files outside the designated directory tree. Such an artificial root directory is called a chroot jail, and its purpose is to limit the directory access of a potential attacker. The chroot jail locks down a given process and any user ID that it is using so that all they see is the directory in which the process is running. To the process, it appears that the directory in which it is running is the root directory
 The chroot mechanism cannot defend against intentional tampering or low-level access to system devices by privileged users. For example, a chroot root user could create device nodes and mount file systems on them. A program can also break out of a chroot jail if it can gain root privilege and use chroot() to change its current working directory to the real root directory. For this reason, you should ensure that a chroot jail does not contain any setuid or setgid executables that are owned by root
 https://docs.oracle.com/cd/E37670_01/E36387/html/ol_cj_sec.html


  •  the chroot “Jail” we built to watch his activities

 With these holes he could often subvert the uucp and bin accounts in short order, and then root.
 We added a few fake services, and I wrote a script to scan the logs daily.

 FTP: The scanner produces a report of all login names that were attempted.
 It also reports the use of a tilde (a possible probe of an old FTP bug), all attempts to obtain FTP’s /etc/passwd and /etc/group files, and a list of all files stored in the
pub directory.

 Telnet/login:All login attempts are logged and reviewed daily

 Guest/visitor accounts:A public computer account is the first thing a cracker looks for.
 The cracker can also get a list of hosts trusted by this machine from the /etc/hosts.equiv and various personal .rhosts files

 Finger: Finger provides a lot of information useful to crackers: account names, when the account was last used, and a few things to try as passwords.

 http://www.cheswick.com/ches/papers/berferd.pdf

  • How do I change the root directory of a command?
How do I change the root directory of a process such as web-server using a chroot command to isolate file system?
How do I use a chroot to recover password or fix the damaged Linux/Unix based environment?
Each process/command on Linux and Unix-like system has current working directory called root directory of a process/command.
You can change the root directory of a command using chroot command, which ends up changing the root directory for both current running process and its children.
A process/command that is run in such a modified environment cannot access files outside the root directory. This modified environment is commonly known as “jailed directory” or “chroot jail”.
Only a privileged process and root user can use chroot command

    Privilege separation for unprivileged process such as Web-server or DNS server.
    Setting up a test environment.
    Run old programs or ABI in-compatibility programs without crashing application or system.
    System recovery.
    Reinstall the bootloader such as Grub or Lilo.
    Password recovery – Reset a forgotten password and more.

https://www.cyberciti.biz/faq/unix-linux-chroot-command-examples-usage-syntax/


  • Chroot is an operation that changes the apparent root directory for the current running process and their children. A program that is run in such a modified environment cannot access files and commands outside that environmental directory tree. This modified environment is called a chroot jail.
Changing root is commonly done for performing system maintenance on systems where booting and/or logging in is no longer possible. Common examples are:

    Reinstalling the bootloader.
    Rebuilding the initramfs image.
    Upgrading or downgrading packages.
    Resetting a forgotten password.
    Building packages in a clean chroot, see DeveloperWiki:Building in a Clean Chroot.
https://wiki.archlinux.org/index.php/change_root

A chroot is basically a special directory on your computer which prevents applications, if run from inside that directory, from accessing files outside the directory. In many ways, a chroot is like installing another operating system inside your existing operating system.
https://help.ubuntu.com/community/BasicChroot


  • ‘sudo‘ is a root binary setuid, which executes root commands on behalf of authorized users and the users need to enter their own password to execute system command followed by ‘sudo‘.

https://www.tecmint.com/su-vs-sudo-and-how-to-configure-sudo-in-linux/


  • Sudo (sometimes considered as short for Super-user do) is a program designed to let system administrators allow some users to execute some commands as root (or another user).

Auditing/logging: when a sudo command is executed, the original username and the command are logged.
For the reasons above, switching to root using sudo -i (or sudo su) is usually deprecated because it cancels the above features.
https://wiki.debian.org/sudo