In this chapter, we will look at SELinux debugging through the following recipes:
Identifying whether SELinux is to blame
Analyzing SELINUX_ERR messages
Logging positive policy decisions
Looking through SELinux constraints
Ensuring an SELinux rule is never allowed
Using strace to clarify permission issues
Using strace against daemons
Auditing system behavior
On an SELinux-enabled system, the SELinux policy defines how applications should behave. Any change in behavior might trigger SELinux denials for certain actions of that application. As a result, end users can notice unexpected permission issues or erratic application behavior.
Troubleshooting such situations is usually done through analysis of the AVC events. Many resources already cover AVC events in great detail. The basic premise is that an AVC event uses a set of key-value pairs, as follows:
type=AVC msg=audit(1369306885.125:4702304): avc: denied { append } for pid=1787 comm="syslog-ng" name="oracle_audit.log" dev=dm-18 ino=65 scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=fileIn this example, we can deduce the following from the AVC event:
The event is a denial (avc: denied)
The operation that was denied is appending to a file ({ append } … tclass=file)
The process that tried to append to the file has PID 1787 and name syslog-ng (pid=1787...
Before blaming the SELinux subsystem and policies for a problem, it is important to verify whether SELinux is to blame at all. Too often, hours of troubleshooting are put in analyzing the SELinux policies and subsystem only to find out that the problem also persists when SELinux is not enabled.
In order to be confident that SELinux is (or isn't) to blame, the following set of steps can be taken:
Is it possible to get more information through the application's internal debugging system? Consider the following instance:
~# puppet master Error: Could not find class puppet::agent for foo.bar on node foo.bar ~# puppet master --debug --no-daemonize --verbose
Is an AVC denial related to the problem shown in the audit logs? If not, try disabling the dontaudit rules and try again:
~# semodule -DB
Is the application that gives problems SELinux-aware? Most SELinux-aware applications are linked with the libselinux.so library, so we can verify whether...
When the SELinux subsystem is asked to perform an invalid SELinux-specific operation, it will log this through the audit subsystem using the SELINUX_ERR message type.
Make sure that the audit subsystem is up and running as we will be using the ausearch application to (re)view audit events:
~# service auditd start
Analyzing SELINUX_ERR messages is done by viewing the entry in the audit logs and understanding the individual fields; this is done by completing the following steps:
Note the current date/time, or reload the SELinux policy, to have a clear point in the audit logs from where to look:
~# semodule -R
Trigger the behavior in the application.
Ask the audit subsystem to show the last events of the SELINUX_ERR and MAC_POLICY_LOAD types:
~# ausearch -m SELINUX_ERR,MAC_POLICY_LOAD -ts recent
Look at the beginning of the message to find out what problematic situation SELinux is informing us about.
On some occasions, the system performs actions that the administrator might not expect, but which are allowed by the SELinux policy, making it harder to debug potential problems. An application might be SELinux-aware, causing its own behavior to depend on the SELinux policy, without actually using the SELinux subsystem to enforce access. The SELinux policy might also be configured to behave differently than expected.
In such situations, it might be important to have SELinux log activities that were actually allowed rather than denied; for instance, logging domain transitions to make sure that a transition has indeed occurred.
In order to have domain transitions logged, create an SELinux policy by performing the following steps:
Identify the source and target domains to look out for.
Create an SELinux policy that calls the auditallow statement on the access vector we want to log:
auditallow initrc_t postgresql_t:process transition;
Build and load the...
Some denials are caused by SELinux constraints—additional restrictions imposed by the SELinux policy that are not purely based on the SELinux types, but also on the SELinux role and SELinux user. This is often not clear from the denial.
The audit2why application helps in informing developers that a denial came from a constraint violation:
~# ausearch -m avc -ts recent | grep type=AVC | audit2why type=AVC msg=audit(1401134596.932:62843): avc: denied { search } for pid=19384 comm="mount.nfs4" scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=dir Was caused by: Policy constraint violation. May require adding a type attribute to the domain or type to satisfy the constraint. Constraints are defined in the policy sources in policy/constraints (general), policy/mcs (MCS), and policy/mls (MLS).
This is, however, not always the case, so we need to find a way to investigate...
It is possible to include statements in the SELinux policy that ensure that a particular access vector cannot be allowed, not even by enhancing the SELinux policy later. This is done with the neverallow statement.
To include the neverallow statements in the policy and enforce them, go through the following steps:
In /etc/selinux/semanage.conf, enable support for the neverallow statements by setting the expand-check variable to 1:
expand-check=1
Create an SELinux policy in which the access vectors that should be explicitly forbidden are listed. Consider the following instance:
neverallow user_t system_mail_t:process transition;
Build and load the policy.
Generate another policy that will allow the statement and attempt to load it:
~$ semodule -i mytest.pp libsepol.check_assertion_helper: neverallow violated by allow user_t system_mail_t:process { transition }; libsemanage.semanage_expand_sandbox: Expand module failed semodule: Failed!
The strace application is a popular debugging application on Linux systems. It allows developers and administrators to look at various system calls made by an application. As SELinux often has access controls on specific system calls, using strace can prove to be very useful in debugging permission issues.
To properly use strace, follow the next set of steps:
Enable the allow_ptrace Boolean:
~# setsebool allow_ptrace on
Run the application with strace:
~$ strace -o strace.log -f -s 256 tmux
In the resulting logfile, look for the error message that needs to be debugged.
The allow_ptrace Boolean (on some distributions, the inverse Boolean called deny_ptrace is available) needs to be toggled so that the domain that calls strace can use ptrace (the method that strace uses to view system calls) against the target domain. As the ptrace method can be a security concern (it allows reading target process' memory, for instance), it is...
The strace application not only makes sense for command-line applications but also for daemons. A popular approach to debugging daemons is to start them from the command line, possibly with a specific debug flag, so that the daemon doesn't detach and run in the background. However, this is often not possible on SELinux: the policy will not allow the daemon to run as a command-line foreground process.
The approach to use strace against daemons is similar as with command lines, focusing on the process ID rather than the command:
Find out what the process ID of the daemon is:
~$ pidof postgres 2557
Use strace to attach to the running process:
~$ strace -o strace.log -f -s 256 -p 2557
Specify which system calls to watch out for. For instance, permission issues while binding or connecting to ports or sockets can be filtered as follows:
~$ strace -e poll,select,connect,recvfrom,sendto -o strace.log -f -s 256 -p 2557
Press Ctrl + C to interrupt the strace...
Another approach to debugging application behavior is through Linux auditing, especially when it is not clear which process is responsible for performing a specific action, as this might make SELinux development a lot more difficult. When developers do not know which domain(s) they need to update privileges for, or do not know how exactly a resource is created, then the Linux audit subsystem can help.
With the Linux auditing subsystem, administrators can enable rules to log activities. In the audit log, the SELinux context of the subject (process) is shown as well, allowing SELinux developers to properly identify the domain to work with.
Let's look at how we can ask the Linux audit subsystem which process is responsible for creating a particular directory in a user's home directory through the following steps:
As the root Linux user (and in an SELinux role with sufficient privileges), tell the audit subsystem to log all write- and attribute-changing operations...
© 2014 Packt Publishing Limited All Rights Reserved
