Tinkering with SELinux in the trenches

Last week I came home only to find my computer in a very advanced state of disdain. Booting my Fedora 22 on fast SSD disk is a matter of 3-5 seconds on average. But this was not the case, the Plymouth boot screen froze at 100% and nothing happened from then on. Hitting the Esc key after like two minutes to see the systemd bootlog I discovered that each and every important service simply fails to start:

Failed to start message bus: Could not get UID and GID for username "dbus"  

But you could not see this in the giant bootlog spew, the only clues I had was merely just the last log screen:

audit: type=1131 audit(1443219137.710:298): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'  
Started D-Bus System Message Bus.  
systemd-logind.service has no holdoff time, scheduling restart  
dbus.service failed.  
Unit dbus.service entered failed state.  
dbus.service: main process exited, code=exited, status=1/FAILURE  
systemd-logind.service failed.  
Unit systemd-logind.service entered failed state.  
Failed to start Login Service.  
systemd-logind.service: main process exited, code=exited, status=1/FAILURE  

At first I was puzzled and thinking "What the hell is systemd doing?".

In case of emergency ...

After having restarted the computer a few times I figured that I need to bring in some heavy weaponry, restarted again and set boot parameters in Grub to boot in some form of recovery/single user mode.
First I tried appending init=/bin/sh to the boot prompt which seemed to just completely brick the system, but hey, my /root is on encrypted partition so I should have expected that.
Next in my bag of tricks was appending systemd.unit=rescue.target to the boot prompt, but that died with similar log message as above with the little exception that the first mentioned log message about bad uid/gid for the dbus user was now sitting close to the upper edge of the screen. My final attempt was systemd.unit=emergency.target but that ended up the same as the rescue.target.
Carefully re-reading several times in awe that user dbus could not be found in the system made me realize that only one thing I've recently toyed with can make the system behave in such a way that even root process can't access particular file.

Enter SELinux and Docker

SELinux was blocking access to /etc/passwd because it was labelled as svirt_sandbox_file_t, and not passwd_file_t. But how could that happen? I certainly don't go around and relabel my file system on a whim. People familiar with containers and Docker in particular know this label. If you use bind-mounts with docker run on a SELinux enabled system and set the mount options to :z or :Z, the on host file/directory will be relabeled to svirt_sandbox_file_t type, and that is the same type which all containers process are spawned with.
But that was just my hunch at the moment since I realized that as part of the HICA project I've certainly tested the io.hica.bind_users_and_groups injector, which bind mounts hosts users and groups database into the container, pretty simple and benign stuff at the first glance, but SELinux twisted it into deadly strike.

Fedora 22 LIVE i686 ... wait, what?

For cases like this I have always a live variant of the installed release available handy on a USB thumb drive. What I didn't notice until then was that it's actually 32-bit only.
Not so terrible problem in and of itself, but it made the system recovery really spicy along the way.
It was like 2:00am at this moment and I was already tired, so I figured that I'd just sit back, relax and click the crap out of it through GNOME desktop.
Mounting the encrypted /root partition at /mnt/myroot went like breeze with Disks, after that I realized that I need to drop to the terminal and focus again, brrr. Now comes the funny part.

$ restorecon -R /mnt/myroot

Failed horribly because restorecon expects absolute file system paths to correspond with the layout of the file system. It knows how to label /etc/passwd but has no clue about /mnt/myroot/etc/passwd. Pivoting a file system path can be easily done with chroot1, so that was my next step:

$ chroot /mnt/myroot restorecon -R /

Oh, wait wait wait ... We're now running the chroot'ed version of
restorecon, which is x86_64 and that just can't work on i686 system. The i686 file system root needs to be present in the chroot and the easiest way I could figure out at the moment was to do a simple bind mount:

$ mkdir -p /mnt/myroot/otherroot
$ mount -B / /mnt/myroot/otherroot

Now just alter the paths a bit:

$ chroot /mnt/myroot /otherroot/usr/sbin/restorecon -R /

Fails in flames too! Something about bad executable image for one of the shared objects that restorecon links against. Hmm, makes sense because it tries to load libraries directly from / which is my 64-bit /root at the moment. Need to change the chroot command a bit:

$ chroot /mnt/myroot bash -c\
'LD_LIBRARY_PATH=/otherroot/lib/ \  
/otherroot/usr/sbin/restorecon -R /'

Voila! That finally succeeded! Or did it not? Restarting the system and booting from disk ended up in the same flames as before, what is going on?

Stop carpet bombing the file system and rather do a laser-cut

Returning to the LIVE usb desktop I quickly probed /etc and saw that restorecon somehow missed the most important one /etc/passwd:

$ stat -c "%C" /mnt/myroot/etc/passwd

Damn, restorecon why did you do this to me? Alright, I suppose that precise cut with chcon can not fail, also I don't need to bother with the chroot anymore:

$ chcon -t passwd_file_t -l s0 /mnt/myroot/etc/{passwd,group}

Re-check before restarting to save myself some time:

$ stat -c "%C" /mnt/myroot/etc/passwd

Splendid! Finally, at around 3am I was able to boot my system and log in! Always carry a rescue USB thumb drive when playing with fire^WSELinux :)

  1. Much cleaner way that I found out about later would probably be:

  • setfiles -F -r /mnt/myroot /mnt/myroot/etc/selinux/targeted/contexts/files/file_contexts /mnt/myroot