Qualys Security Advisory

System Down: A systemd-journald exploit


========================================================================
Contents
========================================================================

Summary
CVE-2018-16864
- Analysis
- Exploitation
CVE-2018-16865
- Analysis
- Exploitation
CVE-2018-16866
- Analysis
- Exploitation
Combined Exploitation of CVE-2018-16865 and CVE-2018-16866
- amd64 Exploitation
- i386 Exploitation
Acknowledgments
Timeline

    Conversion, software version 7.0
        -- System of a Down, "Toxicity"


========================================================================
Summary
========================================================================

We discovered three vulnerabilities in systemd-journald
(https://en.wikipedia.org/wiki/Systemd):

- CVE-2018-16864 and CVE-2018-16865, two memory corruptions
  (attacker-controlled alloca()s);

- CVE-2018-16866, an information leak (an out-of-bounds read).

CVE-2018-16864 was introduced in April 2013 (systemd v203) and became
exploitable in February 2016 (systemd v230). We developed a proof of
concept for CVE-2018-16864 that gains eip control on i386.

CVE-2018-16865 was introduced in December 2011 (systemd v38) and became
exploitable in April 2013 (systemd v201). CVE-2018-16866 was introduced
in June 2015 (systemd v221) and was inadvertently fixed in August 2018.

We developed an exploit for CVE-2018-16865 and CVE-2018-16866 that
obtains a local root shell in 10 minutes on i386 and 70 minutes on
amd64, on average. We will publish our exploit in the near future.

To the best of our knowledge, all systemd-based Linux distributions are
vulnerable, but SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora
28 and 29 are not exploitable because their user space is compiled with
GCC's -fstack-clash-protection.

This confirms https://grsecurity.net/an_ancient_kernel_hole_is_not_closed.php:
"It should be clear that kernel-only attempts to solve [the Stack Clash]
will necessarily always be incomplete, as the real issue lies in the
lack of stack probing."


========================================================================
CVE-2018-16864
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

    The waves all keep on crashing by
        -- System of a Down, "Suggestions"

We accidentally discovered CVE-2018-16864 while working on the exploit
for Mutagen Astronomy (CVE-2018-14634); if we pass several megabytes of
command-line arguments to a program that calls syslog(), then journald
crashes:

systemd-journal[472]: segfault at 7ffe9a077420 ip 00007f45f6174877 sp 00007ffe9a0773f0 error 6 in systemd-journald[7f45f6169000+3f000]

(gdb) disassemble 0x7f45f6174877 - 0x7f45f6169000
Dump of assembler code for function dispatch_message_real.4064:
   ...
   0x000000000000b82c <+988>:   callq  0x2bd10 <get_process_cmdline.constprop.96>
   0x000000000000b831 <+993>:   test   %eax,%eax
   0x000000000000b833 <+995>:   js     0xb8ea <dispatch_message_real.4064+1178>
   0x000000000000b839 <+1001>:  mov    -0x218(%rbp),%rbx
   0x000000000000b840 <+1008>:  test   %rbx,%rbx
   0x000000000000b843 <+1011>:  je     0xd31b <dispatch_message_real.4064+7883>
   0x000000000000b849 <+1017>:  mov    %rbx,%rdi
   0x000000000000b84c <+1020>:  callq  0x5360 <strlen@plt>
   0x000000000000b851 <+1025>:  add    $0xa,%eax
   0x000000000000b854 <+1028>:  cltq
   0x000000000000b856 <+1030>:  add    $0x1e,%rax
   0x000000000000b85a <+1034>:  and    $0xfffffffffffffff0,%rax
   0x000000000000b85e <+1038>:  sub    %rax,%rsp
   0x000000000000b861 <+1041>:  movabs $0x454e494c444d435f,%rax
   0x000000000000b86b <+1051>:  lea    0x37(%rsp),%r15
   0x000000000000b870 <+1056>:  and    $0xfffffffffffffff0,%r15
   0x000000000000b874 <+1060>:  test   %rbx,%rbx
   0x000000000000b877 <+1063>:  mov    %rax,(%r15)
   0x000000000000b87a <+1066>:  mov    $0x3d,%eax
   0x000000000000b87f <+1071>:  mov    %ax,0x8(%r15)
   0x000000000000b884 <+1076>:  lea    0x9(%r15),%rax
   0x000000000000b888 <+1080>:  je     0xb895 <dispatch_message_real.4064+1093>
   0x000000000000b88a <+1082>:  mov    %rbx,%rsi
   0x000000000000b88d <+1085>:  mov    %rax,%rdi
   0x000000000000b890 <+1088>:  callq  0x5370 <stpcpy@plt>

538 static void dispatch_message_real(
...
604                 r = get_process_cmdline(ucred->pid, 0, false, &t);
605                 if (r >= 0) {
606                         x = strjoina("_CMDLINE=", t);

919 #define strjoina(a, ...)                                                \
920         ({                                                              \
921                 const char *_appendees_[] = { a, __VA_ARGS__ };         \
922                 char *_d_, *_p_;                                        \
923                 int _len_ = 0;                                          \
924                 unsigned _i_;                                           \
925                 for (_i_ = 0; _i_ < ELEMENTSOF(_appendees_) && _appendees_[_i_]; _i_++) \
926                         _len_ += strlen(_appendees_[_i_]);              \
927                 _p_ = _d_ = alloca(_len_ + 1);                          \
928                 for (_i_ = 0; _i_ < ELEMENTSOF(_appendees_) && _appendees_[_i_]; _i_++) \
929                         _p_ = stpcpy(_p_, _appendees_[_i_]);            \
930                 *_p_ = 0;                                               \
931                 _d_;                                                    \
932         })

This vulnerability, an attacker-controlled alloca()
(https://wiki.sei.cmu.edu/confluence/display/c/MEM05-C.+Avoid+large+stack+allocations)
at instruction 0xb85e and line 927, was introduced in systemd v203:

commit ae018d9bc900d6355dea4af05119b49c67945184
Date:   Mon Apr 22 23:10:13 2013 -0300
...
                 r = get_process_cmdline(ucred->pid, 0, false, &t);
                 if (r >= 0) {
-                        cmdline = strappend("_CMDLINE=", t);
+                        cmdline = strappenda("_CMDLINE=", t);

(strappenda() was renamed strjoina() in systemd v219) and became
exploitable in systemd v230:

commit ac2e41f5103ce2c679089c4f8fb6be61d7caec07
Date:   Fri Feb 12 04:59:57 2016 -0800
...
    This adds a wait flag to journal_file_set_offline(), when false the offline is
    performed asynchronously in a separate thread.

------------------------------------------------------------------------
Exploitation
------------------------------------------------------------------------

    ... it's the race
    Can you break out?
        -- System of a Down, "36"

CVE-2018-16864 is similar to a Stack Clash vulnerability
(https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt), but:

- Steps 1 (Clash the stack with another memory region) and 2 (Run the
  stack pointer to the start of the stack) are not needed, because the
  attacker-controlled alloca() can be very large (several megabytes of
  command-line arguments); only Steps 3 (Jump over the stack guard page,
  into another memory region) and 4 (Smash the stack, or another memory
  region) are needed.

- In Step 4 (Smash), the alloca() is fully written to (the vulnerability
  is essentially a stpcpy(alloca(strlen(cmdline) + 1), cmdline)), and
  the stpcpy() (a "wild copy") will therefore always crash into a
  read-only or unmapped memory region:

  https://googleprojectzero.blogspot.com/2015/03/taming-wild-copy-parallel-thread.html
  https://cansecwest.com/slides/2015/Taming%20wild%20copies%20-%20Chris%20evans.pdf

We tried to asynchronously interrupt this stpcpy() before it crashes,
with a signal or a timer, but we failed because journald uses signalfd()
and timerfd_create() to handle these events synchronously.

We eventually gained control of eip (i386's instruction pointer) by
jumping into and smashing the stack of a concurrent thread (a "Parallel
Thread Corruption"):

- First, we send a large, high-priority message (LOG_CRIT or higher) to
  journald, from a process whose cmdline is small; this message forces a
  large write() (between 1MB and 2MB) to /var/log/journal/ and forces
  the creation of a short-lived thread that fsync()s the journal (the
  stack of this thread is allocated in the mmap region).

- Next, we create several processes (between 32 and 64) that write() and
  fsync() large files (between 1MB and 8MB) to /var/tmp/ (for example);
  these processes stall journald's fsync() thread and will allow us to
  win a tight race: exploit the "wild copy" before it crashes.

- Last, we send a small, low-priority message to journald, from a
  process whose cmdline is very large (roughly 128MB, the distance
  between the main stack and the mmap region); this message forces a
  very large alloca() that jumps from journald's main stack into the
  stack of the fsync() thread, and smashes a saved eip before fsync()
  returns from kernel space.

On a Debian stable (9.5), our proof of concept wins this race and gains
eip control after a dozen tries (systemd automatically restarts journald
after each crash):

systemd-journal[2195]: segfault at 41414141 ip 41414141 sp b5f3d22c error 14

Despite this initial success, we abandoned the exploitation of
CVE-2018-16864: while working on our proof of concept, we discovered two
different vulnerabilities (CVE-2018-16865, another attacker-controlled
alloca(), and CVE-2018-16866, an information leak) that are reliably
exploitable on both i386 and amd64.


========================================================================
CVE-2018-16865
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

    Can you feel their haunting presence?
        -- System of a Down, "Holy Mountains"

Surprised by the heavy usage of alloca() in journald, we searched for
another attacker-controlled alloca() and found CVE-2018-16865:

1963 int journal_file_append_entry(JournalFile *f, const dual_timestamp *ts, const struct iovec iovec[], unsigned n_iovec, uint64_t *seqnum, Object **ret, uint64_t *offset) {
....
1986         items = alloca(sizeof(EntryItem) * MAX(1u, n_iovec));
1987
1988         for (i = 0; i < n_iovec; i++) {
1989                 uint64_t p;
1990                 Object *o;
1991
1992                 r = journal_file_append_data(f, iovec[i].iov_base, iovec[i].iov_len, &o, &p);
1993                 if (r < 0)
1994                         return r;
1995
1996                 xor_hash ^= le64toh(o->data.hash);
1997                 items[i].object_offset = htole64(p);
1998                 items[i].hash = o->data.hash;
1999         }

This vulnerability was introduced in systemd v38:

commit cf244689e9d1ab50082c9ddd0f3c4d1eb982badc
Date:   Thu Dec 29 15:00:57 2011 +0100
...
-        items = new(EntryItem, n_iovec);
-        if (!items)
-                return -ENOMEM;
+        items = alloca(sizeof(EntryItem) * n_iovec);

and became exploitable in systemd v201:

commit c4aa09b06f835c91cea9e021df4c3605cff2318d
Date:   Mon Apr 8 20:32:03 2013 +0200
...
-#define ENTRY_SIZE_MAX (1024*1024*64)
-#define DATA_SIZE_MAX (1024*1024*64)
...
+#define ENTRY_SIZE_MAX (1024*1024*768)
+#define DATA_SIZE_MAX (1024*1024*768)

If we send a large "native" message to /run/systemd/journal/socket:
since the maximum size of a "native" entry is 768MB, and the minimum
length of a "native" item is 3 ("A=\n"), and the size of an EntryItem
structure is 16 (a 64-bit offset and a 64-bit hash), the maximum size of
the attacker-controlled alloca() in journal_file_append_entry() is 768MB
/ 3 * 16 = 4GB, large enough to jump from journald's main stack into the
mmap region, even on amd64.

On amd64, as described in the "64-bit exploitation" of our Stack Clash
advisory, the randomized distance between the main stack and the mmap
region is shorter than 4GB with a probability of (approximately):

SUM(d = 0; d < 4GB; d++) d / (16GB * 1TB) ~= 1 / 2048

------------------------------------------------------------------------
Exploitation
------------------------------------------------------------------------

    Jump (pogo, pogo, pogo, pogo, pogo, pogo, pogo)
        -- System of a Down, "Bounce"

CVE-2018-16865 is basically a simplified Stack Clash vulnerability:

- Steps 1 (Clash) and 2 (Run) of the Stack Clash are not needed, since
  the largest attacker-controlled alloca() is 4GB; only Steps 3 (Jump)
  and 4 (Smash) are needed.

- In Step 4 (Smash), the alloca() is not necessarily fully written to:
  if the size of an item is larger than 128MB (DEFAULT_MAX_SIZE_UPPER),
  then journal_file_append_data() returns an error that breaks the "for"
  loop in journal_file_append_entry() (at lines 1992-1994) and avoids a
  crash into a read-only or unmapped memory region.

We eventually transformed this vulnerability into a crude
"write-what-where" (https://cwe.mitre.org/data/definitions/123.html):

- "write-where": We jump into and smash libc's read-write segment, and
  thereby overwrite a function pointer. Unfortunately this "write-where"
  is not surgical: the stack frames of the functions called from within
  the "for" loop (in journal_file_append_entry()) smash a few kilobytes
  below our target function pointer, and therefore overwrite vital libc
  variables that may crash or deadlock journald. Consequently, we must
  sometimes shift our alloca() jump slightly, to avoid overwriting such
  vital variables.

- "write-what": We want to overwrite our target function pointer with
  the address of another function or ROP chain, but unfortunately the
  stack frames of the functions called from within the "for" loop (in
  journal_file_append_entry()) do not contain any data that we control.
  However, the 64-bit "hash" values that are written to the alloca()ted
  "items" are produced by jenkins_hashlittle2(), a noncryptographic hash
  function: we can easily find a short string (a preimage) that hashes
  to a given value (the address that will overwrite our target function
  pointer) and is also a valid_user_field() (or journal_field_valid()).

  This "write-what" restricts our "write-where" to function pointers
  whose address modulo 16 is equal to 8 (the offset of "hash" in the
  EntryItem structure).

To complete our exploit, we need the address of journald's stack pointer
before the alloca() jump, and the address of our target function pointer
in libc's read-write segment -- we need an information leak.


========================================================================
CVE-2018-16866
========================================================================

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

    When they speak, we can peek from the windows of their mouths
        -- System of a Down, "Know"

We discovered an out-of-bounds read in journald (CVE-2018-16866), and
transformed it into an information leak:

 31 #define WHITESPACE        " \t\n\r"
...
194 size_t syslog_parse_identifier(const char **buf, char **identifier, char **pid) {
195         const char *p;
...
197         size_t l, e;
...
203         p = *buf;
204
205         p += strspn(p, WHITESPACE);
206         l = strcspn(p, WHITESPACE);
207
208         if (l <= 0 ||
209             p[l-1] != ':')
210                 return 0;
211
212         e = l;
...
240         if (strchr(WHITESPACE, p[e]))
241                 e++;
242         *buf = p + e;
243         return e;
244 }

If we send a syslog message to journald (in *buf), and if the last
character of this message is a ':' (before the '\0' terminator), then:

- at line 240, p[e] is the '\0' terminator of our message;

- at line 240, strchr(WHITESPACE, p[e]) returns a pointer to the '\0'
  terminator of the WHITESPACE string (as mentioned in man strchr: "The
  terminating null byte is considered part of the string, so that if c
  is specified as '\0', these functions return a pointer to the
  terminator.");

- at line 241, e is incremented;

- at line 242, *buf points out-of-bounds, to the first character after
  the '\0' terminator of our message;

- later, the out-of-bounds string at *buf (supposedly the body of our
  syslog message) is written (leaked) to the journal.

Consequently, we can read this out-of-bounds string:

- either directly from the journal (if journald's "Storage" is
  "persistent", or "auto" and /var/log/journal/ exists), because
  journald supports extended file ACLs (Access Control Lists):

  $ id
  uid=1000(john) gid=1000(john) groups=1000(john) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

  $ ls -l /var/log/journal/*/user-$UID.journal
  -rw-r-----+ 1 root systemd-journal 8388608 Nov 20 09:35 /var/log/journal/2562d1eced654f44a3d3a217d66b9ff3/user-1000.journal

  $ getfacl /var/log/journal/*/user-$UID.journal
  ...
  user:john:r--

  $ ./infoleak

  $ journalctl --all --user --lines=1 --identifier=infoleak | hexdump -C
  ...
  00000050  2e 20 2d 2d 0a 4e 6f 76  20 32 30 20 31 36 3a 30  |. --.Nov 20 16:0|
  00000060  30 3a 33 36 20 6c 6f 63  61 6c 68 6f 73 74 2e 6c  |0:36 localhost.l|
  00000070  6f 63 61 6c 64 6f 6d 61  69 6e 20 69 6e 66 6f 6c  |ocaldomain infol|
  00000080  65 61 6b 5b 33 35 34 38  5d 3a 20 78 fb 1e 78 54  |eak[3548]: x..xT|
  00000090  7f 0a                                             |..|

- or (if journald's "Storage" is "volatile", or "auto" and
  /var/log/journal/ does not exist) from a tty that we recorded to
  /var/run/utmp, because journald writes ("walls") emergency messages
  (LOG_EMERG) to the tty of every logged-in user; our exploit records a
  tty to /var/run/utmp via an ssh connection to localhost, but other
  methods exist (for example, utempter and gnome-pty-helper):

  $ ./infoleak
  ...
  00003510  0a 07 0d 0d 0a 42 72 6f  61 64 63 61 73 74 20 6d  |.....Broadcast m|
  00003520  65 73 73 61 67 65 20 66  72 6f 6d 20 73 79 73 74  |essage from syst|
  00003530  65 6d 64 2d 6a 6f 75 72  6e 61 6c 64 40 6c 6f 63  |emd-journald@loc|
  00003540  61 6c 68 6f 73 74 2e 6c  6f 63 61 6c 64 6f 6d 61  |alhost.localdoma|
  00003550  69 6e 20 28 54 75 65 20  32 30 31 38 2d 31 31 2d  |in (Tue 2018-11-|
  00003560  32 30 20 31 36 3a 32 35  3a 34 36 20 43 53 54 29  |20 16:25:46 CST)|
  00003570  3a 0d 0d 0a 0d 0d 0a 69  6e 66 6f 6c 65 61 6b 5b  |:......infoleak[|
  00003580  33 38 37 32 5d 3a 20 78  6b a2 e1 2f 7f 0d 0d 0a  |3872]: xk../....|

This vulnerability was introduced in systemd v221:

commit ec5ff4445cca6a1d786b8da36cf6fe0acc0b94c8
Date:   Wed Jun 10 22:33:44 2015 -0700
...
-        e += strspn(p + e, WHITESPACE);
+        if (strchr(WHITESPACE, p[e]))
+                e++;

and was inadvertently fixed in August 2018:

commit a6aadf4ae0bae185dc4c414d492a4a781c80ffe5
Date:   Wed Aug 8 15:06:36 2018 +0900
...
-        if (strchr(WHITESPACE, p[e]))
-                e++;
+        e += strspn(p + e, WHITESPACE);

commit 8595102d3ddde6d25c282f965573a6de34ab4421
Date:   Fri Aug 10 11:07:54 2018 +0900
...
-        e += strspn(p + e, WHITESPACE);
+        /* Single space is used as separator */
+        if (p[e] != '\0' && strchr(WHITESPACE, p[e]))
+                e++;

------------------------------------------------------------------------
Exploitation
------------------------------------------------------------------------

    For today we will take the body parts and put them on the wall
        -- System of a Down, "Dreaming"

To leak a stack address or an mmap address from journald:

- First, we send a large native message to /run/systemd/journal/socket;
  journald mmap()s our message, and malloc()ates a large array of iovec
  structures: most of these structures point into our mmap()ed message,
  but some of them point to the stack (in dispatch_message_real()). The
  contents of this iovec array (especially the mmap and stack pointers)
  are preserved in a heap hole after free() (after journald finishes
  processing our message).

- Next, we send a large syslog message to /run/systemd/journal/dev-log;
  to receive our large message (in server_process_datagram()), journald
  realloc()ates its server buffer into the heap hole that previously
  contained the iovec array (and still contains remains of mmap and
  stack pointers).

- Last, we send a large syslog message that exploits CVE-2018-16866;
  journald receives our large message in its server buffer (in the heap
  chunk that previously contained the iovec array), and if we carefully
  choose the size of our message and position its terminating ":" in
  front of a remaining mmap or stack pointer, then we can leak this
  pointer (it is mistakenly read out-of-bounds as the body of our
  message).

>From this leaked stack pointer we easily deduce journald's stack pointer
before the alloca() jump, because the distance between the two depends
only on journald's executable.

>From the leaked mmap address we can deduce libc's address, but chunks of
unknown sizes are mmap()ed between the two, and we must therefore adopt
different strategies based on our target architecture (i386 or amd64).


========================================================================
Combined Exploitation of CVE-2018-16865 and CVE-2018-16866
========================================================================

    Don't leave your seats now
    Popcorn everywhere ...
        -- System of a Down, "CUBErt"

------------------------------------------------------------------------
amd64 Exploitation
------------------------------------------------------------------------

- To deduce libc's address from the leaked mmap address of our native
  message, we arrange for this message to be mmap()ed into the 2MB hole
  between ld.so's read-execute and read-only segments: from this hole's
  address we deduce ld.so's address, and hence libc's address (with help
  from ldd's output).

- If the resulting stack-to-libc distance is jumpable (if it is shorter
  than 4GB), then we proceed with our "write-what-where"; otherwise, we
  restart journald (we crash it with an alloca() of RLIMIT_STACK -- 8MB
  by default) and try again.

  We have a good chance of obtaining a jumpable stack-to-libc distance
  (and hence a root shell) after 2048 tries * 2 seconds ~= 68 minutes
  (by default, if journald crashes less than 5 times within 10 seconds,
  it is restarted automatically by systemd).

- For the "write-where" part of our "write-what-where", we overwrite
  libc's __free_hook function pointer, whose address modulo 16 is always
  equal to 8 (on every amd64 distribution that we exploited).

- For the "write-what" part of our "write-what-where", we overwrite
  __free_hook with the address of libc's system() function: whenever
  journald free()s data that we control, we achieve arbitrary command
  execution.

Last-minute note: on CentOS 7, the usual function pointers in libc's
read-write segment (__free_hook, __malloc_hook, etc) are not located at
multiples of 16 plus 8. To circumvent this problem:

- First, we overwrite the "_chain" pointer of stderr's FILE structure
  with the address of our own fake FILE structure (this "_chain" pointer
  is located at a multiple of 16 plus 8, in libc's read-write segment).

- Next, we corrupt one of malloc's internal variables (also in libc's
  read-write segment).

- Last, we force a call to malloc() or free(), which detects the
  corruption of its internal variable and calls abort(), which calls
  _IO_flush_all_lockp(), which follows stderr's overwritten "_chain"
  pointer to our fake FILE structure; we eventually achieve arbitrary
  command execution by calling libc's system() via one of the function
  pointers in our fake FILE structure.

------------------------------------------------------------------------
i386 Exploitation
------------------------------------------------------------------------

Our i386 exploit is very similar to the amd64 exploit, but:

- The stack-to-libc distance is always jumpable (it is roughly 128MB).

- There is no hole between ld.so's read-execute and read-only segments.
  However, libc's address is randomized in a narrow range of 1MB and is
  therefore brute forcible: we have a good chance of correctly guessing
  libc's address after 1MB / 4KB = 256 tries * 2 seconds ~= 8 minutes.

- For the "write-where" part of our "write-what-where", we overwrite
  libc's __malloc_hook function pointer (__free_hook was never located
  at a multiple of 16 plus 8 or 12 on the i386 distributions that we
  exploited, but __malloc_hook always is).

- For the "write-what" part of our "write-what-where", we overwrite
  __malloc_hook with the address of a "mov esp, 0x89fffa5d ; ret" gadget
  (or equivalent stack pivot): since our native message can be as large
  as 768MB, we can mmap() it at 0x89fffa5d, take control of the stack,
  and return into libc's execve().


========================================================================
Acknowledgments
========================================================================

We thank systemd's developers, Red Hat Product Security, and the members
of linux-distros@openwall.


========================================================================
Timeline
========================================================================

2018-11-26: Advisory sent to Red Hat Product Security (as recommended by
https://github.com/systemd/systemd/blob/master/docs/CONTRIBUTING.md#security-vulnerability-reports).

2018-12-26: Advisory and patches sent to linux-distros@openwall.

2019-01-09: Coordinated Release Date (6:00 PM UTC).