Core dumps from daemon processes?

[Anthony Howe]

I have a daemon process I've written and trying to debug, however,
whenever it crashes, I get no core file.

1. The daemon does a setrlimit of RLIMIT_CORE to RLIM_INFINITY (same as
if I had done "ulimit -H -c unlimited" in the shell before starting the
process).

2. The daemon sets its working directory to /var/tmp, which is writeable
by every one, yet no core file appears. Likewise /var/crash has no core
file.

3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other
user (which can write into /var/tmp).

4. A find / -name '*.core' finds nothing anywhere.

5. I've even tried sysctl kern.nosuidcoredump=0 and
kern.nosuidcoredump=2 thinking that the setuid family of system
functions might be resetting a flag when I drop privileges.

6. Even forcing a SIGABRT, which should produce a core, does not create
one.


What am I missing in order to get a daemon process on OpenBSD to dump
core when it crashes?

[Jan Stary]

Post by Anthony Howe
I have a daemon process I've written and trying to debug, however,
whenever it crashes, I get no core file.
1. The daemon does a setrlimit of RLIMIT_CORE to RLIM_INFINITY (same as
if I had done "ulimit -H -c unlimited" in the shell before starting the
process).
2. The daemon sets its working directory to /var/tmp, which is writeable
by every one, yet no core file appears. Likewise /var/crash has no core
file.
3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other
user (which can write into /var/tmp).
4. A find / -name '*.core' finds nothing anywhere.
5. I've even tried sysctl kern.nosuidcoredump=0 and
kern.nosuidcoredump=2 thinking that the setuid family of system
functions might be resetting a flag when I drop privileges.
6. Even forcing a SIGABRT, which should produce a core, does not create
one.
What am I missing in order to get a daemon process on OpenBSD to dump
core when it crashes?

Depends on how exactly it "crashes", man sigaction(2).
Are you possibly catching the signals yourself?
The following coredumps as a charm:

#include <stdlib.h>

int
main()
{
abort();
/* NOTREACHED */
return 0;
}

[Otto Moerbeek]

Post by Jan Stary

Post by Anthony Howe
I have a daemon process I've written and trying to debug, however,
whenever it crashes, I get no core file.
1. The daemon does a setrlimit of RLIMIT_CORE to RLIM_INFINITY (same as
if I had done "ulimit -H -c unlimited" in the shell before starting the
process).
2. The daemon sets its working directory to /var/tmp, which is writeable
by every one, yet no core file appears. Likewise /var/crash has no core
file.
3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other
user (which can write into /var/tmp).
4. A find / -name '*.core' finds nothing anywhere.
5. I've even tried sysctl kern.nosuidcoredump=0 and
kern.nosuidcoredump=2 thinking that the setuid family of system
functions might be resetting a flag when I drop privileges.
6. Even forcing a SIGABRT, which should produce a core, does not create
one.
What am I missing in order to get a daemon process on OpenBSD to dump
core when it crashes?

Depends on how exactly it "crashes", man sigaction(2).
Are you possibly catching the signals yourself?
#include <stdlib.h>
int
main()
{
abort();
/* NOTREACHED */
return 0;
}

ktrace might help here too.

-Otto

[Anthony Howe]

Post by Jan Stary

Post by Anthony Howe
I have a daemon process I've written and trying to debug, however,
whenever it crashes, I get no core file.
1. The daemon does a setrlimit of RLIMIT_CORE to RLIM_INFINITY (same as
if I had done "ulimit -H -c unlimited" in the shell before starting the
process).
2. The daemon sets its working directory to /var/tmp, which is writeable
by every one, yet no core file appears. Likewise /var/crash has no core
file.
3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other
user (which can write into /var/tmp).
4. A find / -name '*.core' finds nothing anywhere.
5. I've even tried sysctl kern.nosuidcoredump=0 and
kern.nosuidcoredump=2 thinking that the setuid family of system
functions might be resetting a flag when I drop privileges.
6. Even forcing a SIGABRT, which should produce a core, does not create
one.
What am I missing in order to get a daemon process on OpenBSD to dump
core when it crashes?

Depends on how exactly it "crashes", man sigaction(2).
Are you possibly catching the signals yourself?

I trap several signals, but not SIGABRT.

[Theo de Raadt]

Post by Anthony Howe
3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

[Anthony Howe]

Post by Jan Stary

Post by Anthony Howe
I have a daemon process I've written and trying to debug, however,
whenever it crashes, I get no core file.
1. The daemon does a setrlimit of RLIMIT_CORE to RLIM_INFINITY (same as
if I had done "ulimit -H -c unlimited" in the shell before starting the
process).
2. The daemon sets its working directory to /var/tmp, which is writeable
by every one, yet no core file appears. Likewise /var/crash has no core
file.
3. The program does not use file system setuid bits, BUT does use the
setuid() et al. system calls to drop privileges from root to some other
user (which can write into /var/tmp).
4. A find / -name '*.core' finds nothing anywhere.
5. I've even tried sysctl kern.nosuidcoredump=0 and
kern.nosuidcoredump=2 thinking that the setuid family of system
functions might be resetting a flag when I drop privileges.
6. Even forcing a SIGABRT, which should produce a core, does not create
one.
What am I missing in order to get a daemon process on OpenBSD to dump
core when it crashes?

Depends on how exactly it "crashes", man sigaction(2).
Are you possibly catching the signals yourself?

I trap several signals, but not SIGABRT.

I handle these myself...

SIGHUP
SIGINT
SIGQUIT

And ignore these...

SIGPIPE
SIGTERM
SIGALRM
SIGXCPU
SIGXFSZ
SIGVTALRM

The daemon is threaded, so my server API handles; serverSignalsInit sets
up pthread_sigmask to SIG_BLOCK and serverSignalsLoop is the only thread
to receive signals, so other threads do not interfere. Part of my
library code has works fine.

The issue is core dumps and since SIGABRT should generate a core and I'm
not blocking SIGABRT, the daemon should terminate with a core when I
force SIGABRT.

It don't.

So I assume there is something else interfering with the generation of
core files, possibly sysctl kern.nosuidcoredump; the daemon is not
setuid, but does use setuid family of functions. setrlimits RLIMIT_CORE
is set to infinity, and RLIMIT_FSIZE is unlimited too, so the there
should be no problem concerning the size of the core file.

Sending SIGABRT is just a means to force test core files are created. If
I can do that, then when the daemon crashes I'll get a core file to work
with.
But that example is not a daemon. Its just a regular process. Daemon
servers behave differently.

Post by Jan Stary
#include <stdlib.h>
int
main()
{
abort();
/* NOTREACHED */
return 0;
}

[Remco]

Post by Anthony Howe

Post by Jan Stary
Are you possibly catching the signals yourself?

I trap several signals, but not SIGABRT.
I handle these myself...
SIGHUP
SIGINT
SIGQUIT
And ignore these...
SIGPIPE
SIGTERM
SIGALRM
SIGXCPU
SIGXFSZ
SIGVTALRM
The daemon is threaded, so my server API handles; serverSignalsInit sets
up pthread_sigmask to SIG_BLOCK and serverSignalsLoop is the only thread
to receive signals, so other threads do not interfere. Part of my
library code has works fine.
The issue is core dumps and since SIGABRT should generate a core and I'm
not blocking SIGABRT, the daemon should terminate with a core when I
force SIGABRT.
It don't.

Don't know if I understand signals well enough but is it possible you want
the default action for SIGABRT instead of ignoring it ?

[Anthony Howe]

Here's a little "do nothing" daemon server that demonstrates the problem.

---
#include <stdlib.h>

int
main()
{
(void) daemon(1,1);
(void) chdir("/tmp");

printf("before uid=%d euid=%d\n", getuid(), geteuid());
(void) setuid(1);
printf("after uid=%d euid=%d\n", getuid(), geteuid());
for (;;)
sleep(1);
/* NOTREACHED */
return 0;
}

---

As root...

# gcc a.c build
# ulimit -S -c unlimited set core dumps
# ulimit -a verify coredump unlimited

# sysctl kern.nosuidcoredump=0 or set to =2

# ./a.out start it, becomes user daemon (id 1)
and set work dir to /tmp

# pkill -ABRT a.out kill it

# ls -a /tmp /var/crash OOPS! No core file.

Without the call to setuid, then the daemon will create a core file in /tmp.

What I would like to know is how to get a core file when the daemon
program uses setuid/seteuid family of functions, which appears to make
it subject to kern.nosuidcoredump? I've tried all 3 possible values

/* KERN_NOSUIDCOREDUMP interger values:
*
* 0 dump core,
* 1 disable dump core (default)
* 2 dump core to /var/crash.
*/

Nothing appears to work. Tested on 4.0 and 4.3 systems.

So what am I forgetting?

[Ted Unangst]

Post by Anthony Howe
Without the call to setuid, then the daemon will create a core file in /tmp.
What I would like to know is how to get a core file when the daemon
program uses setuid/seteuid family of functions, which appears to make
it subject to kern.nosuidcoredump? I've tried all 3 possible values

Not sure. I added the 2 setting just for scenarios like this, but it
was several years ago. Looks like it doesn't work. Maybe it never
did. :(

[Philip Guenther]

Post by Ted Unangst

Post by Anthony Howe
Without the call to setuid, then the daemon will create a core file in /tmp.
What I would like to know is how to get a core file when the daemon
program uses setuid/seteuid family of functions, which appears to make
it subject to kern.nosuidcoredump? I've tried all 3 possible values

Not sure. I added the 2 setting just for scenarios like this, but it
was several years ago. Looks like it doesn't work. Maybe it never
did. :(

Look Ted, you perhaps should feel guilty about other things, but not
that: Anthony's test program dumps core in /var/crash/ Just Fine with
kern.nosuidcoredump=2 and -current, and that code hasn't change in
quite a while.


Philip Guenther

[Anthony Howe]

Post by Philip Guenther

Post by Ted Unangst

Post by Anthony Howe
Without the call to setuid, then the daemon will create a core file in /tmp.
What I would like to know is how to get a core file when the daemon
program uses setuid/seteuid family of functions, which appears to make
it subject to kern.nosuidcoredump? I've tried all 3 possible values

Not sure. I added the 2 setting just for scenarios like this, but it
was several years ago. Looks like it doesn't work. Maybe it never
did. :(

Look Ted, you perhaps should feel guilty about other things, but not
that: Anthony's test program dumps core in /var/crash/ Just Fine with
kern.nosuidcoredump=2 and -current, and that code hasn't change in
quite a while.

But it doesn't do that on my OpenBSD servers (4.0 and 4.3) regardless of
what I set kern.nosuidcoredump. Remember too that my claim is that
without the setuid() function call, it works fine; with the setuid()
call it does not.

Theo posted else where in the thread ...

Post by Philip Guenther

Post by Ted Unangst
3. The program does not use file system setuid bits, BUT does use the

Post by Anthony Howe
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

Which I find a very strange choice, but is at least clear in that any
program calling setuid function family will never drop core regardless
of kern.nosuidcoredump setting.

[Theo de Raadt]

Post by Anthony Howe

Post by Theo de Raadt

Post by Anthony Howe
3. The program does not use file system setuid bits, BUT does use the

Post by Anthony Howe
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

Which I find a very strange choice,

I gues it's good that we get to make the choices. In all the other
projects, such choices would not even be thought of.

[Anthony Howe]

Post by Theo de Raadt

Post by Anthony Howe

Post by Theo de Raadt

Post by Anthony Howe
3. The program does not use file system setuid bits, BUT does use the

Post by Anthony Howe
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

Which I find a very strange choice,

I gues it's good that we get to make the choices. In all the other
projects, such choices would not even be thought of.

It is a choice that is hard on application developers when it comes to
debugging problems. Linux has per process PR_SET_DUMPABLE flag; FreeBSD
has (last I looked) a kern.sugid_coredump similar to OpenBSD
kern.nosuidcoredump.

I just find it odd from a practical view point that kern.nosuidcoredump
no longer applies, though understand from a security view point that one
would want to avoid slip ups by the developer between setuid and seteuid
or in forgetting to restore the setting to a secure mode after debugging.

[Theo de Raadt]

Post by Anthony Howe
I just find it odd from a practical view point that kern.nosuidcoredump
no longer applies, though understand from a security view point that one
would want to avoid slip ups by the developer between setuid and seteuid
or in forgetting to restore the setting to a secure mode after debugging.

Sorry, but that underestimates the danger.

A process which has operated as multiple uids _always_ is doing so because
it is mixing up domains of operation; it has some capability which it is
trying to hide or block.

Some capabilities are sockets or ports or such which only root or some
other uid could open but which can continue to be used afterwards, but
the other type of capabilities represent themselves in memory, and
are typically a secret, even if only by the addresses of the block of
data.

Such memory should never end up on disk, in a file.

It is easy to talk about more sophisticated models for turning such
things on and off, and that's great, but then what do the other
systems do with these fancy options?

They leave them turned off. To avoid mails like yours.

So nothing is gained.

Instead, as a group our policy is to turn these things on, not make it
easy for them to be turned off, and thus enforce the policy strictly,
and thereby we educate people using these functions to get used to the
choices they should be making in security software. If we don't make
them aware, they will remain blissfully aware and think they are smart
enough to write setuid or daemon software.

[Anthony Howe]

Post by Theo de Raadt
Instead, as a group our policy is to turn these things on, not make it
easy for them to be turned off, and thus enforce the policy strictly,
and thereby we educate people using these functions to get used to the
choices they should be making in security software. If we don't make
them aware, they will remain blissfully aware and think they are smart
enough to write setuid or daemon software.

When you first mentioned the policy in your first reply (without this
latter elaboration) and knowing how famed you are for such strict and
uncompromising views on security, I deduced your motives behind the
policy choice. It makes my life difficult as a developer, but as you
state, it makes an attacker's life even more so. I'll live now that I
have been edified.

[Anthony Howe]

kern.nosuidcoredump=2 works fine for me in -current (though =0 seems
broken), so please try with a newer OS version. This should work, it's
needed for debugging Xorg too.

OK. I can confirm your results, see below for the verbose blow by blow.

So for myself, its proven that if I upgrade to 4.6, there is at least
one method (sysctl kern.nosuidcoredump=2) to debug setuid() daemon
servers. I can live with that; just document kern.nosuidcoredump=2
please in core(5) or setuid(2) for developers.

Thank you all for your time.

Anthony Howe

--- long version ---

I'll going to skip the side debate of why my home file server (4.0) and
my production machine (4.3) are not at least up to date with 4.6. That
way be fiery dragons.

I've pulled out an old 500 Mhz machine from a cupboard and have
installed 4.6 on it...

OpenBSD 4.6 (GENERIC) #58: Thu Jul 9 21:24:42 MDT 2009
***@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD-K6(tm) 3D processor ("AuthenticAMD" 586-class) 502 MHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,MCE,CX8,PGE,MMX
real mem = 532180992 (507MB)
avail mem = 505774080 (482MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 07/15/95, BIOS32 rev. 0 @ 0xfdb60,
SMBIOS rev. 2.1 @ 0xf5e60 (26 entries)
bios0: vendor American Megatrends Inc. version "062601" date 07/15/97
bios0: M599LMR M599LMR
...

Here is the nulld.c code, slightly tweaked for easy enable/disable of
setuid().

usage: nulld [numeric-uid-to-set]

Please note that this is not how I would write a setuid daemon server;
its strictly meant for testing behaviour of core dumps with respect to
setuid().

---
#include <stdlib.h>
#include <string.h>

int
main(int argc, char **argv)
{
(void) daemon(1,1);
(void) chdir("/tmp");

if (1 < argc) {
printf("before uid=%d euid=%d\n", getuid(), geteuid());
(void) setuid(strtol(argv[1], NULL, 10));
printf("after uid=%d euid=%d\n", getuid(), geteuid());
}

for (;;)
sleep(60);

/* NOTREACHED */
return 0;
}
---

Three tests, assumes logged in as root, assumes ulimit -c unlimited:

1. No setuid(), assert kern.nosuidcoredump=1; core dump in /tmp as
expected. No argument.

***@pizza:# rm /tmp/*core /var/crash/*core
rm: /tmp/*core: No such file or directory
rm: /var/crash/*core: No such file or directory
***@pizza:# sysctl kern.nosuidcoredump=1
kern.nosuidcoredump: 1 -> 1
***@pizza:# ./nulld
***@pizza:# pkill -ABRT nulld
***@pizza:# ls /tmp /var/crash
/tmp:
.ICE-unix/ .X11-unix/ nulld.core

/var/crash:
minfree
***@pizza:#


2. setuid(1), nosuidcoredump=0; no core dump in /tmp; there is a problem
in OpenBSD 4.6, though given Theo's earlier comments this appears to be
policy, which I can accept if its document in core(5) and/or setuid(2)
man pages.

***@pizza:# rm /tmp/*core /var/crash/*core
rm: /var/crash/*core: No such file or directory
***@pizza:# sysctl kern.nosuidcoredump=0
kern.nosuidcoredump: 1 -> 0
***@pizza:# ./nulld 1
***@pizza:# before uid=0 euid=0
after uid=1 euid=1

***@pizza:# pkill -ABRT nulld
***@pizza:# ls /tmp /var/crash
/tmp:
.ICE-unix/ .X11-unix/

/var/crash:
minfree
***@pizza:#


3. setuid(1), nosuidcoredump=2; core dump in /var/crash; no problem with
OpenBSD 4.6.

***@pizza:# rm /tmp/*core /var/crash/*core
rm: /tmp/*core: No such file or directory
rm: /var/crash/*core: No such file or directory
***@pizza:# sysctl kern.nosuidcoredump=2
kern.nosuidcoredump: 0 -> 2
***@pizza:# ./nulld 1
***@pizza:# before uid=0 euid=0
after uid=1 euid=1

***@pizza:# pkill -ABRT nulld
***@pizza:# ls /tmp /var/crash
/tmp:
.ICE-unix/ .X11-unix/

/var/crash:
minfree nulld.core
***@pizza:#

[Rogier Krieger]

Would the following be an improvement for the documentation? Feel free
to flame my mdoc(7) skills or lack thereof.

Regards,

Rogier


### Eclipse Workspace Patch 1.0
#P man5
Index: core.5
===================================================================
RCS file: /cvs/src/share/man/man5/core.5,v
retrieving revision 1.12
diff -u -r1.12 core.5
--- core.5 31 May 2007 19:19:58 -0000 1.12
+++ core.5 24 Feb 2010 18:57:21 -0000
@@ -158,7 +158,16 @@
.Xr gdb 1 ,
.Xr pmdb 1 ,
.Xr setrlimit 2 ,
-.Xr sigaction 2
+.Xr sigaction 2 ,
+.Xr sysctl 3
+.Sh CAVEATS
+Programs with their set-user-ID bit set will not dump core as a security
+precaution. This prevents sensitive information from ending up on disk.
+For debugging programs affected by this, refer to
+.Xr sysctl 3
+for the
+.Li kern.nosuidcoredump
+option for how to deal with this.
.Sh HISTORY
A
.Nm

[Anthony Howe]

Post by Rogier Krieger
Would the following be an improvement for the documentation? Feel free

Not entirely correct. I'd say this:

----
Programs with their set-user-ID bit set or that make use of the setuid
family of functions will not dump core as a security precaution. This
prevents sensitive information from ending up on disk. For debugging,
programs affected by this should set:

sysctl kern.nosuidcoredump=2

Core dumps will then be saved to /var/crash.
----

Simply referring to sysctl(3) doesn't help since the possible values
that kern.nosuidcoredump can be set to are not described there. I don't
think they're described in any man page. You either need to sift through
newsgroups, release notes, or read the source.

[Jason McIntyre]

Post by Anthony Howe
Simply referring to sysctl(3) doesn't help since the possible values
that kern.nosuidcoredump can be set to are not described there. I don't
think they're described in any man page. You either need to sift through
newsgroups, release notes, or read the source.

sysctl(3):

KERN_NOSUIDCOREDUMP
Programs with their set-user-ID bit set will not dump core
when this is set. The special value of 2 means that core
dumps will be allowed, but placed in /var/crash.

jmc

[Anthony Howe]

Post by Anthony Howe
KERN_NOSUIDCOREDUMP
Programs with their set-user-ID bit set will not dump core
when this is set. The special value of 2 means that core
dumps will be allowed, but placed in /var/crash.

So it does now. I had to look at a 4.6 system.

Still it doesn't hurt to state that in core(5).

[Philip Guenther]

...

Post by Anthony Howe

Post by Philip Guenther
Look Ted, you perhaps should feel guilty about other things, but not
that: Anthony's test program dumps core in /var/crash/ Just Fine with
kern.nosuidcoredump=2 and -current, and that code hasn't change in
quite a while.

But it doesn't do that on my OpenBSD servers (4.0 and 4.3) regardless of
what I set kern.nosuidcoredump.

You've apparently chosen to stick with old versions whose patch
branches have been unmaintained for, respectively, more than 2 years
and almost a year. As a developer, I have no interest in tracking
down the change that fixed it and back porting to such versions, as
only someone insane or self-supporting would be still using those.
So, reporting that it doesn't work on those is basically
uninteresting. Best of luck with your systems!

The behavior of kern.nosuidcoredump=0 will almost certainly be fixed before 4.7.

Post by Anthony Howe
Remember too that my claim is that
without the setuid() function call, it works fine; with the setuid()
call it does not.

Umm, yes, of course. Is that a surprise?


Philip Guenther

[Theo de Raadt]

Post by Theo de Raadt

Post by Anthony Howe

Post by Theo de Raadt

Post by Anthony Howe
3. The program does not use file system setuid bits, BUT does use the

Post by Anthony Howe
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

Which I find a very strange choice,

I gues it's good that we get to make the choices. In all the other
projects, such choices would not even be thought of.

It is a choice that is hard on application developers when it comes to
debugging problems. Linux has per process PR_SET_DUMPABLE flag; FreeBSD
has (last I looked) a kern.sugid_coredump similar to OpenBSD
kern.nosuidcoredump.

It is a choice that is hard on people trying to find password or keying
information inside priv-sep daemons.

[Anthony Howe]

Post by Theo de Raadt

Post by Theo de Raadt

Post by Anthony Howe

Post by Theo de Raadt

Post by Anthony Howe
3. The program does not use file system setuid bits, BUT does use the

Post by Anthony Howe
setuid() et al. system calls to drop privileges from root to some other

In OpenBSD -- if you change uids, you don't get core dumps.

Which I find a very strange choice,

I gues it's good that we get to make the choices. In all the other
projects, such choices would not even be thought of.

It is a choice that is hard on application developers when it comes to
debugging problems. Linux has per process PR_SET_DUMPABLE flag; FreeBSD
has (last I looked) a kern.sugid_coredump similar to OpenBSD
kern.nosuidcoredump.

It is a choice that is hard on people trying to find password or keying
information inside priv-sep daemons.

Yes. I understand, now that it has been declared. A paragraph about this
in man core(5) would help as a future reminder.