1:57 p.m.
Hi,
Lately I have been facing a lot of difficulty trying to get the
information I want easily using journalctl. I find the manpage of
limited use; as in, it has the basic information but the more advanced
information is scattered in several manpages and the text is littered
with jargon more appropriate for developers (IMO this last point holds
true for most of systemd documentation)!
So I thought it might be useful to share a few methods I know, followed
by some questions. Maybe others can share their tricks too. This
thread could then serve as a more accessible documentation for users
(which could then be ported to a wiki page).
Some basic comments:
1. If your journal size is large, piping to grep is quite a bit slow.
2. To run journalctl as a regular user, you need to add yourself to the
group systemd-journal, logout, and login again.
Useful commandline switches I'm familiar with:
1. Most recent entries first, `-r/--reverse'.
2. To follow, `-f/--follow'.
3. To limit logs by timestamp, `--since/--until'; it takes absolute
timestamps (2013-12-31) as well as relative time stamps (-2d, -10m).
BTW, the manpage does not say anything about the units for relative
times. I had to find out by trial-and-error m stands for minutes,
not months.
4. Limit output from this boot, `-b/--this-boot'.
$ journalctl -b 'bootid'.
I find the more general interface to filter by _BOOT_ID most
ridiculous. How is the user supposed to know what was the boot-id
for any of the previous sessions?
5. Filter by unit files, `-u/--unit'.
$ journalctl -u <unit_file> # .service extension optional
6. To filter by journal fields, just pass FIELD=<value>.
7. The actual useful documentation for (4), (5) & (6) is really in
systemd.journal-fields(7). No mention is made of this other than the
`SEE ALSO' section at the very end. The fields manpage is also the
perfect example of documentation written for developers instead of
users (another one would be journald.conf(5)).
8. You can list valid values for the fields in (6) with `-F/--field'.
Example, for all known boot-ids do this:
$ journalctl -F _BOOT_ID
Now my questions:
1. How can I filter messages printed to the logs from my cron jobs? I
will try to explain by example:
$ journalctl -ru crond --since=-3d
-- Logs begin at Sun 2013-11-17 02:48:46 CET, end at Wed 2014-01-01 20:31:27 CET. --
$ journalctl -r --since -3d | grep rsnapshot
Jan 01 04:34:08 <hostname> rsnapshot[15294]: /usr/bin/rsnapshot daily:
completed successfully
Jan 01 04:30:01 <hostname> CROND[15270]: (root) CMD (/usr/bin/rsnapshot daily)
Jan 01 04:00:07 <hostname> rsnapshot[15198]: /usr/bin/rsnapshot monthly:
completed successfully
Jan 01 04:00:01 <hostname> CROND[15196]: (root) CMD (/usr/bin/rsnapshot
monthly)
Dec 31 04:35:45 <hostname> rsnapshot[11360]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsnapshot daily: completed, but with some errors
Dec 31 04:35:45 <hostname> rsnapshot[11359]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsync returned 255 while processing user@host:/etc/
Dec 31 04:33:37 <hostname> rsnapshot[11353]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsync returned 255 while processing user@host:/home/user/
Dec 31 04:30:01 <hostname> CROND[11334]: (root) CMD (/usr/bin/rsnapshot daily)
Dec 30 04:36:05 <hostname> rsnapshot[8265]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsnapshot daily: completed, but with some errors
Dec 30 04:36:05 <hostname> rsnapshot[8264]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsync returned 255 while processing user@host:/etc/
Dec 30 04:33:58 <hostname> rsnapshot[8254]: /usr/bin/rsnapshot daily: ERROR:
/usr/bin/rsync returned 255 while processing user@host:/home/user/
Dec 30 04:30:02 <hostname> CROND[8237]: (root) CMD (/usr/bin/rsnapshot daily)
What I understand from this is `-u <unit>' only tells me what happened
when the unit file was (un)loaded, not what the process prints to the
log files. How do I get this information? Is cron special in this
regard?
2. I would like to filter logs that typically go into /var/log/secure
(or other similar files); how do I do that? Is grep my only option
for cases like these?
Thanks in advance for any answers.
Cheers,
--
Suvayu
Open source is the future. It sets us free.
4:52 p.m.
On 01.01.2014 20:57, Suvayu Ali wrote:
Now my questions:
1. How can I filter messages printed to the logs from my cron jobs? I
will try to explain by example:
$ journalctl -ru crond --since=-3d
-- Logs begin at Sun 2013-11-17 02:48:46 CET, end at Wed 2014-01-01 20:31:27 CET. --
# journalctl -u crond.service
Do it as root, and add .service suffix to crond.
Bash completion is your friend here, hitting TAB twice after -u option
will give you units that are recognized.
journalctl is user-aware and show only what's appropriate.
Mateusz Marzantowicz
5:08 p.m.
On Wed, Jan 01, 2014 at 11:52:04PM +0100, Mateusz Marzantowicz wrote:
On 01.01.2014 20:57, Suvayu Ali wrote:
> Now my questions:
>
> 1. How can I filter messages printed to the logs from my cron jobs? I
> will try to explain by example:
>
> $ journalctl -ru crond --since=-3d
> -- Logs begin at Sun 2013-11-17 02:48:46 CET, end at Wed 2014-01-01 20:31:27
CET. --
# journalctl -u crond.service
Do it as root, and add .service suffix to crond.
Bash completion is your friend here, hitting TAB twice after -u option
will give you units that are recognized.
journalctl is user-aware and show only what's appropriate.
Actually, if I'm in the systemd-journal group it is equivalent. That
said, I get the same output as root. I use bash-completion, and the
.service is optional when used with -u (as I mentioned in my email). I
tried with and without the .service. It works both ways for sendmail,
just not crond.
Cheers,
--
Suvayu
Open source is the future. It sets us free.
3:39 a.m.
On 02.01.2014 00:08, Suvayu Ali wrote:
On Wed, Jan 01, 2014 at 11:52:04PM +0100, Mateusz Marzantowicz
wrote:
> On 01.01.2014 20:57, Suvayu Ali wrote:
>
>> Now my questions:
>>
>> 1. How can I filter messages printed to the logs from my cron jobs? I
>> will try to explain by example:
>>
>> $ journalctl -ru crond --since=-3d
>> -- Logs begin at Sun 2013-11-17 02:48:46 CET, end at Wed 2014-01-01 20:31:27
CET. --
>
> # journalctl -u crond.service
>
> Do it as root, and add .service suffix to crond.
>
> Bash completion is your friend here, hitting TAB twice after -u option
> will give you units that are recognized.
>
> journalctl is user-aware and show only what's appropriate.
Actually, if I'm in the systemd-journal group it is equivalent. That
said, I get the same output as root. I use bash-completion, and the
.service is optional when used with -u (as I mentioned in my email). I
tried with and without the .service. It works both ways for sendmail,
just not crond.
Cheers,
OK, so it might be something else (maybe cron not producing log
messages?) I know it doesn't help much but in my case - it works - and I
get plenty of crond messages. I don't have sendmail installed so I have
no associated log messages.
Mateusz Marzantowicz
9:20 p.m.
On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com> wrote:
On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
>
> 2. I would like to filter logs that typically go into /var/log/secure
> (or other similar files); how do I do that?
SYSLOG_FACILITY=authpriv
Sorry. Just thought that I'd try it and it turms out that it takes the
facility as a number not as a name, so "SYSLOG_FACILITY=10".
4:32 a.m.
On Thu, Jan 02, 2014 at 03:20:38AM +0000, Tom H wrote:
On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com>
wrote:
> On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com>
wrote:
>>
>> 2. I would like to filter logs that typically go into /var/log/secure
>> (or other similar files); how do I do that?
>
> SYSLOG_FACILITY=authpriv
Sorry. Just thought that I'd try it and it turms out that it takes the
facility as a number not as a name, so "SYSLOG_FACILITY=10".
Thank you! This will be very helpful. Where is this documented? I
could not find this information in journalctl(1) or
systemd.jounal-fields(7); did I miss some other docs?
--
Suvayu
Open source is the future. It sets us free.
10:52 a.m.
On Thu, Jan 2, 2014 at 10:32 AM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
On Thu, Jan 02, 2014 at 03:20:38AM +0000, Tom H wrote:
> On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com> wrote:
>> On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com>
wrote:
>>>
>>> 2. I would like to filter logs that typically go into /var/log/secure
>>> (or other similar files); how do I do that?
>>
>> SYSLOG_FACILITY=authpriv
>
> Sorry. Just thought that I'd try it and it turms out that it takes the
> facility as a number not as a name, so "SYSLOG_FACILITY=10".
Thank you! This will be very helpful. Where is this documented? I
could not find this information in journalctl(1) or
systemd.jounal-fields(7); did I miss some other docs?
That SYSLOG_FACILITY has to be a number is from systemd.jounal-fields(7).
7:12 p.m.
On Thu, Jan 02, 2014 at 04:52:01PM +0000, Tom H wrote:
On Thu, Jan 2, 2014 at 10:32 AM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
> On Thu, Jan 02, 2014 at 03:20:38AM +0000, Tom H wrote:
>> On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com> wrote:
>>> On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
>>>>
>>>> 2. I would like to filter logs that typically go into /var/log/secure
>>>> (or other similar files); how do I do that?
>>>
>>> SYSLOG_FACILITY=authpriv
>>
>> Sorry. Just thought that I'd try it and it turms out that it takes the
>> facility as a number not as a name, so "SYSLOG_FACILITY=10".
>
> Thank you! This will be very helpful. Where is this documented? I
> could not find this information in journalctl(1) or
> systemd.jounal-fields(7); did I miss some other docs?
That SYSLOG_FACILITY has to be a number is from systemd.jounal-fields(7).
No, I mean which number corresponds to what facility. I don't even know
where to find a comprehensive list of all the facilities.
Okay I think while writing the email I found the list of facilities in
logger(1); but I still do not know where I can find the mapping between
these facilities with the numbers accepted by SYSLOG_FACILITY.
--
Suvayu
Open source is the future. It sets us free.
8:01 p.m.
On Fri, Jan 3, 2014 at 1:12 AM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
On Thu, Jan 02, 2014 at 04:52:01PM +0000, Tom H wrote:
> On Thu, Jan 2, 2014 at 10:32 AM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com>
wrote:
>> On Thu, Jan 02, 2014 at 03:20:38AM +0000, Tom H wrote:
>>> On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com> wrote:
>>>> On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
>>>>>
>>>>> 2. I would like to filter logs that typically go into
/var/log/secure
>>>>> (or other similar files); how do I do that?
>>>>
>>>> SYSLOG_FACILITY=authpriv
>>>
>>> Sorry. Just thought that I'd try it and it turms out that it takes the
>>> facility as a number not as a name, so "SYSLOG_FACILITY=10".
>>
>> Thank you! This will be very helpful. Where is this documented? I
>> could not find this information in journalctl(1) or
>> systemd.jounal-fields(7); did I miss some other docs?
>
> That SYSLOG_FACILITY has to be a number is from systemd.jounal-fields(7).
No, I mean which number corresponds to what facility. I don't even know
where to find a comprehensive list of all the facilities.
Okay I think while writing the email I found the list of facilities in
logger(1); but I still do not know where I can find the mapping between
these facilities with the numbers accepted by SYSLOG_FACILITY.
Sorry, misunderstood you...
Via google: http://wiki.gentoo.org/wiki/Rsyslog
9:44 a.m.
On Fri, Jan 03, 2014 at 02:01:26AM +0000, Tom H wrote:
On Fri, Jan 3, 2014 at 1:12 AM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
> On Thu, Jan 02, 2014 at 04:52:01PM +0000, Tom H wrote:
>> On Thu, Jan 2, 2014 at 10:32 AM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com>
wrote:
>>> On Thu, Jan 02, 2014 at 03:20:38AM +0000, Tom H wrote:
>>>> On Thu, Jan 2, 2014 at 3:00 AM, Tom H <tomh0665(a)gmail.com> wrote:
>>>>> On Wed, Jan 1, 2014 at 7:57 PM, Suvayu Ali
<fatkasuvayu+linux(a)gmail.com> wrote:
>>>>>>
>>>>>> 2. I would like to filter logs that typically go into
/var/log/secure
>>>>>> (or other similar files); how do I do that?
>>>>>
>>>>> SYSLOG_FACILITY=authpriv
>>>>
>>>> Sorry. Just thought that I'd try it and it turms out that it takes
the
>>>> facility as a number not as a name, so "SYSLOG_FACILITY=10".
>>>
>>> Thank you! This will be very helpful. Where is this documented? I
>>> could not find this information in journalctl(1) or
>>> systemd.jounal-fields(7); did I miss some other docs?
>>
>> That SYSLOG_FACILITY has to be a number is from systemd.jounal-fields(7).
>
> No, I mean which number corresponds to what facility. I don't even know
> where to find a comprehensive list of all the facilities.
>
> Okay I think while writing the email I found the list of facilities in
> logger(1); but I still do not know where I can find the mapping between
> these facilities with the numbers accepted by SYSLOG_FACILITY.
Sorry, misunderstood you...
Via google: http://wiki.gentoo.org/wiki/Rsyslog
Thank you Tom. As ever, Gentoo & Arch has the best documentation again!
--
Suvayu
Open source is the future. It sets us free.
1:49 p.m.
On Fri, Jan 3, 2014 at 3:44 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
On Fri, Jan 03, 2014 at 02:01:26AM +0000, Tom H wrote:
>
> Via google: http://wiki.gentoo.org/wiki/Rsyslog
Thank you Tom. As ever, Gentoo & Arch has the best documentation again!
You're welcome.
They do have excellent documentation!
But in this case they both have mistakes.
Gentoo says that "10" corresponds to "security" but
"security" is an
old alias of "auth". "10" should be listed as corresponding to
"authpriv".
Arch doesn't say that "9" corresponds to "cron", even though the
explanation is that it corresponds to a "clock daemon", and that it
actually corresponds to "9". You can look it up in "syslog.h" as
someone pointed out earlier or you can edit rsyslog.conf for rsyslog
to print out the facilities in their word and number forms in the
logs.
9:16 p.m.
On 14-01-02 20:12:28, Suvayu Ali wrote:
On Thu, Jan 02, 2014 at 04:52:01PM +0000, Tom H wrote:
...
> That SYSLOG_FACILITY has to be a number is from
> systemd.jounal-fields(7).
No, I mean which number corresponds to what facility. I don't even
know where to find a comprehensive list of all the facilities.
Okay I think while writing the email I found the list of facilities in
logger(1); but I still do not know where I can find the mapping
between these facilities with the numbers accepted by SYSLOG_FACILITY.
From `man 3 syslog`, <syslog.h>, so `locate syslog` and:
/usr/include/sys/syslog.h
--
____________________________________________________________________
TonyN.:' <mailto:tonynelson@georgeanelson.com>
' <http://www.georgeanelson.com/>
2:40 p.m.
Hi Tony,
On Thu, Jan 02, 2014 at 10:16:07PM -0500, Tony Nelson wrote:
On 14-01-02 20:12:28, Suvayu Ali wrote:
> On Thu, Jan 02, 2014 at 04:52:01PM +0000, Tom H wrote:
...
> > That SYSLOG_FACILITY has to be a number is from
> > systemd.jounal-fields(7).
>
> No, I mean which number corresponds to what facility. I don't even
> know where to find a comprehensive list of all the facilities.
>
> Okay I think while writing the email I found the list of facilities in
> logger(1); but I still do not know where I can find the mapping
> between these facilities with the numbers accepted by SYSLOG_FACILITY.
From `man 3 syslog`, <syslog.h>, so `locate syslog` and:
/usr/include/sys/syslog.h
Indeed, looking at the source is the only way to get this information.
It is a bit ridiculous I think, all this hoopla about making it easier
for the non-technical user, and then the useful docs happen to be in a
header file owned by glibc-headers!
Thanks a lot for pointing it to me.
:)
--
Suvayu
Open source is the future. It sets us free.
4:31 p.m.
On 14-01-03 15:40:11, Suvayu Ali wrote:
Hi Tony,
On Thu, Jan 02, 2014 at 10:16:07PM -0500, Tony Nelson wrote:
...
> From `man 3 syslog`, <syslog.h>, so `locate syslog` and:
>
> /usr/include/sys/syslog.h
Indeed, looking at the source is the only way to get this information.
It is a bit ridiculous I think, all this hoopla about making it easier
for the non-technical user, and then the useful docs happen to be in a
header file owned by glibc-headers!
Yup.
Thanks a lot for pointing it to me.
You're welcome.
--
____________________________________________________________________
TonyN.:' <mailto:tonynelson@georgeanelson.com>
' <http://www.georgeanelson.com/>
4:51 p.m.
Hi
On Fri, Jan 3, 2014 at 3:40 PM, Suvayu Ali wrote:
Indeed, looking at the source is the only way to get this
information.
It is a bit ridiculous I think, all this hoopla about making it easier
for the non-technical user, and then the useful docs happen to be in a
header file owned by glibc-headers!
File a bug report and one of the systemd developers can clarify the man
page. I agree that the current documentation is too brief
Rahul
5:15 a.m.
On Wed, 1 Jan 2014 20:57:38 +0100
Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
1. If your journal size is large, piping to grep is quite a bit slow.
http://www.freedesktop.org/software/systemd/man/systemd-journald.service....
SIGUSR2 ## crontab -e weekly rotate (or your preference),
help keep it manageable
___
Regards,
Frank
www.frankly3d.com
7:07 p.m.
Hi Frank,
On Thu, Jan 02, 2014 at 11:15:38AM +0000, Frank Murphy wrote:
On Wed, 1 Jan 2014 20:57:38 +0100
Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
> 1. If your journal size is large, piping to grep is quite a bit slow.
http://www.freedesktop.org/software/systemd/man/systemd-journald.service....
SIGUSR2 ## crontab -e weekly rotate (or your preference),
help keep it manageable
This is very interesting, I would love to use that. But I don't think I
understand how. What do I put in the crontab, how do I send this
SIGUSR2 signal to journald?
--
Suvayu
Open source is the future. It sets us free.
3:32 a.m.
On Fri, 3 Jan 2014 02:07:13 +0100
Suvayu Ali <fatkasuvayu+linux(a)gmail.com> wrote:
This is very interesting, I would love to use that. But I don't
think I understand how. What do I put in the crontab, how do I send
this SIGUSR2 signal to journald?
An easier method:
/etc/systemd/journald.conf
MaxFileSec=1week # what I use
http://www.freedesktop.org/software/systemd/man/journald.conf.html
___
Regards,
Frank
www.frankly3d.com
3772
days inactive
3774
days old
18 comments
6 participants
participants (6)
-
Frank Murphy -
Mateusz Marzantowicz -
Rahul Sundaram -
Suvayu Ali -
Tom H -
Tony Nelson