= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
== Detailed description == Let's change the default install to no longer install a syslog service by default — let's remove rsyslog from the "comps" default.
The journal has been around for a few releases and is well tested. F19 already enabled persistent journal logging on disk, thus all logs have been stored twice on disk, once in journal files and once in /var/log/messages. This feature hence recommends no longer installing rsyslog by default, leaving only the journal in place.
rsyslog will remain the recommended option to install if users require /var/log/messages, need support for the syslog network protocol, or need to enforce strict data lifecycle policies. It's sufficient to install and start rsyslog to get /var/log/messages and BSD syslog support.
Also see previous attempt: [1] and previous mailing list discussion at [2]
[1] https://fedoraproject.org/wiki/Features/systemd-journal [2] https://lists.fedoraproject.org/pipermail/devel/2012-October/172682.html
== Scope == Simply remove "rsyslog" from all default install groups in "comps".
Packages which strictly require /var/log/messages to exist might need updating to gain dependencies on some kind of syslog daemon (but they needed that before too, so this is mostly just bugfixing that's useful anyway). If any of the packages in the default install is one of those, we need to look at it in detail, and find a solution. However, currently no package of the default install is requiring a syslog implementation.
Some tools such as logcheck might need to be updated to process data from the journal instead of /var/log/message. This should be fairily easy as "journalctl" generates the same output as "cat /var/log/messages" previously did.
Proposal owners: Commit a change to "comps" to remove "rsyslog" from it. Drop in a file /var/log/README informing users where the log files went, and how do get to the same data as before.
Other developers: logcheck needs updating to stay useful. It needs to grep through the output of "journalctl", rather than /var/log/messages.
Release engineering: nothing really.
Policies and guidelines: Guidelines should clarify that /var/log/message doesn't exist on many systems, but that was already the case before -- so little changes. QA should add a few tests and release criteria about journal functionality.
Note that logrotate should stay in the default install, as it is required to rotate wtmp and btmp (the journal synchronously rotates before writing and does not require logrotate for operation). _______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
On Mon, Jul 15, 2013 at 10:11 AM, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
This doesn't stop you from having a traditional syslog service, it's just not installed in the desktop spins by default. "yum install rsyslog" will continue to provide you the status quo.
Peter
On 07/15/2013 11:14 AM, Peter Robinson wrote:
This doesn't stop you from having a traditional syslog service, it's just not installed in the desktop spins by default. "yum install rsyslog" will continue to provide you the status quo.
Yes, I know. But I spoke about *default*. I would like to have traditional syslog as default.
On Mon, Jul 15, 2013 at 01:04:00PM +0200, Miroslav Suchy wrote:
Yes, I know. But I spoke about *default*. I would like to have traditional syslog as default.
But that does not follow the concept systemd is taking. You are voting against this feature proposal then I guess :-)
On Mon, 15.07.13 10:14, Peter Robinson (pbrobinson@gmail.com) wrote:
On Mon, Jul 15, 2013 at 10:11 AM, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
This doesn't stop you from having a traditional syslog service, it's just not installed in the desktop spins by default. "yum install rsyslog" will continue to provide you the status quo.
We are talking about @core and @defeult btw. So this chnage applies to all spins, not only desktop, unless they explicitly readd rsyslog to their spins.
Lennart
On Mon, 15.07.13 11:11, Miroslav Suchý (msuchy@redhat.com) wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
Note that the journal is not optional, since we need it during early boot, in the initrd, late boot, and because it is connected to all stdout/stderr of all daemons to collect logs. It is deeply integrated into service management, to make sure the logs we collect are comprehensive. Due to this, it cannot be optional. (And yes, even people who dislike the journal and want nothing to do with it, do benefit from this, as this makes a lot more data available to their preferred syslog implementation than was ever before).
Hence, the choice between "journal by default + syslog optional" and "journal optional + syslog by default" does not exist. The choice between "journal by default + syslog by default" and "journal by default + syslog optional" however does.
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
Lennart
On 07/15/2013 01:23 PM, Lennart Poettering wrote:
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
Because you are able to read txt file with *anything*. But you are not always able to read binary files. Because journalctl is not available.
Example from real life: Sometimes it happen to me, that I'm fixing totally borked computer. And I have available only what is in memory - because HDD with /usr crashed, but machine is still running (and /var is readable) and with vt open. (BTW - coincidently it happened to me just one month ago). If I want to read last messages, I could not start journactl, but I can read /var/log/messages with bash builtins.
And since I'm not always choosing which machine I'm going to fixing, I would prefer that /var/log/messages is present as text file as default.
If somebody explicitly want to remove rsyslog and live only with journalctl, then let it be. And I do not care about other logs, they can be in binary forms, if you want. But I want to have /var/log/messages available in pure text form.
Because you are able to read txt file with *anything*. But you are not always able to read binary files. Because journalctl is not available.
It's true that if you want to debug/fix your Fedora system from a LiveCD, and the only available LiveCD you have around is a non-systemd distribution, it will be much easier to access plain text files logs than work with journalctl binary files.
It would be easier if systemd was available anywhere, but we're not there yet.
On Mon, 15.07.13 14:46, Miroslav Suchý (msuchy@redhat.com) wrote:
On 07/15/2013 01:23 PM, Lennart Poettering wrote:
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
Because you are able to read txt file with *anything*. But you are not always able to read binary files. Because journalctl is not available.
Example from real life: Sometimes it happen to me, that I'm fixing totally borked computer. And I have available only what is in memory - because HDD with /usr crashed, but machine is still running (and /var is readable) and with vt open. (BTW - coincidently it happened to me just one month ago). If I want to read last messages, I could not start journactl, but I can read /var/log/messages with bash builtins.
Well, assuming that bash is entirely in memory. And also, note that neither cat, nor cp, nor tail are actually bash builtins and will not work. It's pretty hard (though certainly possible) viewing files with just bash builtins.
But anyway, this is quite anecdotal.
And since I'm not always choosing which machine I'm going to fixing, I would prefer that /var/log/messages is present as text file as default.
It's not hard trying journalctl first, and then looking into /var/log/messages, if that didn't work. This is a particularly good idea, given that when you run into problems in the initrd and during early boot, then syslog won't be available yet anyway. Since most system issues probably happen during the initrd/early boot process it is hence a lot more useful to use journalctl anyway. If you care about debugging broken systems journaclt is almost certainly the better option.
(Let's also not forget that depending on the distro the default path for syslog messages differs already, for example sometimes is /var/log/syslog, so you already *now* have to check more than one place if you want to see the logs on any Linux machine you want. Oh, and ArchLinux defaults to journal-only already, so you need to check the journal anyway, if you want to deal with all popular Linux distributions).
If somebody explicitly want to remove rsyslog and live only with journalctl, then let it be. And I do not care about other logs, they can be in binary forms, if you want. But I want to have /var/log/messages available in pure text form.
Note that we do include journalctl on all our livecds. Also, most distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
Lennart
2013/7/15 Lennart Poettering mzerqung@0pointer.de
[...]
Well, assuming that bash is entirely in memory. And also, note that
neither cat, nor cp, nor tail are actually bash builtins and will not work. It's pretty hard (though certainly possible) viewing files with just bash builtins.
echo $(< /var/log/messages)
[...]
Note that we do include journalctl on all our livecds. Also, most distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
There are more Live-CDs in the wild without journalctl than Live-CDs with journalctl.
Regards, Thomas
On Mon, 15.07.13 16:29, Thomas Bendler (ml@bendler-net.de) wrote:
Note that we do include journalctl on all our livecds. Also, most distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
There are more Live-CDs in the wild without journalctl than Live-CDs with journalctl.
But there are certainly more Fedora-Live-CDs in the wild with journalctl than without...
Lennart
On Monday 15 July 2013 16:29:08 Thomas Bendler wrote:
2013/7/15 Lennart Poettering mzerqung@0pointer.de [...] Well, assuming that bash is entirely in memory. And also, note that neither cat, nor cp, nor tail are actually bash builtins and will not work. It's pretty hard (though certainly possible) viewing files with just bash builtins.
echo $(< /var/log/messages)
For what it's worth:
while read -r line; do echo "$line"; done < /var/log/messages
Will do the same and will work line by line instead of putting everything in one *gigantic* single line
Regards,
Marc Deop
On 07/15/2013 03:37 PM, Lennart Poettering wrote:
But anyway, this is quite anecdotal.
I was not laughing when I was on site.
If you are working on some computer game, you can target 90 % of available audience. Or even if pulse audio will work for 99 % of user, then it is fine.
But these core parts should work for 100 % users. In all cases, which are in scenarios you can imagine. And in 90 % of scenarios you can not imagine. Or in scenarios, which are anecdotal for you.
On Mon, 15.07.13 17:02, Miroslav Suchý (msuchy@redhat.com) wrote:
On 07/15/2013 03:37 PM, Lennart Poettering wrote:
But anyway, this is quite anecdotal.
I was not laughing when I was on site.
Hmm? Anecdotal certainly doesn't imply amusing?
If you are working on some computer game, you can target 90 % of available audience. Or even if pulse audio will work for 99 % of user, then it is fine.
But these core parts should work for 100 % users. In all cases, which are in scenarios you can imagine. And in 90 % of scenarios you can not imagine. Or in scenarios, which are anecdotal for you.
Well, if you open calculations like this one:
If you type "journalctl -b" you will already get useful output if you are stuck in the initrd and in early bootup, at which point /var/log/messages is not available yet, and won't be available for quite some time still. Given that these earlier phases of the boot process are a major place where things can go wrong, I am pretty sure "journalctl" is a ton more useful then /var/log/messages to debug such hosed machines...
But anyway, both of us, we can just make guesses which kind of errors are more likely. But please understand that nothing stops you from reinstalling the syslog implemention of your choice in addition to the defaults.
Lennart
On Mon, Jul 15, 2013 at 05:38:36PM +0200, Lennart Poettering wrote:
But anyway, this is quite anecdotal.
I was not laughing when I was on site.
Hmm? Anecdotal certainly doesn't imply amusing?
This is a language issue. An anecdote is a short or amusing story -- halfway to being a joke. Party conversation. "Anecdotal evidence" is therefore a derogatory term. However, we're used to using it so much in software / computer / tech terms that it can _also_ be used in a purely 'straight' sense with no insult meant, to simply say that it's a hard-to-reproduce story.
Hi Lennart,
I suppose someone should mention small flash-disk-only computers.
There traditionally we fling syslog messages to the serial console or a LRU buffer in RAM (often the dmesg buffer). The point is to avoid I/O on the flash memory. Syslog daemons tend to do a lot of fsync-ed I/O, which just chews up flash write cycles. With some configuration the syslog daemons can be made to not to fsync, and with additional configuration to write to the serial port or to the dmesg ring buffer.
These small computers aren't specialised embedded systems anymore -- if you buy a cheap ARM-based laptop then you are buying a such a system. Their increasing popularity is very much the reason ARM is becoming a top-teir architecture in Fedora. These systems are *cheap*, so they don't have the write cycles of an expensive SSD.
I'm not across journald at all. But the questions in my mind are:
- Is is possible to run journald without writing to disk; that is: to serial as text, or as binary to a ring buffer which can then by used by journalctl?
- When writing to disk does journald fsync, and if so can that be disabled by a non-guru laptop user?
- Is journalctl available from the dracut shell, so that we can get bug reports for early system failures? There is a lot more variation in small computers, and thus more early system failures.
Thank you for making the binary format portable between computers. Allowing a 32b ARM journal file to be displayed on a x86_64 desktop is very useful.
Thank you for your time, glen
On Sat, Jul 20, 2013 at 08:15:11AM +0930, Glen Turner wrote:
Hi Lennart,
I suppose someone should mention small flash-disk-only computers.
There traditionally we fling syslog messages to the serial console or a LRU buffer in RAM (often the dmesg buffer). The point is to avoid I/O on the flash memory. Syslog daemons tend to do a lot of fsync-ed I/O, which just chews up flash write cycles. With some configuration the syslog daemons can be made to not to fsync, and with additional configuration to write to the serial port or to the dmesg ring buffer.
These small computers aren't specialised embedded systems anymore -- if you buy a cheap ARM-based laptop then you are buying a such a system. Their increasing popularity is very much the reason ARM is becoming a top-teir architecture in Fedora. These systems are *cheap*, so they don't have the write cycles of an expensive SSD.
I'm not across journald at all. But the questions in my mind are:
I'm not Lennart, but I'll try to answer your questions:
- Is is possible to run journald without writing to disk; that is: to serial as text, or as binary to a ring buffer which can then by used by journalctl?
Yes, it's possible to keep journal completely in /run/ by setting Storage=volatile or not creating /var/log/journal at all. See journald.conf(5).
- When writing to disk does journald fsync, and if so can that be disabled by a non-guru laptop user?
Yes, see SyncIntervalSec in journald.conf(5).
- Is journalctl available from the dracut shell, so that we can get bug reports for early system failures? There is a lot more variation in small computers, and thus more early system failures.
Yes, dracut uses systemd and journald too.
Thank you for making the binary format portable between computers. Allowing a 32b ARM journal file to be displayed on a x86_64 desktop is very useful.
Yes, systemd should be completely portable between architectures.
Zbyszek
On Sat, Jul 20, 2013 at 01:02:00AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
- Is is possible to run journald without writing to disk; that is: to
serial as text, or as binary to a ring buffer which can then by used by journalctl?
Yes, it's possible to keep journal completely in /run/ by setting Storage=volatile or not creating /var/log/journal at all. See journald.conf(5).
Plus add ForwardToConsole=yes and you can have it write to a serial console.
But these core parts should work for 100 % users. In all cases, which are in scenarios you can imagine. And in 90 % of scenarios you can not imagine. Or in scenarios, which are anecdotal for you.
+1 - anything which reduces the tools we have to repair systems or investigate security issues, is a bad thing.
On Mon, Jul 15, 2013 at 03:37:00PM +0200, Lennart Poettering wrote:
Sometimes it happen to me, that I'm fixing totally borked computer. And I have available only what is in memory - because HDD with /usr crashed, but machine is still running (and /var is readable) and with vt open. (BTW - coincidently it happened to me just one month ago). If I want to read last messages, I could not start journactl, but I can read /var/log/messages with bash builtins.
Well, assuming that bash is entirely in memory. And also, note that neither cat, nor cp, nor tail are actually bash builtins and will not work. It's pretty hard (though certainly possible) viewing files with just bash builtins. But anyway, this is quite anecdotal.
Even in less extreme situations than this, it's a fair point that often one can't run binaries from the system where you need to look at the logs, and needing special tools (rather than just any viewer or editor) to analyze those logs _is_ increased pain. Now you need a systemd-aware rescue image, not just a tinylinux livecd that's kicking around the server room.
Soo....
Note that we do include journalctl on all our livecds. Also, most distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
Yeah. Let's put this in the release notes.
On Mon, 2013-07-15 at 12:27 -0400, Matthew Miller wrote:
Even in less extreme situations than this, it's a fair point that often one can't run binaries from the system where you need to look at the logs, and needing special tools (rather than just any viewer or editor) to analyze those logs _is_ increased pain. Now you need a systemd-aware rescue image, not just a tinylinux livecd that's kicking around the server room
Personally the situation I've found myself in most often is trying to debug a VM guest from a RHEL6 host. I'm able to mount the guest filesystem offline with libguestfs, but having something just a journal viewer backported to EPEL would be pretty neat.
What I've been doing mostly instead to debug early boot problems is adding systemd.log_target=kmsg to the console and redirecting the console to a file.
On Jul 15, 2013, at 7:37 AM, Lennart Poettering mzerqung@0pointer.de wrote:
If you care about debugging broken systems journaclt is almost certainly the better option.
So far I haven't seen anything in /var/log/messages that isn't in journalctl -xb. Is it fair to say that journalctl -xb should be a superset of messages (for the current boot)?
Chris Murphy
On Mon, 15.07.13 12:12, Chris Murphy (lists@colorremedies.com) wrote:
On Jul 15, 2013, at 7:37 AM, Lennart Poettering mzerqung@0pointer.de wrote:
If you care about debugging broken systems journaclt is almost certainly the better option.
So far I haven't seen anything in /var/log/messages that isn't in journalctl -xb. Is it fair to say that journalctl -xb should be a superset of messages (for the current boot)?
The journal will forward everything we have to a running syslog daemon (or actually, recent rsyslog turned this around and now pulls out everything it can). However, that only happens during late boot, i.e. after /var is up. The journal however works pretty much any time, it is already available in the initrd, and in early-boot. If your initrd hangs, you can type journalctl and it will give you log messages already.
This works, since the journal stores things to /run during early boot, and this is then flushed to /var as soon as that's available.
So, this isn't really about the amount of data, it's about when it is available. And if your boot-up fails, then journalctl is more likely to help you than /var/log/messages, since for that you'd first to have /var mounted... (well, beyond that it's also about the metadata. journalctl will always provide you with the full set of metadata, while /var/log/messages by default only contains teh actualy text messages, timezone-less timestamp, hostname, "tag" and PID).
Lennart
Note that we do include journalctl on all our livecds. Also, most distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
Not true, RHEL 6 and its friends do not include journalctl. So if I want to cover both Fedora default and RHEL default, now I do have to look at two places instead of one.
On 07/16/2013 02:56 AM, Ding Yi Chen wrote:
Also, most
distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
Not true, RHEL 6 and its friends do not include journalctl. So if I want to cover both Fedora default and RHEL default, now I do have to look at two places instead of one.
Or simply install rsyslog or syslog-ng and afaik you cant run rsyslog on AIX so same logic applies there and any other log solution are not available cross linux distros and cross another os.
JBG
----- Original Message -----
On 07/16/2013 02:56 AM, Ding Yi Chen wrote:
Also, most
distributions do include it in some way or another, and you do not need to boot systemd to use to it access your journal files.
Not true, RHEL 6 and its friends do not include journalctl. So if I want to cover both Fedora default and RHEL default, now I do have to look at two places instead of one.
Or simply install rsyslog or syslog-ng and afaik you cant run rsyslog on AIX so same logic applies there and any other log solution are not available cross linux distros and cross another os.
AIX has logger, which outputs to /var/log/messages http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.cm...
Regards,
Le Lun 15 juillet 2013 13:23, Lennart Poettering a écrit :
On Mon, 15.07.13 11:11, Miroslav Suchý (msuchy@redhat.com) wrote:
Hence, the choice between "journal by default + syslog optional" and "journal optional + syslog by default" does not exist. The choice between "journal by default + syslog by default" and "journal by default
- syslog optional" however does.
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
IMHO it is way too early to switch of syslog, as systemd+syslog hosed systems a few weeks ago, so the reliability of the systemd parts is anything but proven so far. And I'm quite sure there are scores of nasty surprises to find yet. For example, systemd still does not like being updated by yum, and I'm pretty sure no one looked at the journal parts when that happens.
On Mon, 15.07.13 15:34, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 13:23, Lennart Poettering a écrit :
On Mon, 15.07.13 11:11, Miroslav Suchý (msuchy@redhat.com) wrote:
Hence, the choice between "journal by default + syslog optional" and "journal optional + syslog by default" does not exist. The choice between "journal by default + syslog by default" and "journal by default
- syslog optional" however does.
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
IMHO it is way too early to switch of syslog, as systemd+syslog hosed systems a few weeks ago, so the reliability of the systemd parts is anything but proven so far. And I'm quite sure there are scores of nasty surprises to find yet. For example, systemd still does not like being updated by yum, and I'm pretty sure no one looked at the journal parts when that happens.
Any references to open bugs about this?
systemd is fine being updated by yum, with some exceptions for major updates between F18 and F19 for example, where you are supposed to use fedup anyway...
Also, let's really not intermingle topics here...
Lennart
Le Lun 15 juillet 2013 15:39, Lennart Poettering a écrit :
On Mon, 15.07.13 15:34, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 13:23, Lennart Poettering a écrit :
On Mon, 15.07.13 11:11, Miroslav Suchý (msuchy@redhat.com) wrote:
Hence, the choice between "journal by default + syslog optional" and "journal optional + syslog by default" does not exist. The choice between "journal by default + syslog by default" and "journal by
default
- syslog optional" however does.
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
IMHO it is way too early to switch of syslog, as systemd+syslog hosed systems a few weeks ago, so the reliability of the systemd parts is anything but proven so far. And I'm quite sure there are scores of nasty surprises to find yet. For example, systemd still does not like being updated by yum, and I'm pretty sure no one looked at the journal parts when that happens.
Any references to open bugs about this?
not really, since one of the things I've not figured out yet is how to get reliable logs of late shutdown stages
On Mon, 15.07.13 15:42, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Any references to open bugs about this?
not really, since one of the things I've not figured out yet is how to get reliable logs of late shutdown stages
See:
http://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1
There's the general problem that once /var is read-only we cannot really store logs anywhere anymore that survive the reboot. On our TODO list is to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
Lennart
Le Lun 15 juillet 2013 15:47, Lennart Poettering a écrit :
On Mon, 15.07.13 15:42, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Any references to open bugs about this?
not really, since one of the things I've not figured out yet is how to get reliable logs of late shutdown stages
See:
http://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1
There's the general problem that once /var is read-only we cannot really store logs anywhere anymore that survive the reboot. On our TODO list is to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
And of course the second problem is that after a forced (reset) reboot shutdown works fine one again, and I can never predict which shutdown will fail
Le Lun 15 juillet 2013 15:47, Lennart Poettering a écrit :
There's the general problem that once /var is read-only we cannot really store logs anywhere anymore that survive the reboot. On our TODO list is to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
BTW another case I've seen where systemd disappointed be, that's when in case of problem, instead of trying to salvage logs at the next boot, it just considers the log file corrupted and ignores it. (there was a useless message about it, I have zero wish to try to salvage a binary data file manually)
So where in a pre-journald world I'd have found everything till the crash in /var/log messages (possibly, with some corrupted lines in the middle I would just ignore), in a journald world there is a whole blackhole right where the incident occurred, and the data is not parsed by journald nor relayed to syslog.
This is seriously underwhelming.
On Tue, 16.07.13 11:37, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 15:47, Lennart Poettering a écrit :
There's the general problem that once /var is read-only we cannot really store logs anywhere anymore that survive the reboot. On our TODO list is to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
BTW another case I've seen where systemd disappointed be, that's when in case of problem, instead of trying to salvage logs at the next boot, it just considers the log file corrupted and ignores it. (there was a useless message about it, I have zero wish to try to salvage a binary data file manually)
I am pretty sure that is just a misunderstanding. Note that journald (i.e. the *server* side) will immediately move away (i.e. "rotate") all journal files that it finds have not been set to "offline" when it starts up before writing, in order to make sure that it will not interfere with journal files that are incompletely written (possibly further corrupting them). However journalctl (i.e. the *client* side) will still access the file, and interleave it with all others it finds, and show it to you as far as that's possible.
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
Lennart
Le Mar 16 juillet 2013 13:25, Lennart Poettering a écrit :
On Tue, 16.07.13 11:37, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 15:47, Lennart Poettering a écrit :
There's the general problem that once /var is read-only we cannot
really
store logs anywhere anymore that survive the reboot. On our TODO list
is
to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
BTW another case I've seen where systemd disappointed be, that's when in case of problem, instead of trying to salvage logs at the next boot, it just considers the log file corrupted and ignores it. (there was a useless message about it, I have zero wish to try to salvage a binary data file manually)
I am pretty sure that is just a misunderstanding.
I certainly hope so :)
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
However even if that's the case that means some events just got hidden in a file only journalctl will consult, and not relayed to syslog (as they should)
On Tue, 16.07.13 13:38, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
However even if that's the case that means some events just got hidden in a file only journalctl will consult, and not relayed to syslog (as they should)
No. That's not correct.
journald will immediately forward all messages it gets to a running syslog daemon if there is one -- it will actually do so before even writing the message to disk!
Of course, as mentioned the newest rsyslog does not rely on the forwarding anymore but instead pulls the data out of the journal on its own. It hence will get the exact same stream of data that journalctl gets, i.e. also the compelte stream.
Lennart
On Tue, Jul 16, 2013 at 01:25:54PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 11:37, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 15:47, Lennart Poettering a écrit :
There's the general problem that once /var is read-only we cannot really store logs anywhere anymore that survive the reboot. On our TODO list is to optionally store all logs generated beyond that point in some UEFI variable, and collect it on next boot.
BTW another case I've seen where systemd disappointed be, that's when in case of problem, instead of trying to salvage logs at the next boot, it just considers the log file corrupted and ignores it. (there was a useless message about it, I have zero wish to try to salvage a binary data file manually)
I am pretty sure that is just a misunderstanding. Note that journald (i.e. the *server* side) will immediately move away (i.e. "rotate") all journal files that it finds have not been set to "offline" when it starts up before writing, in order to make sure that it will not interfere with journal files that are incompletely written (possibly further corrupting them). However journalctl (i.e. the *client* side) will still access the file, and interleave it with all others it finds, and show it to you as far as that's possible.
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
On Tue, Jul 16, 2013 at 01:25:54PM +0200, Lennart Poettering wrote:
I am pretty sure that is just a misunderstanding. Note that journald (i.e. the *server* side) will immediately move away (i.e. "rotate") all journal files that it finds have not been set to "offline" when it starts up before writing, in order to make sure that it will not interfere with journal files that are incompletely written (possibly further corrupting them). However journalctl (i.e. the *client* side) will still access the file, and interleave it with all others it finds, and show it to you as far as that's possible.
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
Will journalctl also mention that there is a broken jorunal file? Does it support to "fsck" the journal or to show the not properly referenced data to the admin?
Regards Till
On Tue, 16.07.13 14:41, Till Maas (opensource@till.name) wrote:
On Tue, Jul 16, 2013 at 01:25:54PM +0200, Lennart Poettering wrote:
I am pretty sure that is just a misunderstanding. Note that journald (i.e. the *server* side) will immediately move away (i.e. "rotate") all journal files that it finds have not been set to "offline" when it starts up before writing, in order to make sure that it will not interfere with journal files that are incompletely written (possibly further corrupting them). However journalctl (i.e. the *client* side) will still access the file, and interleave it with all others it finds, and show it to you as far as that's possible.
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
Will journalctl also mention that there is a broken jorunal file? Does it support to "fsck" the journal or to show the not properly referenced data to the admin?
journalctl will not show you that, but libraries do see this. The reason we don't show this in journalctl is that when you "live" follow the output of a journal being written then the file frequently has half-written entries, which however will quickly be corrected as the entry is completed.
There's "journalctl --verify" whose main purpose is to verify FSS seals. It will also check the integraty of the data structures in a journal however.
Lennart
On Tue, Jul 16, 2013 at 04:21:43PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 14:41, Till Maas (opensource@till.name) wrote:
On Tue, Jul 16, 2013 at 01:25:54PM +0200, Lennart Poettering wrote:
I am pretty sure that is just a misunderstanding. Note that journald (i.e. the *server* side) will immediately move away (i.e. "rotate") all journal files that it finds have not been set to "offline" when it starts up before writing, in order to make sure that it will not interfere with journal files that are incompletely written (possibly further corrupting them). However journalctl (i.e. the *client* side) will still access the file, and interleave it with all others it finds, and show it to you as far as that's possible.
So yeah, you could say that journald will 'ignore' the file. But journalctl won't, it will show them to you. And that's *good* that way. That's how it *should* be.
Will journalctl also mention that there is a broken jorunal file? Does it support to "fsck" the journal or to show the not properly referenced data to the admin?
journalctl will not show you that, but libraries do see this. The reason we don't show this in journalctl is that when you "live" follow the output of a journal being written then the file frequently has half-written entries, which however will quickly be corrected as the entry is completed.
But in case journald decided to rotate a journal because it is not clean now new entries are to be expected there. Therefore journalctl should know that the journal is broken.
There's "journalctl --verify" whose main purpose is to verify FSS seals. It will also check the integraty of the data structures in a journal however.
What can be done if the verification fails? How can one get the incomplete data in case journald rotated a journal?
Regards Till
On Tue, 16.07.13 17:57, Till Maas (opensource@till.name) wrote:
Will journalctl also mention that there is a broken jorunal file? Does it support to "fsck" the journal or to show the not properly referenced data to the admin?
journalctl will not show you that, but libraries do see this. The reason we don't show this in journalctl is that when you "live" follow the output of a journal being written then the file frequently has half-written entries, which however will quickly be corrected as the entry is completed.
But in case journald decided to rotate a journal because it is not clean now new entries are to be expected there. Therefore journalctl should know that the journal is broken.
journalctl actually has an inotify watch on the journal dir. It will notice when a file is rotated and will immeidately add the new journal file that has been created as replaced to the files it watches and interleaves. This is all hidden inside the library btw, to ensure that clients do not need to be aware of rotation or anything and always see the full set of logs, regardless what is currently going on.
There's "journalctl --verify" whose main purpose is to verify FSS seals. It will also check the integraty of the data structures in a journal however.
What can be done if the verification fails? How can one get the incomplete data in case journald rotated a journal?
The tool will tell you how much of the file could be verified, starting from the beginning.
For tail corruptions your normal journalctl tool will provide you with as much information as is possible to salvage from the file. It will output the last complete log line and then finish. This is pretty close to how good you can get.
Things are different for corruptions in the middle. We have no nice tool for salvaging data from such corruption, but they could be written relatively easily. However, since they are highly unlikely due to the "append-only" model of the journal this hasn't been on our TODO list.
Lennart
On Monday 15 July 2013 13:23:29 Lennart Poettering wrote:
Hence, the choice between "journal by default + syslog optional" and "journal optional + syslog by default" does not exist. The choice between "journal by default + syslog by default" and "journal by default
- syslog optional" however does.
I would choose "journal by default + syslog by default". In fact, I would rather have systemd write text fies... (if binary files are *needed* I would even dare to say that we should write the text version as well)
But anyway, I'd still like to hear the technical reasoning for the opinion you expressed.
Technical reason? It's been already mentioned on this thread:
You can read text files with whatever reader you'd like (or even shell built-ins) rather than relying on a binary reader.
I've read the "Benefits to Fedora section" and I don't think they are significant enough to remove syslog from the default installation.
Just my two cents,
Regards,
Marc Deop
+1 - same here. You're far from being alone.
I'm still trying to get used to the new systemd in Fedora and still trying to think why I need it. Altogether for my day to day use I find it as added complexity with no real benefit cerca f15.
Unix/Linux for me is the simplicity of text files. If I lose the simplicity of text files I just wonder what is left for me? A bunch of vague files in a binary format I need complex tools to decipher? Might as well just install win7 and utilize my gfx card.
Same happened to me when I moved from being a big fan of KDE to LXDE. Why is that? KDE became just too heavy, too much **** I just didn't need. LXDE just lets me open a terminal, a web browser, a music player and get my **** done with.
A change like that is something probably the ubuntu community would like to adopt. A change that would just give me yet another reason to not use ubuntu. Linux is text files. It's grep, sed, bash, awk and all the other amazing text manipulation utilities we have.
A change like that might actually make me move away from Fedora after being loyal to it for as long as it existed. And I know I will not be alone.
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
On Mon, Jul 15, 2013 at 7:11 PM, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/**wiki/Changes/NoDefaultSysloghttps://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
-- Miroslav Suchy Red Hat, Software Engineer -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.**org/mailman/listinfo/develhttps://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, 16.07.13 00:55, Dan Fruehauf (malkodan@gmail.com) wrote:
+1 - same here. You're far from being alone.
I'm still trying to get used to the new systemd in Fedora and still trying to think why I need it. Altogether for my day to day use I find it as added complexity with no real benefit cerca f15.
Unix/Linux for me is the simplicity of text files. If I lose the simplicity of text files I just wonder what is left for me? A bunch of vague files in a binary format I need complex tools to decipher? Might as well just install win7 and utilize my gfx card.
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix, for example in wtmp and utmp. We have binary files in /etc, and everywhere else.
journalctl is certainly not a "complex tool" to understand. Command lines usually get shorter by using it rather than the equivalent for /var/log/messages.
"cat /var/log/messages" becomes just "journalctl"
"tail -f /var/log/messages" becomes "journalctl -f"
"tail -n 100 /var/log/messages" becomes "journalctl -n 100"
"grep foo /var/log/messages" becomes journalctl | grep foo"
And the outputs of these files are the exact same text streams you are used to. However, enhanced with a lot of niceties that make them more user friendly. For example, you get colors based on the log level, or there's a line drawn between reboots. You get the time zone corrected, and you get unconditional PID data, you can filter very very easy, the data is unfakable and so on.
Just think about this:
"journalctl -b" shows you output of the current boot only
"journalctl --since=today" shows you the output of today only
"journalctl -p notice" shows you only notice and error messages (i.e. all the important stuff)
"journaclctl -u crond" shows you only the messages from cron.
... and so on.
Now try to think how hard it is to express queries the same way on classic syslog. And how slow they become on larger database because they aren't indexed.
journalctl makes a lot of things easier, much easier. If you say it is complex or difficult to use I am pretty sure you never had a closer look at it.
(Also, I am not sure what you mean by "vague". Please note that the file format is fully documented: http://www.freedesktop.org/wiki/Software/systemd/journal-files/ also, we commited to an API for them and more)
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
Lennart
On Monday 15 July 2013 17:55:34 Lennart Poettering wrote:
On Tue, 16.07.13 00:55, Dan Fruehauf (malkodan@gmail.com) wrote:
+1 - same here. You're far from being alone.
I'm still trying to get used to the new systemd in Fedora and still trying to think why I need it. Altogether for my day to day use I find it as added complexity with no real benefit cerca f15.
Unix/Linux for me is the simplicity of text files. If I lose the simplicity of text files I just wonder what is left for me? A bunch of vague files in a binary format I need complex tools to decipher? Might as well just install win7 and utilize my gfx card.
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix, for example in wtmp and utmp. We have binary files in /etc, and everywhere else.
journalctl is certainly not a "complex tool" to understand. Command lines usually get shorter by using it rather than the equivalent for /var/log/messages.
"cat /var/log/messages" becomes just "journalctl"
"tail -f /var/log/messages" becomes "journalctl -f"
"tail -n 100 /var/log/messages" becomes "journalctl -n 100"
"grep foo /var/log/messages" becomes journalctl | grep foo"
And the outputs of these files are the exact same text streams you are used to. However, enhanced with a lot of niceties that make them more user friendly. For example, you get colors based on the log level, or there's a line drawn between reboots. You get the time zone corrected, and you get unconditional PID data, you can filter very very easy, the data is unfakable and so on.
Just think about this:
"journalctl -b" shows you output of the current boot only
"journalctl --since=today" shows you the output of today only
"journalctl -p notice" shows you only notice and error messages (i.e. all the important stuff)
"journaclctl -u crond" shows you only the messages from cron.
... and so on.
Now try to think how hard it is to express queries the same way on classic syslog. And how slow they become on larger database because they aren't indexed.
journalctl makes a lot of things easier, much easier. If you say it is complex or difficult to use I am pretty sure you never had a closer look at it.
(Also, I am not sure what you mean by "vague". Please note that the file format is fully documented: http://www.freedesktop.org/wiki/Software/systemd/journal-files/ also, we commited to an API for them and more)
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
Some may say that you are making things harder by introducing tools (thus new things to learn) that are not really needed on a day to day basis (which is, in fact, what Dan wrote).
However, I don't think we are discussing here whether we keep systemd or not rather than syslog, are we?
;-)
Regards,
Marc Deop
And the outputs of these files are the exact same text streams you are used to. However, enhanced with a lot of niceties that make them more user friendly. For example, you get colors based on the log level, or there's a line drawn between reboots. You get the time zone corrected, and you get unconditional PID data, you can filter very very easy, the data is unfakable and so on.
So, despite what you just said, the output is *not* the exact same text, and thus not a direct replacement for it?
/me shudders to think of all the sed/awk/perl scripts you're planning on breaking.
I still vote "no" on dropping a default /var/log/messages.
On Mon, 15.07.13 14:20, DJ Delorie (dj@redhat.com) wrote:
And the outputs of these files are the exact same text streams you are used to. However, enhanced with a lot of niceties that make them more user friendly. For example, you get colors based on the log level, or there's a line drawn between reboots. You get the time zone corrected, and you get unconditional PID data, you can filter very very easy, the data is unfakable and so on.
So, despite what you just said, the output is *not* the exact same text, and thus not a direct replacement for it?
The color stuff is turned off if you pipe things. So it *is* the the same ouput if you process things with any tool you like...
/me shudders to think of all the sed/awk/perl scripts you're planning on breaking.
They don't break. It's like color ls. Just because you nowadays get color output on the terminal this doesn't mean it broke all the scripts that parsed ls output, because "ls" much like journalctl turns off all fancy shit if you output to a pipe.
I still vote "no" on dropping a default /var/log/messages.
It's not a democracy btw, it's about convincing FESCO for the change or against it.
Lennart
On Jul 15, 2013, at 9:55 AM, Lennart Poettering mzerqung@0pointer.de wrote:
"tail -f /var/log/messages" becomes "journalctl -f"
It's so useful I would like this a default during anaconda installations in one of the shells.
journalctl makes a lot of things easier, much easier. If you say it is complex or difficult to use I am pretty sure you never had a closer look at it.
I find it quite a bit easier to use and have disabled rsyslog as part of my new setup procedure since F18; but that's only after using journalctl and messages side by side for a while and finding, so far, that journalctl contains more information but is easier to filter than messages.
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
I think the simple reference is in regards to the implied rewiring of biomatter that'll be needed to drop rsyslog.
Chris Murphy
On Tue, Jul 16, 2013 at 1:55 AM, Lennart Poettering mzerqung@0pointer.dewrote:
On Tue, 16.07.13 00:55, Dan Fruehauf (malkodan@gmail.com) wrote:
+1 - same here. You're far from being alone.
The +1 BTW was on Miroslav's comment. Obviously I'm against that change.
I'm still trying to get used to the new systemd in Fedora and still
trying
to think why I need it. Altogether for my day to day use I find it as
added
complexity with no real benefit cerca f15.
Unix/Linux for me is the simplicity of text files. If I lose the
simplicity
of text files I just wonder what is left for me? A bunch of vague files
in
a binary format I need complex tools to decipher? Might as well just install win7 and utilize my gfx card.
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix, for example in wtmp and utmp. We have binary files in /etc, and everywhere else.
And the reason for that being? I have no idea either, it's probably too old to be dug. Howver yes, I'd like these to be text based as well.
journalctl is certainly not a "complex tool" to understand. Command lines usually get shorter by using it rather than the equivalent for /var/log/messages.
People might also argue the event viewer in windows is not a "complex tool". See where it got them. Mind you event viewer is probably a bit more mature than journalctl.
"cat /var/log/messages" becomes just "journalctl"
"tail -f /var/log/messages" becomes "journalctl -f"
"tail -n 100 /var/log/messages" becomes "journalctl -n 100"
"grep foo /var/log/messages" becomes journalctl | grep foo"
All of these examples add a magnitude of complexity in terms of code. I'm unconvinced.
And the outputs of these files are the exact same text streams you are used to. However, enhanced with a lot of niceties that make them more user friendly. For example, you get colors based on the log level, or there's a line drawn between reboots. You get the time zone corrected, and you get unconditional PID data, you can filter very very easy, the data is unfakable and so on.
Just think about this:
"journalctl -b" shows you output of the current boot only
"journalctl --since=today" shows you the output of today only
"journalctl -p notice" shows you only notice and error messages (i.e. all the important stuff)
"journaclctl -u crond" shows you only the messages from cron.
... and so on.
Now try to think how hard it is to express queries the same way on classic syslog. And how slow they become on larger database because they aren't indexed.
journalctl makes a lot of things easier, much easier. If you say it is complex or difficult to use I am pretty sure you never had a closer look at it.
(Also, I am not sure what you mean by "vague". Please note that the file format is fully documented: http://www.freedesktop.org/wiki/Software/systemd/journal-files/ also, we commited to an API for them and more)
I'm pretty sure somewhere also the windows binary format is documented.
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
Nice query tools? Lets go with a RDBMS to store the logs then?
And for a short story about a system (I had nothing to do with) which based their log files in a RDBMS and their configuration in XML format. Failed miserably. The end.
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
Lennart
-- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, Jul 16, 2013 at 09:13:24AM +1000, Dan Fruehauf wrote:
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
A stable, documented journal format is a prerequisite for this feature.
http://www.freedesktop.org/wiki/Software/systemd/journal-files/
That page outlines the plan for compatibility and for future extensions, and it seems reasonable to me.
On Mon, Jul 15, 2013 at 07:25:28PM -0400, Matthew Miller wrote:
On Tue, Jul 16, 2013 at 09:13:24AM +1000, Dan Fruehauf wrote:
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
A stable, documented journal format is a prerequisite for this feature.
http://www.freedesktop.org/wiki/Software/systemd/journal-files/
That page outlines the plan for compatibility and for future extensions, and it seems reasonable to me.
Just to add to that: the format has already been sucessfully extended twice: first to add XZ compression of fields, and second time to add forward sealing.
Zbyszek
On 07/15/2013 07:25 PM, Matthew Miller wrote:
On Tue, Jul 16, 2013 at 09:13:24AM +1000, Dan Fruehauf wrote:
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
A stable, documented journal format is a prerequisite for this feature.
http://www.freedesktop.org/wiki/Software/systemd/journal-files/
That page outlines the plan for compatibility and for future extensions, and it seems reasonable to me.
FWIW, the 'file' utility does not have the magic to recognize systemd journal files---it just sees them as generic 'data'. I filed a bugzilla report and proposed fix:
On Wed, Jul 17, 2013 at 11:24:14AM -0400, Przemek Klosowski wrote:
On 07/15/2013 07:25 PM, Matthew Miller wrote:
On Tue, Jul 16, 2013 at 09:13:24AM +1000, Dan Fruehauf wrote:
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
A stable, documented journal format is a prerequisite for this feature.
http://www.freedesktop.org/wiki/Software/systemd/journal-files/
That page outlines the plan for compatibility and for future extensions, and it seems reasonable to me.
FWIW, the 'file' utility does not have the magic to recognize systemd journal files---it just sees them as generic 'data'. I filed a bugzilla report and proposed fix:
It's been fixed upstream a while ago in http://bugs.gw.com/view.php?id=258.
Zbyszek
On Tue, 16.07.13 09:13, Dan Fruehauf (malkodan@gmail.com) wrote:
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix, for example in wtmp and utmp. We have binary files in /etc, and everywhere else.
And the reason for that being? I have no idea either, it's probably too old to be dug. Howver yes, I'd like these to be text based as well.
Oh, the reasons are pretty well-known for those cases. For example wtmp/utmp is binary so that utmp works nicely as sparse file and the entries may be accessed using the UID number as seek index (multipled by the fixed entry size). So, it's about indexing, exactly as for journal files. The Unix guys back then chose binary when it made sense for their immediate technical requirements. And for us it's the exact same.
journalctl is certainly not a "complex tool" to understand. Command lines usually get shorter by using it rather than the equivalent for /var/log/messages.
People might also argue the event viewer in windows is not a "complex tool". See where it got them. Mind you event viewer is probably a bit more mature than journalctl.
Well, if I cannot convince you to even try it out, and you insist on conflating the windows log viewer with journalctl (which is about as smart as conflating microsoft word with vi), then I guess there's no point in discussing this with you.
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
Nice query tools? Lets go with a RDBMS to store the logs then?
Nobody suggests that.
And for a short story about a system (I had nothing to do with) which based their log files in a RDBMS and their configuration in XML format. Failed miserably. The end.
We do neither.
Another question while we're at it - what happens if and when you decide to change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based files?
We documented the format, and it is quite extensible. The sources are open. Altogether this are the best guarantees that the stuff stays stable and accessible.
Lennart
On Tue, Jul 16, 2013 at 9:51 AM, Lennart Poettering mzerqung@0pointer.dewrote:
On Tue, 16.07.13 09:13, Dan Fruehauf (malkodan@gmail.com) wrote:
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix,
for
example in wtmp and utmp. We have binary files in /etc, and everywhere else.
And the reason for that being? I have no idea either, it's probably too
old
to be dug. Howver yes, I'd like these to be text based as well.
Oh, the reasons are pretty well-known for those cases. For example wtmp/utmp is binary so that utmp works nicely as sparse file and the entries may be accessed using the UID number as seek index (multipled by the fixed entry size). So, it's about indexing, exactly as for journal files. The Unix guys back then chose binary when it made sense for their immediate technical requirements. And for us it's the exact same.
What made sense back in the day is today yet another vague decision that was being dragged with us for decades - "because it made sense at the time". Going from text to binary is easy. Always easy to break things. Going back to text (if we want to) will be difficult. The above example is a really good one for why not to go binary, by all means.
journalctl is certainly not a "complex tool" to understand. Command lines usually get shorter by using it rather than the equivalent for /var/log/messages.
People might also argue the event viewer in windows is not a "complex tool". See where it got them. Mind you event viewer is probably a bit
more
mature than journalctl.
Well, if I cannot convince you to even try it out, and you insist on conflating the windows log viewer with journalctl (which is about as smart as conflating microsoft word with vi), then I guess there's no point in discussing this with you.
Perhaps there isn't. On the day I will boot a Linux system and have no /var/log/messages out of the box, it'll be time for a change.
By all means any windows person I know would much rather have logs as plain text. Not the other way around.
Lets try to keep things simple. This is why we use Fedora. This is
why I
use Fedora.
We are certainly making things simpler by introducing nice query tools like journalctl and by reducing the number of packages we install by default, and by removing redundancy.
Nice query tools? Lets go with a RDBMS to store the logs then?
Nobody suggests that.
What's wrong with RDBMS? You only need to "echo 'select * from log' | SOME_SQL_PROGRAM | less", it's exactly the same and you also have all the querying capabilities of a RDBMS.
And for a short story about a system (I had nothing to do with) which
based
their log files in a RDBMS and their configuration in XML format. Failed miserably. The end.
We do neither.
You do neither, but you follow the very same path.
Another question while we're at it - what happens if and when you decide
to
change your binary format for storing files for whatever reason? Is that when we understand that we should have been left with just text based
files?
We documented the format, and it is quite extensible. The sources are open. Altogether this are the best guarantees that the stuff stays stable and accessible.
Thanks, I'd still prefer it text.
Lennart
-- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On 07/15/2013 11:20 PM, Dan Fruehauf wrote:
By all means any windows person I know would much rather have logs as plain text. Not the other way around.
I am not a windows person but I liked the way event viewer allowed to select and re-sort entries, even with their very limited capabilities.
In contrast, I kept feeling annoyed by the difficulty of doing simple things with /var/log/* files: for instance, a collated, time-sorted excerpt of events from all files during the last weekend (hint: grep and sort don't quite suffice; also, /var/log/* doesn't work unless you like the utmp/wtmp junk in your output).
On Jul 15, 2013 5:51 PM, "Lennart Poettering" mzerqung@0pointer.de wrote:
On Tue, 16.07.13 09:13, Dan Fruehauf (malkodan@gmail.com) wrote:
Well, there are certain things on Unix that are text files and many things that are not. Binary log files have a long tradition on Unix,
for
example in wtmp and utmp. We have binary files in /etc, and everywhere else.
And the reason for that being? I have no idea either, it's probably too
old
to be dug. Howver yes, I'd like these to be text based as well.
Oh, the reasons are pretty well-known for those cases. For example wtmp/utmp is binary so that utmp works nicely as sparse file and the entries may be accessed using the UID number as seek index (multipled by the fixed entry size). So, it's about indexing, exactly as for journal files. The Unix guys back then chose binary when it made sense for their immediate technical requirements. And for us it's the exact same.
I don't believe that was the reason, since when utmp was invented, Unix had 16-bit UIDs and did not have sparse files. On the other hand, I don't know the actual reason.
Eric
On 07/15/2013 10:55 AM, Dan Fruehauf wrote:
+1 - same here. You're far from being alone.
I'm still trying to get used to the new systemd in Fedora and still trying to think why I need it. Altogether for my day to day use I find it as added complexity with no real benefit cerca f15.
Unix/Linux for me is the simplicity of text files. If I lose the simplicity of text files I just wonder what is left for me? A bunch of vague files in a binary format I need complex tools to decipher? Might as well just install win7 and utilize my gfx card.
Same happened to me when I moved from being a big fan of KDE to LXDE. Why is that? KDE became just too heavy, too much **** I just didn't need. LXDE just lets me open a terminal, a web browser, a music player and get my **** done with.
A change like that is something probably the ubuntu community would like to adopt. A change that would just give me yet another reason to not use ubuntu. Linux is text files. It's grep, sed, bash, awk and all the other amazing text manipulation utilities we have.
A change like that might actually make me move away from Fedora after being loyal to it for as long as it existed. And I know I will not be alone.
Lets try to keep things simple. This is why we use Fedora. This is why I use Fedora.
On Mon, Jul 15, 2013 at 7:11 PM, Miroslav Suchý <msuchy@redhat.com mailto:msuchy@redhat.com> wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote: = Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org> No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.) The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default. My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option. -- Miroslav Suchy Red Hat, Software Engineer -- devel mailing list devel@lists.fedoraproject.org <mailto:devel@lists.fedoraproject.org> https://admin.fedoraproject.org/mailman/listinfo/devel
+1 for me too.
On Mon, Jul 15, 2013 at 11:11:17AM +0200, Miroslav Suchý wrote:
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
Can you elaborate on why?
I'll accept "changing where sysadmins look for log data is expensive" as an answer, but wonder if you have something else as well.
I know -- first hand, as a sysadmin! -- that this kind of change is disruptive. However, the journal has many benefits from a sysadmin point of view (see http://www.youtube.com/watch?v=i4CACB7paLc), and I think it's within Fedora's mission and Features/First mandate to explore it and see how we can bring those benefits while also working to reduce the transition cost.
On 15.07.2013 11:11, Miroslav Suchý wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
Maybe Fedora developers can arrange some simple survey. Only one question: do you know what is journalctl and what's it's purpose? If more than 50% of users (sysadmins, developers and others) answers "yes", than maybe it's time to switch. Six out of nine of my colleagues which do server administration haven't even heard of new syslogd replacement. They just don't care right now.
Mateusz Marzantowicz
On Mon, Jul 15, 2013 at 07:48:24PM +0200, Mateusz Marzantowicz wrote:
Maybe Fedora developers can arrange some simple survey. Only one question: do you know what is journalctl and what's it's purpose? If more than 50% of users (sysadmins, developers and others) answers "yes", than maybe it's time to switch. Six out of nine of my colleagues which do server administration haven't even heard of new syslogd replacement. They just don't care right now.
If we're going to do a survey, I can think of bigger questions to ask than that. :) But, in any case, this is certainly not the way Fedora is supposed to operate.
Change for its own sake isn't good, and I have had my share of frustrations with systemd and some degree of pain with journald growing pains too, but, from the page describing Fedora's mission, " The Fedora Project always strives to lead, not follow."
On Mon, 15.07.13 13:53, Matthew Miller (mattdm@fedoraproject.org) wrote:
If we're going to do a survey, I can think of bigger questions to ask than that. :) But, in any case, this is certainly not the way Fedora is supposed to operate.
Change for its own sake isn't good, and I have had my share of frustrations with systemd and some degree of pain with journald growing pains too, but, from the page describing Fedora's mission, " The Fedora Project always strives to lead, not follow."
(Just to mention this: we are neither the pioneer on the no-default-syslog feature nor on no-default-sendmail... A lot of other distros stopped installing an MTA by default a long time ago -- including Ubuntu since 2007 or so -- and some others have switched to only-journal mode already too -- for example ArchLinux
So, we are just following on these ones, on the sendmail thing even with a 6 year lag...)
Lennart
On Mon, Jul 15, 2013 at 12:01 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(Just to mention this: we are neither the pioneer on the no-default-syslog feature nor on no-default-sendmail... A lot of other distros stopped installing an MTA by default a long time ago -- including Ubuntu since 2007 or so -- and some others have switched to only-journal mode already too -- for example ArchLinux
So, we are just following on these ones, on the sendmail thing even with a 6 year lag...)
I can see advantages to not having an MTA by default, but I don't see the advantage in not having /var/log/messages. I don't want to have to use special tools in place of a text editor, grep, etc. As I said elsewhere, I don't care whether /var/log/messages is written by syslog or by systemd, but the default should still be to provide it. It's fine by me if the default is to generate a binary journal IN ADDITION to /var/log/messages.
Eric
On Mon, Jul 15, 2013 at 12:10:11PM -0600, Eric Smith wrote:
On Mon, Jul 15, 2013 at 12:01 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(Just to mention this: we are neither the pioneer on the no-default-syslog feature nor on no-default-sendmail... A lot of other distros stopped installing an MTA by default a long time ago -- including Ubuntu since 2007 or so -- and some others have switched to only-journal mode already too -- for example ArchLinux
So, we are just following on these ones, on the sendmail thing even with a 6 year lag...)
I can see advantages to not having an MTA by default, but I don't see the advantage in not having /var/log/messages.
One process less in the system → faster bootup. Logs not duplicated → less disk usage.
Zbyszek
One process less in the system → faster bootup.
As has been pointed out, it could easily be the systemd journal process that writes to /var/log/messages, so there wouldn't necessarily be one fewer process.
Logs not duplicated → less disk usage.
True. So keep /var/log/messages and make the binary journal optional.
Eric
On Mon, 15.07.13 14:28, Eric Smith (brouhaha@fedoraproject.org) wrote:
One process less in the system → faster bootup.
As has been pointed out, it could easily be the systemd journal process that writes to /var/log/messages, so there wouldn't necessarily be one fewer process.
Logs not duplicated → less disk usage.
True. So keep /var/log/messages and make the binary journal optional.
Nope. /var/log/messages is highly lossy, for example, we cannot even use it for showing the "systemctl status" output we now have, which includes the last 10 log lines.
That train has left the station. We created the journal with real usecases in mind, and we are not giving those up.
Lennart
On Mon, Jul 15, 2013 at 2:40 PM, Lennart Poettering mzerqung@0pointer.de wrote:
We created the journal with real usecases in mind, and we are not giving those up.
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
Eric
On Mon, Jul 15, 2013 at 02:46:27PM -0600, Eric Smith wrote:
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
I just want to look at the file without having to learn to type journalctl isn't very compelling to me, even though I do recognize the change cost.
However, the diagnostic value of an easily-accessible /var/log/messages is pretty high and I think there are real use cases there. I'm not convinced that this needs to be at the minimal level, though, and I'm not sure it should be the default for the desktop case, either.
And, in other cases, you're probably customizing your package install set anyway, right? (And, probably logging to a remote host.)
On Mon, Jul 15, 2013 at 2:53 PM, Matthew Miller mattdm@fedoraproject.org wrote:
I just want to look at the file without having to learn to type journalctl isn't very compelling to me, even though I do recognize the change cost.
However, the diagnostic value of an easily-accessible /var/log/messages is pretty high
How are those two statements not contradictory? Either an easily-accessible /var/log/messages is of high value, or it isn't.
and I think there are real use cases there. I'm not convinced that this needs to be at the minimal level, though, and I'm not sure it should be the default for the desktop case, either.
And, in other cases, you're probably customizing your package install set anyway, right? (And, probably logging to a remote host.)
On the desktop is exactly where I think it's important to keep /var/log/messages. On servers it is expected that sysadmins will do a lot of unique configuration.
Eric
On Mon, Jul 15, 2013 at 02:46:27PM -0600, Eric Smith wrote:
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
If you use bash or ksh you could just replace /var/log/messages with <(journalctl) in your command line and stuff should just work (when reading). Other shells can probably do the same. It obviously depends on journalctl being able to run.
----- Original Message -----
On Mon, Jul 15, 2013 at 02:46:27PM -0600, Eric Smith wrote:
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
If you use bash or ksh you could just replace /var/log/messages with <(journalctl) in your command line and stuff should just work (when reading). Other shells can probably do the same. It obviously depends on journalctl being able to run.
And you want to do the system wide search and replace all the 3rd party programs, scripts, and documents that mentioned /var/log/messages?
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
Image the following scenario: Suppose a Fedora newbie (or linux newbie) encounters a problem, most of the search results state: Check /var/log/messages
He/She will be very frustrated if he/she cannot find it.
I am not saying that journalctl itself is not a good tool, in fact, I've played with it, so far it is working.
However, I strongly recommend that we keep the default syslog until journalctl is well aware and adopted.
On , Ding Yi Chen wrote:
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
Image the following scenario: Suppose a Fedora newbie (or linux newbie) encounters a problem, most of the search results state: Check /var/log/messages
He/She will be very frustrated if he/she cannot find it.
This is not really an argument, or it is valid for any changes. What about the switch from up2date to yum years ago? I know, I first tried to run up2date before being shout at that it's normal it does not work anymore since yum is there. What about the future change from yum to dnf? Should we try to keep both in place just because google has more reference to yum?
So, the fact that tutorials and blogs are mentioning /var/log/messages is not a valid argument :)
Pierre
----- Original Message -----
On , Ding Yi Chen wrote:
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
Image the following scenario: Suppose a Fedora newbie (or linux newbie) encounters a problem, most of the search results state: Check /var/log/messages
He/She will be very frustrated if he/she cannot find it.
This is not really an argument, or it is valid for any changes. What about the switch from up2date to yum years ago? I know, I first tried to run up2date before being shout at that it's normal it does not work anymore since yum is there. What about the future change from yum to dnf? Should we try to keep both in place just because google has more reference to yum?
So, the fact that tutorials and blogs are mentioning /var/log/messages is not a valid argument :)
You still have not addressed the third party programs and scripts that monitor /var/log/messages
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
B) Fedora, RHEL, and most Red Hat derived distributions use /var/log/messages, but not all do -- for example, Rocks (common in HPC) breaks out syslog messages into individual files per facility. Debian and Ubuntu? Totally diferent. (/var/log/syslog)
So, these third-party scripts need to be flexible anyway. I don't think this is a very strong point in the conversation.
----- Original Message -----
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
a) From what command they know they need to install rsyslog if they want yum -y whatprovides /var/log/messages gives no result. If I did not follow the fedora-devel, I would not know why the scripts failed.
b) For user that do upgrade from backup user data/scripts->fresh install->restore data/scrips How they suppose to know that /var/log/messages is gone without checking the Release-Notes? Not everyone have time to go through all the changes.
c) why add the complexity on the script when you don't have to?
B) Fedora, RHEL, and most Red Hat derived distributions use /var/log/messages, but not all do -- for example, Rocks (common in HPC) breaks out syslog messages into individual files per facility. Debian and Ubuntu? Totally diferent. (/var/log/syslog)
So, these third-party scripts need to be flexible anyway. I don't think this is a very strong point in the conversation.
You falsely assume that: a) 3rd party developers support every operating systems under the sun, including all version of Windows, DOS and MacOS. b) 3rd party developers aware the changes c) 3rd party developers can and will diligently update their script just for Fedora.
Don't tell me that you have not seen people writing multiple platform scripts like this:
case $OS) Windows* ) some_windows_scripts ...... Linux* ) grep /var/log/messages .....
For them: What? Fedora 20 does not work while Fedora 19 does? Blame Fedora then.
And for fedora specific 3rd party scripts, now they need to add additional check logic on their script. Sometime that's just too much to ask.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, Jul 16, 2013 at 11:08 PM, Ding Yi Chen dchen@redhat.com wrote:
Don't tell me that you have not seen people writing multiple platform scripts like this:
case $OS) Windows* ) some_windows_scripts ...... Linux* ) grep /var/log/messages .....
For them: What? Fedora 20 does not work while Fedora 19 does? Blame Fedora then.
And for fedora specific 3rd party scripts, now they need to add additional check logic on their script. Sometime that's just too much to ask.
How about this idea. Before "No Defualt Syslog", systemd needs to completely replicate all functionality provided by syslog, including /var/log/messages, by default. Syslog emulation would be an option, and if people don't want it, they can still turn that option off. But by default, everyone can still grep /var/log/messages.
On Wed, 17.07.13 01:53, Billy Crook (billycrook@gmail.com) wrote:
On Tue, Jul 16, 2013 at 11:08 PM, Ding Yi Chen dchen@redhat.com wrote:
Don't tell me that you have not seen people writing multiple platform scripts like this:
case $OS) Windows* ) some_windows_scripts ...... Linux* ) grep /var/log/messages .....
For them: What? Fedora 20 does not work while Fedora 19 does? Blame Fedora then.
And for fedora specific 3rd party scripts, now they need to add additional check logic on their script. Sometime that's just too much to ask.
How about this idea. Before "No Defualt Syslog", systemd needs to completely replicate all functionality provided by syslog, including /var/log/messages, by default. Syslog emulation would be an option, and if people don't want it, they can still turn that option off. But by default, everyone can still grep /var/log/messages.
Not. Gonna. Happen.
The journal is not an implementation of syslog, we already have that in rsyslog. Also, the feature is about ending the duplicate storage of the log messages, so your suggestion is completely against what the feature is about.
Lennart
On Wed, 17.07.13 15:08, Lennart Poettering (mzerqung@0pointer.de) wrote:
How about this idea. Before "No Defualt Syslog", systemd needs to completely replicate all functionality provided by syslog, including /var/log/messages, by default. Syslog emulation would be an option, and if people don't want it, they can still turn that option off. But by default, everyone can still grep /var/log/messages.
Not. Gonna. Happen.
The journal is not an implementation of syslog, we already have that in rsyslog. Also, the feature is about ending the duplicate storage of the log messages, so your suggestion is completely against what the feature is about.
And to add to this /var/log/messages the way it stands right now is *very* badly designed:
The timestamps do not contain a year The timestamps do not contain a timezone The timestamps are accurate to the second only PID information is optional, not implied PID information is fakeable, because user supplied The file "tag" part is completely optional, free-form, and fakeable by unpriveed clients The files do not carry any information about the log priority The files do not carry any information about who is logging (service, process name, argv, binary path..) The files do not carry any information about the credentials of who is logging (uid, gid, selinux context, audit, ...)
And so on and so on.
It's so bad, that rsyslog upstream even suggests not to use these files anymore, but write them in a more modern formatting that leaves a bit more information in (such as iso timestamps). But you know what? If you do that than all your compatibility is gone too.
The interesting things is that "journalctl" is *better* at generating the same text stream that is normally contained in /var/log/messages than /var/log/messages itself is. "journalctl" can stuff more information into it then /var/log/messages. And how does that happen? Because we have more data around. We can agument the ouput with colors (indicating priorities), we can add additional informational separator lines (indicating reboots), we can add add in fields that aren't there (such as the tag from the comm field, or the PID). We can timezone correct the timestamps (because we have UTC times). And we can filter by any of the fields, securely.
So, yeah, /var/log/messages sucks, and journalctl is better at generating a compatible output that that file ever was in itself.
Lennart
Lennart Poettering mzerqung@0pointer.de writes:
[...]
How about this idea. Before "No Defualt Syslog", systemd needs to completely replicate all functionality provided by syslog, including /var/log/messages, by default. Syslog emulation would be an option, and if people don't want it, they can still turn that option off. But by default, everyone can still grep /var/log/messages.
Not. Gonna. Happen.
OK, how about this other idea. Include a default-on systemd service that runs
journalctl --full -f > /var/log/messages
in the background.
The journal is not an implementation of syslog, we already have that in rsyslog. Also, the feature is about ending the duplicate storage of the log messages [...]
Considering log rotation, and an effect of the above (> vs >>) being that only the current boot interval would be logged in the text file, there would not be much savings to worry about. Perhaps this could be an acceptable compromise.
- FChE
On Wed, Jul 17, 2013 at 10:06:27AM -0400, Frank Ch. Eigler wrote:
Lennart Poettering mzerqung@0pointer.de writes:
[...]
How about this idea. Before "No Defualt Syslog", systemd needs to completely replicate all functionality provided by syslog, including /var/log/messages, by default. Syslog emulation would be an option, and if people don't want it, they can still turn that option off. But by default, everyone can still grep /var/log/messages.
Not. Gonna. Happen.
OK, how about this other idea. Include a default-on systemd service that runs
journalctl --full -f > /var/log/messages
in the background.
Such service is called ”rsyslog” and the whole thread is about installing it by default.
On Wed, 17.07.13 10:06, Frank Ch. Eigler (fche@redhat.com) wrote:
Not. Gonna. Happen.
OK, how about this other idea. Include a default-on systemd service that runs
journalctl --full -f > /var/log/messages
in the background.
Doesn't do rotation or anything like that. Also, that already exists in rsyslog, no need to duplicate things here.
The journal is not an implementation of syslog, we already have that in rsyslog. Also, the feature is about ending the duplicate storage of the log messages [...]
Considering log rotation, and an effect of the above (> vs >>) being that only the current boot interval would be logged in the text file, there would not be much savings to worry about. Perhaps this could be an acceptable compromise.
The feature is about getting rid of the duplicate log copies, it's hardly a compromise if you leave that bit in...
Lennart
On Wed, Jul 17, 2013 at 8:08 AM, Lennart Poettering mzerqung@0pointer.dewrote:
The journal is not an implementation of syslog, we already have that in rsyslog. Also, the feature is about ending the duplicate storage of the log messages, so your suggestion is completely against what the feature is about.
What about a special filesystem mounted at /var/log or filesystem trickery therein that presents contents similar to what everyone expects, backed out of journalctl and its storage then?
Syslog is a valuable, long standing feature of *nix. It's a bit early to default to it's absence; particularly with there being such little benefit from removing it, and so much to break. Perhaps once systemd has gained wide adoption, but that is at least five years away. Of course it's wonderful that Fedora gets systemd first. But I don't think Fedora should be the first to default off widely adopted components of *nix.
Duplicate storage of log message was not a problem that syslog created. I don't think the right way to solve it is to default no syslog.
Once upon a time, Billy Crook billycrook@gmail.com said:
What about a special filesystem mounted at /var/log or filesystem trickery therein that presents contents similar to what everyone expects, backed out of journalctl and its storage then?
While you could probably do something like this with FUSE, it would be a hack, and it would also interfere with other logs under /var/log that are not syslog-related.
On Wed, Jul 17, 2013 at 4:28 PM, Chris Adams linux@cmadams.net wrote:
Once upon a time, Billy Crook billycrook@gmail.com said:
What about a special filesystem mounted at /var/log or filesystem
trickery
therein that presents contents similar to what everyone expects, backed
out
of journalctl and its storage then?
While you could probably do something like this with FUSE, it would be a hack, and it would also interfere with other logs under /var/log that are not syslog-related.
Oh I do agree. This is a difficult problem to solve, which is why I so strongly oppose creating it. Especially when duplicate log storage is such a trivial price to pay for backwards compatibility.
On Wed, Jul 17, 2013 at 04:23:54PM -0500, Billy Crook wrote:
What about a special filesystem mounted at /var/log or filesystem trickery therein that presents contents similar to what everyone expects, backed out of journalctl and its storage then?
It's probably straightforward to write a FUSE filesystem that grabs the needed information from journalctl when read. Mount it somewhere under /run and set up /var/log/messages as a symlink to the corresponding file.
But I don't see the point. Just install rsyslog. It's not going away any time soon.
Lars
On Wed, 17.07.13 00:08, Ding Yi Chen (dchen@redhat.com) wrote:
that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
a) From what command they know they need to install rsyslog if they want yum -y whatprovides /var/log/messages gives no result. If I did not follow the fedora-devel, I would not know why the scripts failed.
The release notes addition we suggested in the feature page tells you what to do.
The feature page also says we'll add /var/log/README explaining the situation.
It would be useful actually if you raised these points only after reading the thread, because this has been discussed quite a few times already on this thread.
b) For user that do upgrade from backup user data/scripts->fresh install->restore data/scrips How they suppose to know that /var/log/messages is gone without checking the Release-Notes? Not everyone have time to go through all the changes.
If you look for the files, you should hopefully notice /var/log/README and understand.
B) Fedora, RHEL, and most Red Hat derived distributions use /var/log/messages, but not all do -- for example, Rocks (common in HPC) breaks out syslog messages into individual files per facility. Debian and Ubuntu? Totally diferent. (/var/log/syslog)
So, these third-party scripts need to be flexible anyway. I don't think this is a very strong point in the conversation.
You falsely assume that: a) 3rd party developers support every operating systems under the sun, including all version of Windows, DOS and MacOS. b) 3rd party developers aware the changes c) 3rd party developers can and will diligently update their script just for Fedora.
Well, but it is very simple: if the 3rd party developer notices the file is gone, he will look for them in /var/log, and hopefully see the README. If he doesn't he will likely start googling. And is likely to find something quickly, given that notoriety of this thread.
We will continue to make changes in Fedora. We are the distribution which is supposed to bring you the new stuff first. This means changes, this means you need to keep yourself up-to-date a bit. This is just another iteration of this.
If you never want any changes, then Fedora is simply not the distribution for you. Slackware might be.
Don't tell me that you have not seen people writing multiple platform scripts like this:
case $OS) Windows* ) some_windows_scripts ...... Linux* ) grep /var/log/messages
This *already* doesn't work. On Debian-based distros you would already have to grep /var/log/daemon.log. On ArchLinux-based distros you would already have to grep journalctl.
I am sorry, but this is not where Unix was unified, ever.
For them: What? Fedora 20 does not work while Fedora 19 does? Blame Fedora then.
And for fedora specific 3rd party scripts, now they need to add additional check logic on their script. Sometime that's just too much to ask.
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
(And let's never forget that Fedora is not the pioneer here. ArchLinux went journal-only already. We are not actually the innovators here, we just follow.)
Lennart
----- Original Message -----
On Wed, 17.07.13 00:08, Ding Yi Chen (dchen@redhat.com) wrote:
that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
a) From what command they know they need to install rsyslog if they want yum -y whatprovides /var/log/messages gives no result. If I did not follow the fedora-devel, I would not know why the scripts failed.
The release notes addition we suggested in the feature page tells you what to do.
The feature page also says we'll add /var/log/README explaining the situation.
It would be useful actually if you raised these points only after reading the thread, because this has been discussed quite a few times already on this thread.
In theory, everyone should read the ChangeLog, RELEASE-NOTES, and manual on every thing he/she uses.
In reality, well ... What's the percentage of your friend and family actually finish read all the document they supposed to read?
b) For user that do upgrade from backup user data/scripts->fresh install->restore data/scrips How they suppose to know that /var/log/messages is gone without checking the Release-Notes? Not everyone have time to go through all the changes.
If you look for the files, you should hopefully notice /var/log/README and understand.
B) Fedora, RHEL, and most Red Hat derived distributions use /var/log/messages, but not all do -- for example, Rocks (common in HPC) breaks out syslog messages into individual files per facility. Debian and Ubuntu? Totally diferent. (/var/log/syslog)
So, these third-party scripts need to be flexible anyway. I don't think this is a very strong point in the conversation.
You falsely assume that: a) 3rd party developers support every operating systems under the sun, including all version of Windows, DOS and MacOS. b) 3rd party developers aware the changes c) 3rd party developers can and will diligently update their script just for Fedora.
Well, but it is very simple: if the 3rd party developer notices the file is gone, he will look for them in /var/log, and hopefully see the README. If he doesn't he will likely start googling. And is likely to find something quickly, given that notoriety of this thread.
Should I mention point c) again? Haven't you ever heard of package been orphaned or retired?
We will continue to make changes in Fedora. We are the distribution which is supposed to bring you the new stuff first. This means changes, this means you need to keep yourself up-to-date a bit. This is just another iteration of this.
If you never want any changes, then Fedora is simply not the distribution for you. Slackware might be.
I want sane changes that does not break my system. Not just change to for change's sake.
For example, change from Gnome 2 to 3 does look somewhat nicer, but it break whole lots of applet, and WM like xmonad.
That will driven away users.
Don't tell me that you have not seen people writing multiple platform scripts like this:
case $OS) Windows* ) some_windows_scripts ...... Linux* ) grep /var/log/messages
This *already* doesn't work. On Debian-based distros you would already have to grep /var/log/daemon.log. On ArchLinux-based distros you would already have to grep journalctl.
I am sorry, but this is not where Unix was unified, ever.
Please update your knowledge, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=428097
They have /var/log/messages, yes, it might be different with ours. But yes, they have that.
For them: What? Fedora 20 does not work while Fedora 19 does? Blame Fedora then.
And for fedora specific 3rd party scripts, now they need to add additional check logic on their script. Sometime that's just too much to ask.
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
Innovation should not be the cost of reliability and portability.
On Wed, 17.07.13 21:00, Ding Yi Chen (dchen@redhat.com) wrote:
The release notes addition we suggested in the feature page tells you what to do.
The feature page also says we'll add /var/log/README explaining the situation.
It would be useful actually if you raised these points only after reading the thread, because this has been discussed quite a few times already on this thread.
In theory, everyone should read the ChangeLog, RELEASE-NOTES, and manual on every thing he/she uses.
In reality, well ... What's the percentage of your friend and family actually finish read all the document they supposed to read?
We actually stick the README in /var/log, i.e. where the admin looks for the log files. We do not hide it in /usr/share/doc or so where nobody looks.
If you never want any changes, then Fedora is simply not the distribution for you. Slackware might be.
I want sane changes that does not break my system.
Well, this won't "break" systems as the change is only for new installations. Existing systems will stay exactly as they are, rsyslog stays installed, and will work as always.
I am sorry, but this is not where Unix was unified, ever.
Please update your knowledge, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=428097
They have /var/log/messages, yes, it might be different with ours. But yes, they have that.
So, they store different stuff in it. The interesting stuff is mostly in daemon.log on Debian. So with your suggested program you'd miss out all the interesting bit son Debian. This stuff is certainly not standardized on Unix systems...
Innovation should not be the cost of reliability and portability.
This change touches neither. /var/log/messages already isn't standard in whether it exists at all, and what it contains, so we certainly don't make "portability" worse...
Lennart
----- Original Message -----
On Wed, 17.07.13 21:00, Ding Yi Chen (dchen@redhat.com) wrote:
If you never want any changes, then Fedora is simply not the distribution for you. Slackware might be.
I want sane changes that does not break my system.
Well, this won't "break" systems as the change is only for new installations. Existing systems will stay exactly as they are, rsyslog stays installed, and will work as always.
1. What if they update the system like this: Backed up user data/script -> Fresh install -> Restore user data/script For that, it won't work.
2. Like other already point out, Windows/Fedora dual boot. You can see /var/log/messages from Windows, but how can you get journalctl output in Windows?
Please update your knowledge, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=428097
They have /var/log/messages, yes, it might be different with ours. But yes, they have that.
So, they store different stuff in it. The interesting stuff is mostly in daemon.log on Debian. So with your suggested program you'd miss out all the interesting bit son Debian. This stuff is certainly not standardized on Unix systems...
a) If debian output the thing I want in /var/log/messages anyway, why should I care whether other daemon output in other files? b) If my environment only contains RHEL and Fedora, why should I care how Debian, Arch and Ubuntu handle their logs?
Innovation should not be the cost of reliability and portability.
This change touches neither. /var/log/messages already isn't standard in whether it exists at all, and what it contains, so we certainly don't make "portability" worse...
Something is not standard does not mean nobody using it. Especially it is there quite a long time. Remove it simply break their expectation and scripts. For that, you do make the portability worse.
Oh come on you are really reaching now. The below two points are especially ridiculous.
- What if they update the system like this: Backed up user data/script -> Fresh install -> Restore user data/script For that, it won't work.
This is called a fresh install and not an upgrade. In this scenario there would be a substantial amount of work anyway... A sane backup in this workflow should include an rpm list to restore anyway (which would then include rsyslog). Would you just do the install and then have over the machine? No you'd do various customisation bits and as an informed admin you know of the change and can add rsyslog if you want.
If an inattentive admin in this scenario you'd do your install and check for problems in /var/log/messages and see the README ... If such a bad admin even that is missed then the person is in the wrong job.
- Like other already point out, Windows/Fedora dual boot. You can see /var/log/messages from Windows, but how can you get
journalctl output in Windows?
Oh how do you get your logs to read in windows from your lvm/ext4/btrfs filesystems currently in a disk boot scenario?
On Jul 18, 2013 12:22 AM, "James Hogarth" james.hogarth@gmail.com wrote:
Oh how do you get your logs to read in windows from your lvm/ext4/btrfs
filesystems currently in a disk boot scenario?
Using ext2fsd: http://www.ext2fsd.com
Eric
Oh how do you get your logs to read in windows from your lvm/ext4/btrfs
filesystems currently in a disk boot scenario?
Using ext2fsd: http://www.ext2fsd.com
... I'd suggest you read that page and then look at my question and think real hard...
On Thu, Jul 18, 2013 at 1:56 AM, James Hogarth james.hogarth@gmail.com wrote:
Oh how do you get your logs to read in windows from your lvm/ext4/btrfs filesystems currently in a disk boot scenario?
Using ext2fsd: http://www.ext2fsd.com
... I'd suggest you read that page and then look at my question and think real hard...
Maybe your question is poorly stated, then.
What I thought you asked was how to read Linux log files from a Windows installation, e.g., when Linux fails to boot. In the past I've been able to do that using ext2fsd without much difficulty. I used that method when I wasn't able to boot a rescue or live CD, and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it. But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
My main complaint is that removing the default syslog to /var/log/messages makes it harder for me to diagnose broken machines that OTHER people have set up, because those other people aren't going to have installed a non-default syslog daemon. Certainly if it's a machine I'm installing, I'll know to install syslog.
Eric
On Thu, Jul 18, 2013 at 09:51:32AM -0600, Eric Smith wrote:
What I thought you asked was how to read Linux log files from a Windows installation, e.g., when Linux fails to boot. In the past I've been able to do that using ext2fsd without much difficulty. I used that method when I wasn't able to boot a rescue or live CD, and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it. But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
I think this case is relatively obscure, but as with RHEL6, it would be nice to have a journal file viewer for Windows.
On 07/18/2013 03:55 PM, Matthew Miller wrote:
On Thu, Jul 18, 2013 at 09:51:32AM -0600, Eric Smith wrote:
What I thought you asked was how to read Linux log files from a Windows installation, e.g., when Linux fails to boot. In the past I've been able to do that using ext2fsd without much difficulty. I used that method when I wasn't able to boot a rescue or live CD, and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it. But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
I think this case is relatively obscure, but as with RHEL6, it would be nice to have a journal file viewer for Windows.
Why not read this files on another Fedora host ( or some other distro that uses systemd )? What's the reason for this hard dependency on Windows?
JBG
On Thu, Jul 18, 2013 at 03:54:49PM +0000, "Jóhann B. Guðmundsson" wrote:
I've been able to do that using ext2fsd without much difficulty. I used that method when I wasn't able to boot a rescue or live CD, and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it. But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
I think this case is relatively obscure, but as with RHEL6, it would be nice to have a journal file viewer for Windows.
Why not read this files on another Fedora host ( or some other distro that uses systemd )? What's the reason for this hard dependency on Windows?
I think the use case here is dual-boot system where the Fedora installation is somehow broken but the Windows boot works.
On 07/18/2013 04:23 PM, Matthew Miller wrote:
On Thu, Jul 18, 2013 at 03:54:49PM +0000, "Jóhann B. Guðmundsson" wrote:
I've been able to do that using ext2fsd without much difficulty. I used that method when I wasn't able to boot a rescue or live CD, and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it. But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
I think this case is relatively obscure, but as with RHEL6, it would be nice to have a journal file viewer for Windows.
Why not read this files on another Fedora host ( or some other distro that uses systemd )? What's the reason for this hard dependency on Windows?
I think the use case here is dual-boot system where the Fedora installation is somehow broken but the Windows boot works.
If you have physical access to the machine you should be able to access those journal files from within dracut shell. If not that something we need to look at and solve I would think.
JBG
On Thu, Jul 18, 2013 at 9:54 AM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Why not read this files on another Fedora host ( or some other distro that uses systemd )? What's the reason for this hard dependency on Windows?
Because I was about six hundred miles away from my office, didn't want to take the user's computer apart if I could avoid it, and didn't have a drive dock to hook up the user's drive to my laptop. The user had Windows available on the machine, so I took advantage of it to figure out what was wrong with Linux and fix it.
Eric
On 18 Jul 2013 18:42, "Eric Smith" brouhaha@fedoraproject.org wrote:
Because I was about six hundred miles away from my office, didn't want to take the user's computer apart if I could avoid it, and didn't have a drive dock to hook up the user's drive to my laptop. The user had Windows available on the machine, so I took advantage of it to figure out what was wrong with Linux and fix it.
Without sounding too blunt I hope this does sound like we're entering the territory of "lack of planning on your part does not constitute and emergency on mine" as I have to occasionally remind people at work...
This is such an extreme use case and as pointed out by Lennart as well is not viable on current systems anyway without huge hoop jumping...
This hack of a workaround you attempted once can no way realistically be considered a blocker to this as it's so far off a support matrix it's almost comical to suggest it as such...
You could have used his windows partition to download a live CD and use that as a less fragile solution that would be less likely to cause filesystem corruption and work with a default fedora on lvm and so on... Which as pointed out your workaround would not.
Le Jeu 18 juillet 2013 19:56, James Hogarth a écrit :
Without sounding too blunt I hope this does sound like we're entering the territory of "lack of planning on your part does not constitute and emergency on mine" as I have to occasionally remind people at work...
This is such an extreme use case and as pointed out by Lennart as well is not viable on current systems anyway without huge hoop jumping...
Without sounding too blunt, this is business as usual from a repair end-user system point of view. I had dozens of such "oh btw can you fix my system" experiences
On 18 July 2013 19:26, Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Without sounding too blunt, this is business as usual from a repair end-user system point of view. I had dozens of such "oh btw can you fix my system" experiences
Yeah I've been there in the past ... which is why I have spare USB pen drives in my rucksack - some empty some with a live instance already on there...
And if going to diagnose/repair a Linux system it'd be sane to grab a centos and fedora live instance before heading out ...
It's just part of the toolset - just as screwdrivers and so on are as well for many engineers ...
I've also been there with dodgy hacky workarounds to deal with strange stuff - but I wouldn't expect it to weight in an argument for something like this ...
Le Jeu 18 juillet 2013 20:34, James Hogarth a écrit :
I've also been there with dodgy hacky workarounds to deal with strange stuff - but I wouldn't expect it to weight in an argument for something like this ...
Why not? In the imperfect world we live in, I'm quite sure they comprise a large part of the home linux market. Linux is solid server-side but the desktop-side is far from there (and has been busy dismantling the bits that made server linux reliable)
On 18 July 2013 16:51, Eric Smith brouhaha@fedoraproject.org wrote:
Maybe your question is poorly stated, then.
What I thought you asked was how to read Linux log files from a Windows installation, e.g., when Linux fails to boot.
This is indeed the question - so given you understood it so it seems I would say that it was not poorly stated.
In the past I've been able to do that using ext2fsd without much difficulty.
This will not work depending on ext4 options, if LVM is in use or if BTRFS is used which is of course now supported as an option in the installer.
I used that method when I wasn't able to boot a rescue or live CD,
Then you were not using it with a default installed Fedora anyway which has a default of LVM in place
and the last resort would have been to pull the hard drive from the machine and use a different computer to inspect it.
That or live media is the best option in general... I know above you said you couldn't use a live CD and I'm quite curious as to why.
But if /var/log/messages is not made available by default, then using ext2fsd won't work, and other methods become more difficult also.
It already won't work for a default installed Fedora ... there is no difference.
My main complaint is that removing the default syslog to /var/log/messages makes it harder for me to diagnose broken machines that OTHER people have set up, because those other people aren't going to have installed a non-default syslog daemon. Certainly if it's a machine I'm installing, I'll know to install syslog.
Well fortunately you pay attention to these lists so you know to look at the README and if /var/log/messages is not there (or if Fedora in general now) you should use journalctl instead...
On Thu, 18.07.13 17:11, James Hogarth (james.hogarth@gmail.com) wrote:
On 18 July 2013 16:51, Eric Smith brouhaha@fedoraproject.org wrote:
Maybe your question is poorly stated, then.
What I thought you asked was how to read Linux log files from a Windows installation, e.g., when Linux fails to boot.
This is indeed the question - so given you understood it so it seems I would say that it was not poorly stated.
In the past I've been able to do that using ext2fsd without much difficulty.
This will not work depending on ext4 options, if LVM is in use or if BTRFS is used which is of course now supported as an option in the installer.
Actually, it's worse. The driver requires you to turn of driver signature verification of Windows. That's just a huge mess. (Also, it doesn't support the current Windows version).
I don't think that using ext2fsd is possible "without much difficulty". It's great that such a tool exists, but it's a hacker tool, for somebody who is willing to alter his Windows installations in non-trivial ways.
I am pretty sure that just downloading an ISO of the latest Fedora livecd and dd'ing it to an USB disk is a ton more fun that the ext2fsd dance, and is a lot more comprehensive with its LVM, LUKS, btrfs support that pretty much just works.
Lennart
On Thu, Jul 18, 2013 at 10:11 AM, James Hogarth james.hogarth@gmail.com wrote:
Then you were not using it with a default installed Fedora anyway which has a default of LVM in place
I don't remember why there wasn't LVM. I don't remember whether I was the one that installed Linux on that machine in the first place. I might have been.
That or live media is the best option in general... I know above you said you couldn't use a live CD and I'm quite curious as to why.
The machine didn't have a working optical drive. If I'd had a live image on USB that probably would have worked. If I hadn't been hundreds of miles from the nearest computer store, I'd have just bought another optical drive, or a drive dock, or a cable to use the optical drive out of my laptop, or any number of things that would have made it easier to solve the problem. I was motivated to try to solve it using only what happened to be at hand.
Eric
On Wed, 17.07.13 22:08, Ding Yi Chen (dchen@redhat.com) wrote:
Well, this won't "break" systems as the change is only for new installations. Existing systems will stay exactly as they are, rsyslog stays installed, and will work as always.
- What if they update the system like this: Backed up user data/script -> Fresh install -> Restore user data/script For that, it won't work.
In such a case, you already need to manually reinstall all packages you need beyond the default set after the reinstallation. The fewest people probably stick to exactly the set of packages we install by default for their systems. rsyslog is now one more of those packages you need to reinstall after your system is back up.
- Like other already point out, Windows/Fedora dual boot. You can see /var/log/messages from Windows, but how can you get journalctl output in Windows?
Well, as pointed out before, "journalctl" on Windows helps little if you cannot access the Linux partitions in the first place, because they are ext4 or btrfs.
Please update your knowledge, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=428097
They have /var/log/messages, yes, it might be different with ours. But yes, they have that.
So, they store different stuff in it. The interesting stuff is mostly in daemon.log on Debian. So with your suggested program you'd miss out all the interesting bit son Debian. This stuff is certainly not standardized on Unix systems...
a) If debian output the thing I want in /var/log/messages anyway, why should I care whether other daemon output in other files?
Well, most likely it won't include the interesting bits, because they are in daemon.log.
I mean, you claim that all distros have /var/log/messages and that that's where the interesting stuff goes. And that is simply not true. No ifs, it's just simply not true.
b) If my environment only contains RHEL and Fedora, why should I care how Debian, Arch and Ubuntu handle their logs?
Well, "journalctl" has been available for some time already on Fedora, and will be in RHEL7 too, so you shouldn't be too concerned there.
Innovation should not be the cost of reliability and portability.
This change touches neither. /var/log/messages already isn't standard in whether it exists at all, and what it contains, so we certainly don't make "portability" worse...
Something is not standard does not mean nobody using it.
No it doesn't. Every package in the Fedora archive is used by somebody, but that doesn't mean we install *all* packages always. We try to install a default set that tries neither to be minimal, nor to include everything possible. Something that one can work with and that has little redundancy.
Especially it is there quite a long time. Remove it simply break their expectation and scripts. For that, you do make the portability worse.
No, not true by any definition of the word "portability".
Lennart
----- Original Message -----
On Wed, 17.07.13 22:08, Ding Yi Chen (dchen@redhat.com) wrote:
Well, this won't "break" systems as the change is only for new installations. Existing systems will stay exactly as they are, rsyslog stays installed, and will work as always.
- What if they update the system like this: Backed up user data/script -> Fresh install -> Restore user data/script For that, it won't work.
In such a case, you already need to manually reinstall all packages you need beyond the default set after the reinstallation. The fewest people probably stick to exactly the set of packages we install by default for their systems. rsyslog is now one more of those packages you need to reinstall after your system is back up.
In that setting, users usually start with default choose the packages they EXPLICITLY want to install/remove, they are very likely to assume that the rest of the system environment, including /var/log/messages will still be there.
Besides, rsyslog is in core, which is hidden from users and most of them are unaware what the rsyslog actually do and generated.
- Like other already point out, Windows/Fedora dual boot. You can see /var/log/messages from Windows, but how can you get journalctl output in Windows?
Well, as pointed out before, "journalctl" on Windows helps little if you cannot access the Linux partitions in the first place, because they are ext4 or btrfs.
Do some web search, and you will find out there are handful utility let you read ext4 partitions.
I've used http://www.fs-driver.org/ and it can read ext3 partitions, BTW.
Please update your knowledge, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=428097
They have /var/log/messages, yes, it might be different with ours. But yes, they have that.
So, they store different stuff in it. The interesting stuff is mostly in daemon.log on Debian. So with your suggested program you'd miss out all the interesting bit son Debian. This stuff is certainly not standardized on Unix systems...
a) If debian output the thing I want in /var/log/messages anyway, why should I care whether other daemon output in other files?
Well, most likely it won't include the interesting bits, because they are in daemon.log.
Didn't I say I don't care other daemons output to any other files (including daemon.log)?
I mean, you claim that all distros have /var/log/messages and that that's where the interesting stuff goes. And that is simply not true. No ifs, it's just simply not true.
Did I claim that all distros and OSes have /var/log/messages? DOS and Windows don't have it. Happy now?
Let's talk about the system that do have and use /var/log/messages, like Fedora 19 and RHEL6. How do you deal with the programs that write for either or both that use /var/log/messages.
Do a grep -cslR '/var/log/messages' /usr
you will have a brief idea what's the problem size.
b) If my environment only contains RHEL and Fedora, why should I care how Debian, Arch and Ubuntu handle their logs?
Well, "journalctl" has been available for some time already on Fedora, and will be in RHEL7 too, so you shouldn't be too concerned there.
Please note that RHEL6 and RHEL5 are still in their life cycle. And they are unlikely to have journalctl.
Innovation should not be the cost of reliability and portability.
This change touches neither. /var/log/messages already isn't standard in whether it exists at all, and what it contains, so we certainly don't make "portability" worse...
Something is not standard does not mean nobody using it.
No it doesn't. Every package in the Fedora archive is used by somebody, but that doesn't mean we install *all* packages always. We try to install a default set that tries neither to be minimal, nor to include everything possible. Something that one can work with and that has little redundancy.
The problem is, to you, /var/log/messages is redundant, but for others, it is not. By the react of the mailling list and the results from grep the system, it is still used by those.
Are you sure you are not going to break those? Have you tested those?
Especially it is there quite a long time. Remove it simply break their expectation and scripts. For that, you do make the portability worse.
No, not true by any definition of the word "portability".
Yes, right, you simply let those programs and documents lost their portability. :-P
----- Original Message -----
On Wed, 17.07.13 21:00, Ding Yi Chen (dchen@redhat.com) wrote:
The release notes addition we suggested in the feature page tells you what to do.
The feature page also says we'll add /var/log/README explaining the situation.
It would be useful actually if you raised these points only after reading the thread, because this has been discussed quite a few times already on this thread.
In theory, everyone should read the ChangeLog, RELEASE-NOTES, and manual on every thing he/she uses.
In reality, well ... What's the percentage of your friend and family actually finish read all the document they supposed to read?
We actually stick the README in /var/log, i.e. where the admin looks for the log files. We do not hide it in /usr/share/doc or so where nobody looks.
That would definitely help for people that actively go into /var/log and look for /var/log/messages.
It does not help on the case of some monitor program that read /var/log/messages then generate the reports.
User might be aware the monitor program stop working, but not aware the root cause is /var/log/messages no longer exists.
On Wed, 2013-07-17 at 14:58 +0200, Lennart Poettering wrote:
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
(And let's never forget that Fedora is not the pioneer here. ArchLinux went journal-only already. We are not actually the innovators here, we just follow.)
I believe openSUSE 12.3 does not install syslog anymore either. (I think they decided they did not want to log everything twice? :) Fedora's following this time.
They had a couple of mailing list fights over the issue, which I thought were pretty impressive at the time, but you guys put them to shame!
On 07/18/2013 06:36 AM, Michael Catanzaro wrote:
On Wed, 2013-07-17 at 14:58 +0200, Lennart Poettering wrote:
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
(And let's never forget that Fedora is not the pioneer here. ArchLinux went journal-only already. We are not actually the innovators here, we just follow.)
I believe openSUSE 12.3 does not install syslog anymore either. (I think they decided they did not want to log everything twice? :) Fedora's following this time.
Is this of any importance? May be you should think about the reasons we are using Fedora and are not using openSUSE.
That said Fedora should draw its own decisions and not try to be an imitation cult.
On 18 July 2013 10:17, Ralf Corsepius rc040203@freenet.de wrote:
Is this of any importance? May be you should think about the reasons we are using Fedora and are not using openSUSE.
That said Fedora should draw its own decisions and not try to be an imitation cult.
Well given that one of the cases complaining against the change stated how they liked how they could grab a linux system and grep /var/log/messages and it's a defacto standard that shouldn't change and so on .... yes I'd say that would be of importance as a counter point.
On Wed, 17.07.13 23:36, Michael Catanzaro (mike.catanzaro@gmail.com) wrote:
On Wed, 2013-07-17 at 14:58 +0200, Lennart Poettering wrote:
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
(And let's never forget that Fedora is not the pioneer here. ArchLinux went journal-only already. We are not actually the innovators here, we just follow.)
I believe openSUSE 12.3 does not install syslog anymore either. (I think they decided they did not want to log everything twice? :) Fedora's following this time.
They do still install syslog. ArchLinux doesn't anymore however.
Lennart
On Thu, 2013-07-18 at 14:17 +0200, Lennart Poettering wrote:
I believe openSUSE 12.3 does not install syslog anymore either. (I
think
they decided they did not want to log everything twice? :) Fedora's following this time.
Hm OK. They definitely dropped it from the default install late last year (caused a bit of a stir) and from a quick check online it looked like they had not reversed that choice, but I didn't check an actual system.
----- Original Message -----
On Wed, 2013-07-17 at 14:58 +0200, Lennart Poettering wrote:
We ask this constantly on Fedora. Because Fedora is where innovation is supposed to take place, not where things are stay frozen in carbonite forever.
(And let's never forget that Fedora is not the pioneer here. ArchLinux went journal-only already. We are not actually the innovators here, we just follow.)
I believe openSUSE 12.3 does not install syslog anymore either. (I think they decided they did not want to log everything twice? :) Fedora's following this time.
http://doc.opensuse.org/documentation/html/openSUSE/opensuse-tuning/cha.tuni...
Looks like they either still have /var/log/messages, or their documentation team are lazy.
Hi
On Thu, Jul 18, 2013 at 9:43 PM, Ding Yi Chen wrote:
http://doc.opensuse.org/documentation/html/openSUSE/opensuse-tuning/cha.tuni...
Looks like they either still have /var/log/messages, or their documentation team are lazy.
Be careful about such assumptions. There are bugs in the Fedora documentation as well and you wouldn't want anyone implying that the Fedora docs team is lazy. If you want to actually confirm, download the ISO and check.
Rahul
----- Original Message -----
Hi
On Thu, Jul 18, 2013 at 9:43 PM, Ding Yi Chen wrote:
http://doc.opensuse.org/documentation/html/openSUSE/opensuse-tuning/cha.tuni...
Looks like they either still have /var/log/messages,
or their documentation team are lazy.
Be careful about such assumptions. There are bugs in the Fedora documentation as well and you wouldn't want anyone implying that the Fedora docs team is lazy. If you want to actually confirm, download the ISO and check.
Rahul
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Thanks for point this out.
I am sorry if any one is offended.
(Download ISO and check? No, I am lazy.)
On Tue, Jul 16, 2013 at 6:13 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
So? The same argument could be made for programs that expect the binary journal.
Eric
On Tue, Jul 16, 2013 at 09:26:22PM -0700, Eric Smith wrote:
On Tue, Jul 16, 2013 at 6:13 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
So? The same argument could be made for programs that expect the binary journal.
No it isn't. It was already said: journald is here always, rsyslog is optional. The question is: should we install optional rsyslog by default?
Matthew Miller (mattdm@fedoraproject.org) said:
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
But, at the packaging level, I think it's been demonstrated that 'a program that expects this file' is not something we do a good job of consistently expressing. That should be fixed regardless of this feature proposal, but I think should definitely be a required component of accepting it.
Bill
On Wed, Jul 17, 2013 at 11:41:24AM -0400, Bill Nottingham wrote:
A) If someone is installing a program that expects this file, they can also install rsyslog.
But, at the packaging level, I think it's been demonstrated that 'a program that expects this file' is not something we do a good job of consistently expressing. That should be fixed regardless of this feature proposal, but I think should definitely be a required component of accepting it.
I'm reading this to mean by adding package dependencies where missing, right?
This is actually Kinda Hard, because /var/log/messages is really more an artifact of our default rsyslog config than part of any package. If we wanted to say that some syslog service always provides that file, we should make it clear that changing it is actually not available for end-user configuration and should never been changed. We've never done that.
So, adding "Requires: rsyslog" isn't _really_ sufficient. We could ship the /var/log/messages configuration in a subpackage that goes into /etc/rsyslog.d/ and ask packages which expect /var/log/messages to require that.
Matthew Miller (mattdm@fedoraproject.org) said:
On Wed, Jul 17, 2013 at 11:41:24AM -0400, Bill Nottingham wrote:
A) If someone is installing a program that expects this file, they can also install rsyslog.
But, at the packaging level, I think it's been demonstrated that 'a program that expects this file' is not something we do a good job of consistently expressing. That should be fixed regardless of this feature proposal, but I think should definitely be a required component of accepting it.
I'm reading this to mean by adding package dependencies where missing, right?
This is actually Kinda Hard, because /var/log/messages is really more an artifact of our default rsyslog config than part of any package. If we wanted to say that some syslog service always provides that file, we should make it clear that changing it is actually not available for end-user configuration and should never been changed. We've never done that.
So, adding "Requires: rsyslog" isn't _really_ sufficient. We could ship the /var/log/messages configuration in a subpackage that goes into /etc/rsyslog.d/ and ask packages which expect /var/log/messages to require that.
I think it's more correct for packages that expect to work on a textual /var/log/messages (or similar) to have a requirement on a meta 'syslog' package (which all three major daemons provide), even if it's implicitly depending on the default configuration of those packages, rather than the alternative of *no* dependencies that we have now.
Bill
On Wed, Jul 17, 2013 at 12:10:35PM -0400, Bill Nottingham wrote:
I think it's more correct for packages that expect to work on a textual /var/log/messages (or similar) to have a requirement on a meta 'syslog' package (which all three major daemons provide), even if it's implicitly depending on the default configuration of those packages, rather than the alternative of *no* dependencies that we have now.
Sounds good, but:
$ repoquery --whatprovides syslog syslog-ng-0:3.4.1-1.fc19.x86_64 sysklogd-0:1.5-13.fc19.x86_64 rsyslog-0:7.2.6-1.fc19.x86_64 systemd-0:204-9.fc19.x86_64
$ rpm -q --changelog systemd|grep -B1 'Provide syslog' * Tue Oct 23 2012 Lennart Poettering lpoetter@redhat.com - 195-2 - Provide syslog because the journal is fine as a syslog implementation
Lennart, maybe this should be re-thought as to what "syslog" means exactly in this context? It's true that from an application-logging-stuff point of view systemd-jourald provides this, but maybe that's not right.
On the other hand, it looks like _only_ perl-Unix-Syslog has this as an explicit requirement, and that wants it for the logging capability, not for mucking with log output.
On Wed, 17.07.13 12:24, Matthew Miller (mattdm@fedoraproject.org) wrote:
On Wed, Jul 17, 2013 at 12:10:35PM -0400, Bill Nottingham wrote:
I think it's more correct for packages that expect to work on a textual /var/log/messages (or similar) to have a requirement on a meta 'syslog' package (which all three major daemons provide), even if it's implicitly depending on the default configuration of those packages, rather than the alternative of *no* dependencies that we have now.
Sounds good, but:
$ repoquery --whatprovides syslog syslog-ng-0:3.4.1-1.fc19.x86_64 sysklogd-0:1.5-13.fc19.x86_64 rsyslog-0:7.2.6-1.fc19.x86_64 systemd-0:204-9.fc19.x86_64
$ rpm -q --changelog systemd|grep -B1 'Provide syslog'
- Tue Oct 23 2012 Lennart Poettering lpoetter@redhat.com - 195-2
- Provide syslog because the journal is fine as a syslog implementation
Lennart, maybe this should be re-thought as to what "syslog" means exactly in this context? It's true that from an application-logging-stuff point of view systemd-jourald provides this, but maybe that's not right.
There were some packages which required "syslog" because they were clients to it, which is a pretty bogus thing to do.
There's one of those packages left over now. Which is per-Unix-Syslog. If we get that one fixed to not require syslog, then we could drop Provides: syslog from journald.
However, I wonder if this would be a good idea. journald does provide the /dev/log socket. I think most people understood the "syslog" dependency so far as a dependency from log client to log server, and not as a dependency on the existance of /var/log/messages.
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
I was thinking of it from the point of view of someone wanting traditional syslog behavior could "yum install syslog", which would get something that writes /var/log/<foo> (which won't happen with the current setup).
I would consider something that handles the syslog() library call (and so /dev/log) to be enough of the traditional Unix core that it shouldn't need a requires.
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
Vít
On Thu, 2013-07-18 at 10:34 +0200, Vít Ondruch wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
Why a virtual provides, can't they just %ghost their log files?
Dne 18.7.2013 10:42, Mathieu Bridon napsal(a):
On Thu, 2013-07-18 at 10:34 +0200, Vít Ondruch wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
Why a virtual provides, can't they just %ghost their log files?
That would be the best of course, unfortunately: https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
But may be YUM/DNF can be improved in this area
Vít
On Thu, 2013-07-18 at 10:56 +0200, Vít Ondruch wrote:
Dne 18.7.2013 10:42, Mathieu Bridon napsal(a):
On Thu, 2013-07-18 at 10:34 +0200, Vít Ondruch wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
Why a virtual provides, can't they just %ghost their log files?
That would be the best of course, unfortunately: https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
But may be YUM/DNF can be improved in this area
Right, but that's only if you need to express a file requirement.
Your point above was that every package which drops a file in /var/log (e.g httpd) should have virtual-provides for them.
But I don't think there's ever any reason to require the file /var/log/https/error_log
On the other hand, Lennart's original mail above was about syslog implementations, not "all package, which creates log in /var/log".
Dne 18.7.2013 11:13, Mathieu Bridon napsal(a):
On Thu, 2013-07-18 at 10:56 +0200, Vít Ondruch wrote:
Dne 18.7.2013 10:42, Mathieu Bridon napsal(a):
On Thu, 2013-07-18 at 10:34 +0200, Vít Ondruch wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
Why a virtual provides, can't they just %ghost their log files?
That would be the best of course, unfortunately: https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
But may be YUM/DNF can be improved in this area
Right, but that's only if you need to express a file requirement.
Your point above was that every package which drops a file in /var/log (e.g httpd) should have virtual-provides for them.
But I don't think there's ever any reason to require the file /var/log/https/error_log
On the other hand, Lennart's original mail above was about syslog implementations, not "all package, which creates log in /var/log".
It was especially about /var/log/messages which should be by Lennart's proposal replaced by "syslog-files" virtual provide. How am I supposed to know what does this virtual provide means? So I propose to come with some generic, understandable way how to let other packages depend on some file in /var/log if they need (I left out the question "how common case it is" on purpose) and avoid the need of inefficient file dependency for this purpose.
Vít
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
Dne 18.7.2013 14:19, Lennart Poettering napsal(a):
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
I would suggest it, but it is not recommended by guidelines :( so I suggest some (not yet) standardized virtual provide, which will be more descriptive than "syslog-files"
Vít
[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
On Thu, 18.07.13 14:23, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 14:19, Lennart Poettering napsal(a):
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
I would suggest it, but it is not recommended by guidelines :( so I suggest some (not yet) standardized virtual provide, which will be more descriptive than "syslog-files"
Vít
[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
I guess this comment doesn't apply if we explicitly add Provides: /var/log/messages to all packages that provide the file. Hmm, or maybe no, I don't grok RPM well enough...
Lennart
On 07/18/2013 03:28 PM, Lennart Poettering wrote:
On Thu, 18.07.13 14:23, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 14:19, Lennart Poettering napsal(a):
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
I would suggest it, but it is not recommended by guidelines :( so I suggest some (not yet) standardized virtual provide, which will be more descriptive than "syslog-files"
Vít
[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
I guess this comment doesn't apply if we explicitly add Provides: /var/log/messages to all packages that provide the file. Hmm, or maybe no, I don't grok RPM well enough...
Well the guideline is really just a recommendation for optimizing yum behavior, nothing more. But yes, an explicit "Provides: /some/path" goes into the main repository metadata so resolving a dependency on that path doesn't require downloading the big bad file lists.
- Panu -
On Thu, 18.07.13 15:47, Panu Matilainen (pmatilai@laiskiainen.org) wrote:
I would suggest it, but it is not recommended by guidelines :( so I suggest some (not yet) standardized virtual provide, which will be more descriptive than "syslog-files"
Vít
[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
I guess this comment doesn't apply if we explicitly add Provides: /var/log/messages to all packages that provide the file. Hmm, or maybe no, I don't grok RPM well enough...
Well the guideline is really just a recommendation for optimizing yum behavior, nothing more. But yes, an explicit "Provides: /some/path" goes into the main repository metadata so resolving a dependency on that path doesn't require downloading the big bad file lists.
Hmm, Panu, but who does this exactly work? If at least one package explicitly provides /some/path, and some others only implicitly provide it, is the big bad file list download skipped?
Which would mean either *none* of the providers shall explicitly provide the file (which would be slow), or *all* of the provides explicitly provide the file? If some would explicitly provide it, and others only implicitly, then things would be broken?
Lennart
On 07/18/2013 04:42 PM, Lennart Poettering wrote:
On Thu, 18.07.13 15:47, Panu Matilainen (pmatilai@laiskiainen.org) wrote:
I would suggest it, but it is not recommended by guidelines :( so I suggest some (not yet) standardized virtual provide, which will be more descriptive than "syslog-files"
Vít
[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
I guess this comment doesn't apply if we explicitly add Provides: /var/log/messages to all packages that provide the file. Hmm, or maybe no, I don't grok RPM well enough...
Well the guideline is really just a recommendation for optimizing yum behavior, nothing more. But yes, an explicit "Provides: /some/path" goes into the main repository metadata so resolving a dependency on that path doesn't require downloading the big bad file lists.
Hmm, Panu, but who does this exactly work? If at least one package explicitly provides /some/path, and some others only implicitly provide it, is the big bad file list download skipped?
Which would mean either *none* of the providers shall explicitly provide the file (which would be slow), or *all* of the provides explicitly provide the file? If some would explicitly provide it, and others only implicitly, then things would be broken?
Hmm, good question. I've no idea what yum does in that situation. Most other depsolvers (have to) always download the full filelists anyway, making the point moot, but since yum tries to avoid it... My *guess* is that yum would go downloading the full filelists anyway when the path is outside the "common paths" stored in the primary metadata directly.
- Panu -
On Thu, Jul 18, 2013 at 02:19:30PM +0200, Lennart Poettering wrote:
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question? Sounds good to me! Matthew?
My main concern with this is that it's a lie. That file only exists because of the default configuration. In many cases, rsyslog will be configured to either write different files, or most likely, to write no local files at all as all data is forwarded. And, as discussed in another subthread, I expect this last configuration to be more and more common. So, not just a lie, but a lie which may actually make it harder to use rsyslog in ways other than the default.
In an ideal view, it makes most sense to provide the rsyslog default configuration in a subpackage which puts the /var/log/messages and /var/log/secure conf files in /etc/rsyslog.d -- then, this subpackage would provide those files. Unfortunately, in order to preserve behavior on upgrade, the main package would have to depend on this, kind of making the distinction moot.
On Thu, 18.07.13 11:11, Matthew Miller (mattdm@fedoraproject.org) wrote:
On Thu, Jul 18, 2013 at 02:19:30PM +0200, Lennart Poettering wrote:
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question? Sounds good to me! Matthew?
My main concern with this is that it's a lie. That file only exists because of the default configuration. In many cases, rsyslog will be configured to either write different files, or most likely, to write no local files at all as all data is forwarded. And, as discussed in another subthread, I expect this last configuration to be more and more common. So, not just a lie, but a lie which may actually make it harder to use rsyslog in ways other than the default.
In an ideal view, it makes most sense to provide the rsyslog default configuration in a subpackage which puts the /var/log/messages and /var/log/secure conf files in /etc/rsyslog.d -- then, this subpackage would provide those files. Unfortunately, in order to preserve behavior on upgrade, the main package would have to depend on this, kind of making the distinction moot.
Well, I am not sure it's really a "lie". It's reasonably close to the truth, and it would be agood thing anyway if the syslog implementation would own /var/log/messages at least as %ghost. I mean, people can always reconfigure things, and muck around with how things are set up and break stuff. That's OK, people should be able to do that.
I think this should really be considered a documentation problem: in the default rsyslog.conf ship a few comments explaining that /var/log/messages is kinda considered API by some other packages, and that the user should only alter its configuration if he knows what he does.
Lennart
Matthew Miller (mattdm@fedoraproject.org) said:
My main concern with this is that it's a lie. That file only exists because of the default configuration. In many cases, rsyslog will be configured to either write different files, or most likely, to write no local files at all as all data is forwarded. And, as discussed in another subthread, I expect this last configuration to be more and more common. So, not just a lie, but a lie which may actually make it harder to use rsyslog in ways other than the default.
In an ideal view, it makes most sense to provide the rsyslog default configuration in a subpackage which puts the /var/log/messages and /var/log/secure conf files in /etc/rsyslog.d -- then, this subpackage would provide those files. Unfortunately, in order to preserve behavior on upgrade, the main package would have to depend on this, kind of making the distinction moot.
You don't need rsyslog to require rsyslog-put-your-files-here-config, you just need both of them to have Obsoletes for the versions before the split.
Bill
On Thu, Jul 18, 2013 at 04:11:47PM -0400, Bill Nottingham wrote:
In an ideal view, it makes most sense to provide the rsyslog default configuration in a subpackage which puts the /var/log/messages and /var/log/secure conf files in /etc/rsyslog.d -- then, this subpackage would provide those files. Unfortunately, in order to preserve behavior on upgrade, the main package would have to depend on this, kind of making the distinction moot.
You don't need rsyslog to require rsyslog-put-your-files-here-config, you just need both of them to have Obsoletes for the versions before the split.
Oooh, clever!
On 07/18/2013 06:11 PM, Matthew Miller wrote:
On Thu, Jul 18, 2013 at 02:19:30PM +0200, Lennart Poettering wrote:
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question? Sounds good to me! Matthew?
My main concern with this is that it's a lie. That file only exists because of the default configuration. In many cases, rsyslog will be configured to either write different files, or most likely, to write no local files at all as all data is forwarded. And, as discussed in another subthread, I expect this last configuration to be more and more common. So, not just a lie, but a lie which may actually make it harder to use rsyslog in ways other than the default.
So... /var/log/messages is not guaranteed to be there even now, because it depends on rsyslog configuration. So any packages which cannot handle missing /var/log/messages are broken already in a non-default (but probably not all that uncommon) config, and nobody is crying out about that.
This whole logfile provides/requires thing seems mostly like a solution in search of a problem to me.
- Panu -
Panu Matilainen (pmatilai@laiskiainen.org) said:
So... /var/log/messages is not guaranteed to be there even now, because it depends on rsyslog configuration. So any packages which cannot handle missing /var/log/messages are broken already in a non-default (but probably not all that uncommon) config, and nobody is crying out about that.
Yes, it's just that if we're moving these packages from being broken in a non-default, administrator-configured configuration to being broken in the out-of-the-box configuration, it might be worthwhile to know that ahead of time, or know if there's a way to have them indicate that they now require the non-default configuration.
I guess it's the equivalent of marking packages that don't work unless SELinux is enabled, or don't work unless the firewall is off. There are likely better ways to fix them.
Bill
On Thu, Jul 18, 2013 at 04:35:22PM -0400, Bill Nottingham wrote:
Yes, it's just that if we're moving these packages from being broken in a non-default, administrator-configured configuration to being broken in the out-of-the-box configuration, it might be worthwhile to know that ahead of time, or know if there's a way to have them indicate that they now require the non-default configuration.
I think that the updated https://fedoraproject.org/wiki/Changes/NoDefaultSyslog#Dependencies is a pretty good representation of the packages in the distribution that actually might be affected.
It's basically the case that we've NEVER had good tools for dealing with log messages. Epylog was the closest to interesting, but the devoper (hi Konstantin!) lost interest. So, we've got a handful of little utilities. We'll probably discover a few more than the above list, but I doubt that there's even _two_ handfuls.
On 07/18/2013 11:35 PM, Bill Nottingham wrote:
Panu Matilainen (pmatilai@laiskiainen.org) said:
So... /var/log/messages is not guaranteed to be there even now, because it depends on rsyslog configuration. So any packages which cannot handle missing /var/log/messages are broken already in a non-default (but probably not all that uncommon) config, and nobody is crying out about that.
Yes, it's just that if we're moving these packages from being broken in a non-default, administrator-configured configuration to being broken in the out-of-the-box configuration, it might be worthwhile to know that ahead of time, or know if there's a way to have them indicate that they now require the non-default configuration.
Yes, I understand the point behind this, its just that a pair of (file) dependencies doesn't really make things "just work" either.
A file provide basically says "this package contains file X", but here the presence of the promised file depends on a service configuration *and* that the service is running (and has been so for some time).
The file presence could be ensured by doing 'touch /var/log/message' in %post, but that doesn't make the file any more usable in the sense eg hplip and lvm2 seem to expect.
Then there's the question (raised by Lennart) of what eg yum will do with virtual file dependencies outside the "normal" paths, plus some other (perhaps academic in this context) issues wrt virtual file provides vs real files.
I guess all I'm saying that if dependencies are going to be used to address the problem, a non-file dependency seems like a better option to me. Perhaps "Provides: logfile(messages)" or such. That way depsolvers will know full filelists will not be needed to resolve it, its easily distinguishable and the semantics it represents can be clearly documented.
I guess it's the equivalent of marking packages that don't work unless SELinux is enabled, or don't work unless the firewall is off. There are likely better ways to fix them.
Yeah... Log analyzers are one thing, but I dont know what eg hplip and lvm2 are doing with the info scraped out of /var/log/messages, it seems somewhat suspect at least.
- Panu -
On Fri, Jul 19, 2013 at 10:26:24AM +0300, Panu Matilainen wrote:
Yeah... Log analyzers are one thing, but I dont know what eg hplip and lvm2 are doing with the info scraped out of /var/log/messages, it seems somewhat suspect at least.
They're just gathering it into a diagnostic report -- not used for normal functioning.
On 07/18/2013 06:19 AM, Lennart Poettering wrote:
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
But what is going to require /var/log/messages? Programs like logwatch and logrotate can operate on *any* file in /var/log (or elsewhere for that matter). They can be useful without /var/log/messages.
On Thu, 18.07.13 11:30, Orion Poplawski (orion@cora.nwra.com) wrote:
On 07/18/2013 06:19 AM, Lennart Poettering wrote:
On Thu, 18.07.13 10:34, Vít Ondruch (vondruch@redhat.com) wrote:
Dne 18.7.2013 01:02, Lennart Poettering napsal(a):
So, maybe, instead of dropping the "Provides syslog" thing from journald, maybe we should add an explicit "syslog-files" dependency (or something named like that) and then make the classic syslog implementations provide that and the packages which actually need /var/log/messages pull that it?
Lennart
So why there are files in /var/log and there is not obvious package, which creates them (unless you want to guess by name)? Shouldn't all package, which creates log in /var/log have some virtual provide to make it obvious? Why not do it properly/consistently?
So, you suggest using "Requires: /var/log/messages" and "Provides: /var/log/messages" as indication for this, and the %ghost /var/log/messages in the packages in question?
Sounds good to me! Matthew?
Lennart
But what is going to require /var/log/messages? Programs like logwatch and logrotate can operate on *any* file in /var/log (or elsewhere for that matter). They can be useful without /var/log/messages.
logrotate certainly shouldn require this, as in its defualt config it does not reference /var/log/messages at all.
logwatch probably should require it as long as it doesn't "speak journal", as the data its rules look for is in /var/log/messages.
Lennart
On 07/17/2013 04:05 PM, Matthew Miller wrote:
I'm reading this to mean by adding package dependencies where missing, right?
This is actually Kinda Hard, because /var/log/messages is really more an artifact of our default rsyslog config than part of any package. If we wanted to say that some syslog service always provides that file, we should make it clear that changing it is actually not available for end-user configuration and should never been changed. We've never done that.
So, adding "Requires: rsyslog" isn't_really_ sufficient. We could ship the /var/log/messages configuration in a subpackage that goes into /etc/rsyslog.d/ and ask packages which expect /var/log/messages to require that.
Solved via virtual provides and sub-package for components, like foo-rsyslogd.rpm foo-syslog-ng.rpm and foo-logwatch.rpm ( logwatch is broken as well as are many other components that seem to just magically expect certain things being in the base/coreOS indefinitely ) in anycase this could have been fixed already in the time FPC has been sitting on the ticket that proposes just that for what close to 2 years now.
Reason
"2) The FPC defers on the second part (subpackaging options for rsyslog) until rsyslog is no longer installed by default in Fedora. " <sigh>
JBG
1. https://fedorahosted.org/fpc/ticket/142 2. https://fedoraproject.org/wiki/User:Johannbg/Packaging/LogFiles
On Wed, 17.07.13 11:41, Bill Nottingham (notting@redhat.com) wrote:
Matthew Miller (mattdm@fedoraproject.org) said:
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
But, at the packaging level, I think it's been demonstrated that 'a program that expects this file' is not something we do a good job of consistently expressing. That should be fixed regardless of this feature proposal, but I think should definitely be a required component of accepting it.
Note that the feature proposal does actually mention this explicitly: we might need some corrected dependencies in some packages. It's a needed for the feature but even independently of the feature is the right thing to do.
Lennart
On Tue, 2013-07-16 at 21:13 -0400, Matthew Miller wrote:
On Tue, Jul 16, 2013 at 08:58:40PM -0400, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
A) If someone is installing a program that expects this file, they can also install rsyslog.
B) Fedora, RHEL, and most Red Hat derived distributions use /var/log/messages, but not all do -- for example, Rocks (common in HPC) breaks out syslog messages into individual files per facility. Debian and Ubuntu? Totally diferent. (/var/log/syslog)
So, these third-party scripts need to be flexible anyway. I don't think this is a very strong point in the conversation.
FWIW, FHS states[1]:
"The following files, or symbolic links to files, must be in /var/log, if the corresponding subsystem is installed:
File Description lastlog record of last login of each user messages system messages from syslogd"
There is clearly some ambiguity there; I'm not sure FHS writers considered the possibility of something that is clearly meant to fulfill the role of a system logging daemon but which does not want to write /var/log/messages .
[1] http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#VARLOGLOGFILESANDDI...
On Wed, Jul 17, 2013 at 01:27:23PM -0700, Adam Williamson wrote:
FWIW, FHS states[1]:
"The following files, or symbolic links to files, must be in /var/log, if the corresponding subsystem is installed:
File Description lastlog record of last login of each user messages system messages from syslogd"
There is clearly some ambiguity there; I'm not sure FHS writers considered the possibility of something that is clearly meant to fulfill the role of a system logging daemon but which does not want to write /var/log/messages .
The intent here is "if such a file exists, it belongs in this directory", not "this file must exist". This pattern repeats throughout the FHS.
See for example the similar requirements for /bin: http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#SPECIFICOPTIONS2 csh is listed, but this isn't meant to mandate that C shell must be installed. So I think we're compliant with both the letter and the spirit of this standard.
On Tue, 16.07.13 20:58, Ding Yi Chen (dchen@redhat.com) wrote:
He/She will be very frustrated if he/she cannot find it.
This is not really an argument, or it is valid for any changes. What about the switch from up2date to yum years ago? I know, I first tried to run up2date before being shout at that it's normal it does not work anymore since yum is there. What about the future change from yum to dnf? Should we try to keep both in place just because google has more reference to yum?
So, the fact that tutorials and blogs are mentioning /var/log/messages is not a valid argument :)
You still have not addressed the third party programs and scripts that monitor /var/log/messages
As mentioned before, if people run programs that require /var/log/messages they should simply install rsyslog and be done with it.
Lennart
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require /var/log/messages they should simply install rsyslog and be done with it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files. Even when you think it's better, many *do not*.
Regards,
Marc Deop
On Wed, Jul 17, 2013 at 12:06 PM, Marc Deop i Argemí marc@marcdeop.com wrote:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require
/var/log/messages they should simply install rsyslog and be done with
it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files. Even when you think it's better, many *do not*.
And many do ... so?
On 07/17/2013 12:06 PM, Marc Deop i Argemí wrote:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require
/var/log/messages they should simply install rsyslog and be done with
it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files.
So do I.
IMO, log files need to be readable without any special tools, because log files often are being read in cases of emergencies/breakdown (occasionally even from other OSes), when one can not rely on the tools being available or usable.
Even when you think it's better, many *do not*.
ACK-
Ralf
On Wed, Jul 17, 2013 at 02:52:40PM +0200, Ralf Corsepius wrote:
On 07/17/2013 12:06 PM, Marc Deop i Argemí wrote:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require
/var/log/messages they should simply install rsyslog and be done with
it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files.
So do I.
IMO, log files need to be readable without any special tools, because log files often are being read in cases of emergencies/breakdown (occasionally even from other OSes), when one can not rely on the tools being available or usable.
Even when you think it's better, many *do not*.
ACK-
Ralf
Is there a way to read binary journals from non-Linux OSes?
On 07/17/2013 02:56 PM, Chuck Anderson wrote:
On Wed, Jul 17, 2013 at 02:52:40PM +0200, Ralf Corsepius wrote:
On 07/17/2013 12:06 PM, Marc Deop i Argemí wrote:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require
/var/log/messages they should simply install rsyslog and be done with
it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files.
So do I.
IMO, log files need to be readable without any special tools, because log files often are being read in cases of emergencies/breakdown (occasionally even from other OSes), when one can not rely on the tools being available or usable.
Even when you think it's better, many *do not*.
ACK-
Ralf
Is there a way to read binary journals from non-Linux OSes?
No, but there are ways to read Linux logs from other OSes. Should be sufficient reason not to imitate their flawed design and be sufficient reasons to curse binary logs as "broken designs".
Ralf
On Wed, Jul 17, 2013 at 08:56:45AM -0400, Chuck Anderson wrote:
On Wed, Jul 17, 2013 at 02:52:40PM +0200, Ralf Corsepius wrote:
On 07/17/2013 12:06 PM, Marc Deop i Argemí wrote:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require
/var/log/messages they should simply install rsyslog and be done with
it.
That terribly sounds like "my way or the high way".
Many people have raised concerns not only about having /var/log/messages in the base installation but about needing/wanting *simple text* logs by default as well.
I for one, advocate against having to use a (forced)tool to read log files.
So do I.
Are you also against compression of log files?
IMO, log files need to be readable without any special tools, because log files often are being read in cases of emergencies/breakdown (occasionally even from other OSes), when one can not rely on the tools being available or usable.
Even when you think it's better, many *do not*.
ACK-
Ralf
Is there a way to read binary journals from non-Linux OSes?
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Zbyszek
On Wed, Jul 17, 2013 at 03:30:20PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 17, 2013 at 08:56:45AM -0400, Chuck Anderson wrote:
Is there a way to read binary journals from non-Linux OSes?
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Zbyszek
I was more thinking of the typical home use case, where a user has installed Fedora alongside Windows or Mac OS. The latter is UNIXy, but the former is not. Maybe as part of this feature, Fedora should release Windows packages of journalctl or a similar tool for reading the logs.
On Wed, Jul 17, 2013 at 09:35:33AM -0400, Chuck Anderson wrote:
On Wed, Jul 17, 2013 at 03:30:20PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 17, 2013 at 08:56:45AM -0400, Chuck Anderson wrote:
Is there a way to read binary journals from non-Linux OSes?
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Zbyszek
I was more thinking of the typical home use case, where a user has installed Fedora alongside Windows or Mac OS. The latter is UNIXy, but the former is not. Maybe as part of this feature, Fedora should release Windows packages of journalctl or a similar tool for reading the logs.
Usually the logs are on ext4 or btrfs partition. Getting the files out of there without running linux is enough of a technological challenge fo most poeple, that actually decoding the files is trivial in comparison. I'm pretty sure journalctl for Windows would find very little use.
Zbyszek
On 07/17/2013 03:35 PM, Chuck Anderson wrote:
On Wed, Jul 17, 2013 at 03:30:20PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 17, 2013 at 08:56:45AM -0400, Chuck Anderson wrote:
Is there a way to read binary journals from non-Linux OSes?
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Zbyszek
I was more thinking of the typical home use case, where a user has installed Fedora alongside Windows or Mac OS.
My use case is using a different Linux distro for trouble shooting, either installed alongside Fedora[1] or booting it from usb-stick/CD/DVD [2] or external HD[3].
Ralf
[1] Often older versions of Fedora or CentOS. [2] e.g. systemrescuecd [3] There has been a time, I carried around a customized CentOS installation on external HD just for troubleshooting.
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Other than 'because the same command as it stands now talks to journald itself', why would a reader for other OSes require dbus and libcap?
(i.e., if you're targeting a foreign reader, I think it should be simpler.)
Bill
On Wed, Jul 17, 2013 at 11:37:16AM -0400, Bill Nottingham wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Other than 'because the same command as it stands now talks to journald itself', why would a reader for other OSes require dbus and libcap?
(i.e., if you're targeting a foreign reader, I think it should be simpler.)
That's a limitation of the build system, it doesn't have enough "smarts" to check just the dependencies of one of the many binaries that are build. journalctl itself doesn't use dbus or libcap, or talk to journald for that matter.
Zbyszek
On Wed, 17.07.13 11:37, Bill Nottingham (notting@redhat.com) wrote:
Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) said:
Compiling journalctl on UINXy OSes should not be too much of a problem. You'd need dbus >= 1.4.0, libcap, liblzma (if used by the journal files in question). Not trivial but certainly doable.
Other than 'because the same command as it stands now talks to journald itself', why would a reader for other OSes require dbus and libcap?
(i.e., if you're targeting a foreign reader, I think it should be simpler.)
journalctl doesn't need libdbus either, it doesn't link to it. journalctl in fact never communicates with journald via IPC. It accesses the files directly. This means you can share a journal file over NFS or so, and have journald running on one host and journalctl on the other, and things will just work, as no other communication is necessary beyond the FS itself.
The lzma and libcgrypt deps are optional deps. lzma is really useful as without it you couldn't read journal files that contain coimpressed objects, however, gcrypt matters only if you want to use FSS.
Lennart
Once upon a time, Marc Deop i Argemí marc@marcdeop.com said:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require /var/log/messages they should simply install rsyslog and be done with it.
That terribly sounds like "my way or the high way".
That is Lennart's standard behavior.
On Wed, 17.07.13 07:57, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Marc Deop i Argemí marc@marcdeop.com said:
On Wednesday 17 July 2013 03:20:42 Lennart Poettering wrote:
As mentioned before, if people run programs that require /var/log/messages they should simply install rsyslog and be done with it.
That terribly sounds like "my way or the high way".
That is Lennart's standard behavior.
And the thread just went ugly.
Not sure what the trigger for getting personal like this is. Do you feel we have the better arguments so you resort to this escape? It certainly isn't the brightest of your arguments if you turn a technical discussion into personal attacks.
Lennart
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Wed, 17.07.13 07:57, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Marc Deop i Argemí marc@marcdeop.com said:
That terribly sounds like "my way or the high way".
That is Lennart's standard behavior.
And the thread just went ugly.
Not sure what the trigger for getting personal like this is. Do you feel we have the better arguments so you resort to this escape? It certainly isn't the brightest of your arguments if you turn a technical discussion into personal attacks.
I quote from your email yesterday:
######################################################################## But anyway, the auto-paging thing is going to stay, you can talk about it as much as you want. Why? Simply because *I* love it. It's one awesome feature. You are welcome to disagree, but discussing this forth and back on fedora-devel is highly unlikely to change my mind on this. As long as I maintain it, this one feature definitely stays in. Sorry for that! ########################################################################
How is that not "my way or the highway"?
On 07/17/2013 01:21 PM, Chris Adams wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Wed, 17.07.13 07:57, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Marc Deop i Argemí marc@marcdeop.com said:
That terribly sounds like "my way or the high way".
That is Lennart's standard behavior.
And the thread just went ugly.
Not sure what the trigger for getting personal like this is. Do you feel we have the better arguments so you resort to this escape? It certainly isn't the brightest of your arguments if you turn a technical discussion into personal attacks.
I quote from your email yesterday:
######################################################################## But anyway, the auto-paging thing is going to stay, you can talk about it as much as you want. Why? Simply because *I* love it. It's one awesome feature. You are welcome to disagree, but discussing this forth and back on fedora-devel is highly unlikely to change my mind on this. As long as I maintain it, this one feature definitely stays in. Sorry for that! ########################################################################
How is that not "my way or the highway"?
Despite Lennart's love for it, those that are arguing for this change are actually asking for the default that have been since the introduction of journal to be changed thus disrupting the workflow for everybody that have gotten accustom to use it that way.
JBG
On Wed, Jul 17, 2013 at 03:14:09PM +0200, Lennart Poettering wrote:
On Wed, 17.07.13 07:57, Chris Adams (linux@cmadams.net) wrote:
That terribly sounds like "my way or the high way".
That is Lennart's standard behavior.
And the thread just went ugly.
Not sure what the trigger for getting personal like this is. Do you feel we have the better arguments so you resort to this escape? It certainly isn't the brightest of your arguments if you turn a technical discussion into personal attacks.
"Not. Gonna. Happen."
Am 17.07.2013 11:21, schrieb Jóhann B. Guðmundsson:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs
you could if you want
* only journald is running -> write to "/var/log/messages" and "/var/log/secure" additionally to the journal a option in "/etc/systemd/journald.conf" to disable this
* if a syslog-daemon is running leave "/var/log/messages" and "/var/log/secure" untouched from journald
so *if* you want you can keep progress without a large impact all the time
On Wed, Jul 17, 2013 at 12:00:05PM +0200, Reindl Harald wrote:
Am 17.07.2013 11:21, schrieb Jóhann B. Guðmundsson:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs
you could if you want
only journald is running -> write to "/var/log/messages" and "/var/log/secure" additionally to the journal a option in "/etc/systemd/journald.conf" to disable this
if a syslog-daemon is running leave "/var/log/messages" and "/var/log/secure" untouched from journald
so *if* you want you can keep progress without a large impact all the time
Except ... that would still keep duplicated logs on the FS. Removing the duplication is the primary reason for wanting to not install rsyslogd by default.
Zbyszek
On 07/17/2013 11:05 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 17, 2013 at 12:00:05PM +0200, Reindl Harald wrote:
Am 17.07.2013 11:21, schrieb Jóhann B. Guðmundsson:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs
you could if you want
only journald is running -> write to "/var/log/messages" and "/var/log/secure" additionally to the journal a option in "/etc/systemd/journald.conf" to disable this
if a syslog-daemon is running leave "/var/log/messages" and "/var/log/secure" untouched from journald
so *if* you want you can keep progress without a large impact all the time
Except ... that would still keep duplicated logs on the FS. Removing the duplication is the primary reason for wanting to not install rsyslogd by default.
Zbyszek
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
-- John Florian
On 07/17/2013 11:21 AM, John.Florian@dart.biz wrote:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
-- John Florian
So you run fedora on an embedded system?
You don't ever work with embedded systems, do you?
I was going to respond with VM guests which generally have small disks...
But to be fair the proponents of the change say "aren't you customising the system anyway? just yum install rsyslogd if you want it" and the same would apply in reverse... "if you are doing VM/embedded you restrict packages anyway... just don't install rsyslogd!"
James Hogarth (james.hogarth@gmail.com) said:
I was going to respond with VM guests which generally have small disks...
But to be fair the proponents of the change say "aren't you customising the system anyway? just yum install rsyslogd if you want it" and the same would apply in reverse... "if you are doing VM/embedded you restrict packages anyway... just don't install rsyslogd!"
To be fair, adding packages to the minimal install is generally seen as a simpler, more supported, configuration than removing packages from the minimal install.
Bill
Am 17.07.2013 17:21, schrieb John.Florian@dart.biz:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
and there you can not distribute a config file with the option to disable this?
having the *option* do the double-log in journald to keep /var/log/messages alive would allow a *lot* of more users to remove rsyslog as with the "eat or die" proposal leading to have rsyslog manually installed on many systems
On Wed, 17.07.13 17:41, Reindl Harald (h.reindl@thelounge.net) wrote:
Am 17.07.2013 17:21, schrieb John.Florian@dart.biz:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
and there you can not distribute a config file with the option to disable this?
having the *option* do the double-log in journald to keep /var/log/messages alive would allow a *lot* of more users to remove rsyslog as with the "eat or die" proposal leading to have rsyslog manually installed on many systems
I am not sure what the benefit of this would be. I mean, journald comes with a built-in option for double-logging and keeping /var/log/messages around. It's easily accessible with one command, which is "yum install rsyslog". Extremely convenient.
Lennart
On 07/17/2013 05:21 PM, John.Florian@dart.biz wrote:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
If you are running systemd on a embedded system, you are clearly not concerned about saving space :)
On Wed, 17.07.13 17:50, Denys Vlasenko (dvlasenk@redhat.com) wrote:
On 07/17/2013 05:21 PM, John.Florian@dart.biz wrote:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
If you are running systemd on a embedded system, you are clearly not concerned about saving space :)
There are actually quite a few embedded devices running systemd these days. Wind generators, outer space telescopes, cars, toys, quite a lot of other stuff. We do get reports about this from time to time.
And they do care about disk space, and since systemd is actually not that bad on that. systemd is quite modular and you can select at build time the components you need. And given that systemd already provides you with most things you need for a device, and reuses a lot of code internally the overall footprint is quite OK. The hierarchal watchdog support in systemd actually originates from embedded people, and they love it.
This is not the kind of embedded that only consists of one kernel and one process, but it is the embedded that runs a small number of services, and systemd is quite suitable for that, and in production already.
I actually attended a number of embedded conferences representing systemd. For example I attended the GENIVI conference twice. They work on standardization of Linux for car IVI systems. The GENIVI specs actually require systemd to be in it now.
So, no snarky comments about embedded devices, please, it's entirely inappropriate.
Lennart
On 07/18/2013 12:40 AM, Lennart Poettering wrote:
On Wed, 17.07.13 17:50, Denys Vlasenko (dvlasenk@redhat.com) wrote:
On 07/17/2013 05:21 PM, John.Florian@dart.biz wrote:
From: sclark@netwolves.com
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
You don't ever work with embedded systems, do you?
If you are running systemd on a embedded system, you are clearly not concerned about saving space :)
There are actually quite a few embedded devices running systemd these days. Wind generators, outer space telescopes, cars, toys, quite a lot of other stuff. We do get reports about this from time to time.
Did I say systemd can't be run on an embedded device?
I said that if one runs systemd on a embedded system, then this device isn't seriously resource constrained.
Do you see that these two statements are not the same?
So, no snarky comments about embedded devices, please, it's entirely inappropriate.
What's inappropriate is giving instructions to others what they can, or can not say.
On 07/17/2013 03:16 PM, Steve Clark wrote:
This seems like such a specious argument. Maybe it made sense when we were talking about disk drives that were megabytes in size, but now we have 500 gigabyte drives usually as a minimum.
Well embedded devices do not contain 500GB as well cloud images and virtualsation images in general...
JBG
On 07/17/2013 05:16 PM, Steve Clark wrote:
On 07/17/2013 11:05 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Jul 17, 2013 at 12:00:05PM +0200, Reindl Harald wrote:
Am 17.07.2013 11:21, schrieb Jóhann B. Guðmundsson:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs
you could if you want
only journald is running -> write to "/var/log/messages" and "/var/log/secure" additionally to the journal a option in "/etc/systemd/journald.conf" to disable this
if a syslog-daemon is running leave "/var/log/messages" and "/var/log/secure" untouched from journald
so *if* you want you can keep progress without a large impact all the time
Except ... that would still keep duplicated logs on the FS. Removing the duplication is the primary reason for wanting to not install rsyslogd by default.
This seems like such a specious argument.
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
My machine:
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 4.5M 3.9G 1% /run tmpfs 3.9G 4.9M 3.9G 1% /tmp
10 megs of *RAM* consumed.
My /var/log/messages is 12 megabytes at the moment.
These same people feel offended by "wasted" 12 megs of *disk space*? Please...
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
My machine:
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 4.5M 3.9G 1% /run tmpfs 3.9G 4.9M 3.9G 1% /tmp
10 megs of *RAM* consumed.
This was more about the cleanest way to have /tmp absolutely clean at start though I always thought .... Anything that sits in there for a while with little access will be paged out anyway...
On Wed, Jul 17, 2013 at 05:48:57PM +0200, Denys Vlasenko wrote:
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
That assumption is not true. (In fact, we disable tmpfs /tmp in the Fedora Cloud image.)
On 07/17/2013 03:48 PM, Denys Vlasenko wrote:
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
So you prefer to have a fragile boot code to empty /run and do you want to be consistent with what other distribution suse/debian/arch and solaris are doing?
If you want to disable tmp on tmpfs simple run|"||systemctl mask tmp.mount" |||
My machine:
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 4.5M 3.9G 1% /run tmpfs 3.9G 4.9M 3.9G 1% /tmp
10 megs of*RAM* consumed.
My /var/log/messages is 12 megabytes at the moment.
These same people feel offended by "wasted" 12 megs of*disk space*? Please...
Yes 10 megs of *RAM* consumed by *you* on *your* machine based on *your* setup
Same folders on my fully updated always running F18 work laptop
tmpfs 1.9G 1.2M 1.9G 1% /run tmpfs 1.9G 372K 1.9G 1% /tmp
less then 2MB of *RAM* consumed
du -hs /var/log/ --exclude=/var/log/journal 6.1M /var/log/
And 6.1M of wasted diskspace on my SSD
And your point being?
JBG
On 07/17/2013 06:49 PM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 03:48 PM, Denys Vlasenko wrote:
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
So you prefer to have a fragile boot code to empty /run and do you want to be consistent with what other distribution suse/debian/arch and solaris are doing?
If you want to disable tmp on tmpfs simple run|"||systemctl mask tmp.mount" |||
My machine:
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 4.5M 3.9G 1% /run tmpfs 3.9G 4.9M 3.9G 1% /tmp
10 megs of *RAM* consumed.
My /var/log/messages is 12 megabytes at the moment.
These same people feel offended by "wasted" 12 megs of *disk space*? Please...
Yes 10 megs of *RAM* consumed by *you* on *your* machine based on *your* setup
Same folders on my fully updated always running F18 work laptop
tmpfs 1.9G 1.2M 1.9G 1% /run tmpfs 1.9G 372K 1.9G 1% /tmp
less then 2MB of *RAM* consumed
du -hs /var/log/ --exclude=/var/log/journal 6.1M /var/log/
And 6.1M of wasted diskspace on my SSD
And your point being?
My point is that it is hypocritical to decry a "huge performance cost" of /var/log/messages one day and next day eat many megabytes of RAM and claim that "it's not a big deal".
On Wed, 17.07.13 17:48, Denys Vlasenko (dvlasenk@redhat.com) wrote:
Except ... that would still keep duplicated logs on the FS. Removing the duplication is the primary reason for wanting to not install rsyslogd by default.
This seems like such a specious argument.
Note that the argument comes from the same group of people who pushed for mounting tmpfs on /run and /tmp.
My machine:
# df -h Filesystem Size Used Avail Use% Mounted on tmpfs 3.9G 4.5M 3.9G 1% /run tmpfs 3.9G 4.9M 3.9G 1% /tmp
10 megs of *RAM* consumed.
Note that tmpfs is backed by swappable memory. So it's backed by your swap partition and paged in only when needed or there's nothing else to store in RAM.
Lennart
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
The whole purpose of distros (of *computers*) is to run software, some of it not included in the distro, some it highly customized scripts that people have written themselves.
Text log files are not some sort of esoteric feature that hardly anyone uses.
Rich.
On 07/17/2013 12:05 PM, Richard W.M. Jones wrote:
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
Dumb I see so you have established a time frame for us how long we should hold back progress in the project and or you have devised an implementation plan on features and cleanups with a rate that a third party can keep up with in the distribution, maybe even chosen which third parties we wait for and which we dont?
You think it's good for the community to be dependent on third party I dont since think we should first and foremost be thinking about ourselves and our community not some third party of the interweb or even a downstream distribution to us like like RHEL.
We as a community need to be able to set the pace for ourselves and the fact is unless you are closed source the best thing you can do as a third party is actually participate in the Fedoraproject, packaging you software or application stack and ship it within the distribution so that our existing processes will catch any fallout which our features or cleanups might bring and allow for the community to actually fix it with your or for you.
JBG
Am 17.07.2013 14:20, schrieb Jóhann B. Guðmundsson:
On 07/17/2013 12:05 PM, Richard W.M. Jones wrote:
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
Dumb I see so you have established a time frame for us how long we should hold back progress in the project and or you have devised an implementation plan on features and cleanups with a rate that a third party can keep up with in the distribution, maybe even chosen which third parties we wait for and which we dont?
You think it's good for the community to be dependent on third party I dont since think we should first and foremost be thinking about ourselves and our community not some third party of the interweb or even a downstream distribution to us like like RHEL
your definition of "community" needs to be fixed because with this statement you exclude a) users and b) even packages and maintainers inside the project from the community
you need to understand that a distribution does not exist self-contained it exists and survives because it is used and you can hardly come out twice a year and break happily whatever people are using because you think it's the way to go - well, you can, but if you do it too often you will sooner or later be the community at all
/var/log/messages has a lot of usecases
you can *hardly* demand from every developer and sysadmin on this planet permanently follow your way because it's the better for you
On 07/17/2013 08:20 AM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:05 PM, Richard W.M. Jones wrote:
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
Dumb I see so you have established a time frame for us how long we should hold back progress in the project and or you have devised an implementation plan on features and cleanups with a rate that a third party can keep up with in the distribution, maybe even chosen which third parties we wait for and which we dont?
You think it's good for the community to be dependent on third party I dont since think we should first and foremost be thinking about ourselves and our community not some third party of the interweb or even a downstream distribution to us like like RHEL.
We as a community need to be able to set the pace for ourselves and the fact is unless you are closed source the best thing you can do as a third party is actually participate in the Fedoraproject, packaging you software or application stack and ship it within the distribution so that our existing processes will catch any fallout which our features or cleanups might bring and allow for the community to actually fix it with your or for you.
JBG
But it seems the community is only the people driving all these changes, what about the "whole user" community, not just the developer community
On Wed, Jul 17, 2013 at 2:20 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
On 07/17/2013 12:05 PM, Richard W.M. Jones wrote:
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
Dumb I see so you have established a time frame for us how long we should hold back progress in the project and or you have devised an implementation plan on features and cleanups with a rate that a third party can keep up with in the distribution, maybe even chosen which third parties we wait for and which we dont?
Progress does not that frequently depend on removing older functionality. Specifically in this case, removing rsyslog does not make journal in any way better.
We as a community need to be able to set the pace for ourselves and the fact is unless you are closed source the best thing you can do as a third party is actually participate in the Fedoraproject, packaging you software or application stack and ship it within the distribution so that our existing processes will catch any fallout which our features or cleanups might bring and allow for the community to actually fix it with your or for you.
The "we have source, we can improve the API, update all users and end up with a cleaner design" argument is very rarely true in practice outside of fairly tightly coordinating communities (of which the Linux kernel and its approach to internal interfaces is probably the most visible example).
In fact, we as a distribution are getting _worse and worse_ in this regard. There are, instead, more and more subsystems that ship parallel-installable versions with no specific plans about ever fixing "all users" - sometimes apparently just hoping that both the unported users and older versions of the subsystem will die of neglect at approximately the same time.
The argument about updating all users is just not true for proprietary applications, binary-only applications, or cross-platform applications with a release schedule significantly different from Fedora. (Some people think that the Linux ecosystem should be entirely Open Source, so they don't find this relevant. I'd rather not extend this huge thread into a discussion of this particular difference of opinion.)
Finally, as a thought experiment - we have a fairly well developed concept of automated testing, and we (so far?) have Moore's law making CPU, storage and the cost of testing exponentially cheaper over time; it just might be the right thing to do to provide essentially infinite backward compatibility, in the same way the newest Windows can run 64-bit applications completely natively, 32-bit applications almost natively, 16-bit applications in a VM or full software emulation, or even fully emulate old 8-bit systems.
The same thinking applies to individual sets of APIs and other interfaces: write the new implementation, write a compatibility layer for old users that replaces the old implementation, write a test suite of the compatibility layer (... or just use the test suite of the implementation thing that you should have already), keep the compatibility layer shipped and running and forget about the transition. Writing a compatibility layer is roughly the same kind of work as porting applications, so with any interface that has at least a handful of users a single compatibility layer should in fact be less work. With this approach, it's not at all obvious that one shouldn't aim for a backward compatibility 100 years back[1][2]. Mirek
[1] One can't promise the backward compatibility 100 years into the future, though, no more that one can promise that any particular town/nation/language/building will still exist. [2] Don't worry, I'm not at all proposing that Fedora should aim for a backward compatibility 100 years back. I do want to seriously undermine the idea that progress requires removing or breaking things, though.
On 07/19/2013 06:12 PM, Miloslav Trmač wrote:
Progress does not that frequently depend on removing older functionality. Specifically in this case, removing rsyslog does not make journal in any way better.
Perhaps not that' s a matter of opinion but to we should be able to compare it against the discussion and decision that was made when it was declared that rsyslog should become our syslogger over for example syslog-ng to see if it at least meats those standards.
In anycase the fact cannot be ignored or denied that it is unnecessary to ship two sysloggers...
JBG
On Fri, Jul 19, 2013 at 1:28 PM, "Jóhann B. Guðmundsson" <johannbg@gmail.com
wrote:
On 07/19/2013 06:12 PM, Miloslav Trmač wrote:
Progress does not that frequently depend on removing older functionality. Specifically in this case, removing rsyslog does not make journal in any way better.
Perhaps not that' s a matter of opinion but to we should be able to compare it against the discussion and decision that was made when it was declared that rsyslog should become our syslogger over for example syslog-ng to see if it at least meats those standards.
This is *not the same thing*. In case anyone here is unaware, rsyslog and syslog-ng are exclusive. ${Syslog} and systemd are not.
In anycase the fact cannot be ignored or denied that it is unnecessary to ship two sysloggers...
I haven't seen anyone asking to ship two sysloggers.
On 07/19/2013 06:45 PM, Billy Crook wrote:
I haven't seen anyone asking to ship two sysloggers.
I perhaps should have been clearer and say "two logging systems" which we currently are doing and one of those cannot be disabled or removed so the logical choice is to remove the one that can and make him as an option to be install later.
JBG
On 07/19/2013 02:56 PM, "Jóhann B. Guðmundsson" wrote:
On 07/19/2013 06:45 PM, Billy Crook wrote:
I haven't seen anyone asking to ship two sysloggers.
I perhaps should have been clearer and say "two logging systems" which we currently are doing and one of those cannot be disabled or removed so the logical choice is to remove the one that can and make him as an option to be install later.
JBG
This might have merit if the one you want to keep could do everything it does plus what the one you want to remove does.
On 07/19/2013 07:11 PM, Steve Clark wrote:
This might have merit if the one you want to keep could do everything it does plus what the one you want to remove does.
And to establish if it does that, we need to know what the deceive factor was of choosing rsyslog in the first place over syslog-ng and other logging systems.
JBG
"Jóhann B. Guðmundsson" (johannbg@gmail.com) said:
On 07/19/2013 07:11 PM, Steve Clark wrote:
This might have merit if the one you want to keep could do everything it does plus what the one you want to remove does.
And to establish if it does that, we need to know what the deceive factor was of choosing rsyslog in the first place over syslog-ng and other logging systems.
rsyslog was chosen over syslog-ng at the time because of better compatibility with sysklogd. I'd have to dig back to check, but I think the config file was part of it.
Bill
On Jul 19, 2013 2:11 PM, "Steve Clark" sclark@netwolves.com wrote:
On 07/19/2013 02:56 PM, "Jóhann B. Guðmundsson" wrote:
On 07/19/2013 06:45 PM, Billy Crook wrote:
I haven't seen anyone asking to ship two sysloggers.
I perhaps should have been clearer and say "two logging systems" which we currently are doing and one of those cannot be disabled or removed so the logical choice is to remove the one that can and make him as an option to be install later.
JBG
This might have merit if the one you want to keep could do everything it
does
plus what the one you want to remove does.
Well put. This is exactly why NoSyslog is premature.
On Fri, Jul 19, 2013 at 02:16:13PM -0500, Billy Crook wrote:
Well put. This is exactly why NoSyslog is premature.
And that's exactly why it's NoDefaultSyslog, not NoSyslog.
On Fri, 19.07.13 20:12, Miloslav Trmač (mitr@volny.cz) wrote:
On Wed, Jul 17, 2013 at 2:20 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
On 07/17/2013 12:05 PM, Richard W.M. Jones wrote:
On Wed, Jul 17, 2013 at 09:21:39AM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:58 AM, Ding Yi Chen wrote:
You still have not addressed the third party programs and scripts that monitor /var/log/messages
We honestly cant keep progress and cleanup in the distribution back out of fear of breaking some third party programs.
Irrespective of whether journald is good or bad, this is a dumb argument.
Dumb I see so you have established a time frame for us how long we should hold back progress in the project and or you have devised an implementation plan on features and cleanups with a rate that a third party can keep up with in the distribution, maybe even chosen which third parties we wait for and which we dont?
Progress does not that frequently depend on removing older functionality. Specifically in this case, removing rsyslog does not make journal in any way better.
It certainly makes *Fedora* better though, by making the core less redundant and decreasing its footprint.
The same thinking applies to individual sets of APIs and other interfaces: write the new implementation, write a compatibility layer for old users that replaces the old implementation, write a test suite of the compatibility layer (... or just use the test suite of the implementation thing that you should have already), keep the compatibility layer shipped and running and forget about the transition. Writing a compatibility layer is roughly the same kind of work as porting applications, so with any interface that has at least a handful of users a single compatibility layer should in fact be less work. With this approach, it's not at all obvious that one shouldn't aim for a backward compatibility 100 years back[1][2]. Mirek
Note that the journal provides this compatbility layer to a fairly comprehensive degree. We speak the syslog protocol natively so that log clients don't have to be updated. We provide a pretty much perfect identical output to /var/log/messages as default from journalctl. We make it easy to to get the exact files /var/log/messages by forwarding everything to syslog instantly if it is running.
However, we also go one step further: eventually we remove the old implementation from the default installation. That's a very gentle push only, because the old stuff is trivially easy to get back, simply by installing rsyslog again.
Lennart
On Tue, 16.07.13 02:27, Ding Yi Chen (dchen@redhat.com) wrote:
And you want to do the system wide search and replace all the 3rd party programs, scripts, and documents that mentioned /var/log/messages?
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
Image the following scenario: Suppose a Fedora newbie (or linux newbie) encounters a problem, most of the search results state: Check /var/log/messages
He/She will be very frustrated if he/she cannot find it.
Well, that's an excuse to *never* change anything because docs might need updating.
Note that the feature includes that we add in a README file to /var/log/ which explains the situation and suggests the command lines to get the logs with journalctl.
We were in a very similar spot for /etc/r.d/init.d, which is empty now, with some exceptions. There we also added a README explaining the situation.
Documentation issues should be treated as documentation issues: i.e. document changes in the release notes, ship the necessary man pages and add helpful help texts to the places people would looking for the original stuff we replaced.
Lennart
On 07/16/2013 07:33 AM, Lennart Poettering wrote:
Note that the feature includes that we add in a README file to /var/log/ which explains the situation and suggests the command lines to get the logs with journalctl.
Perhaps also add /var/log/messages containing this line:
May 16 17:37:20 Fedora: moved to systemd logging; 'journalctl' displays log files
:), but then again it may actually be a good idea...
On Wed, Jul 17, 2013 at 12:04:37PM -0400, Przemek Klosowski wrote:
Note that the feature includes that we add in a README file to /var/log/ which explains the situation and suggests the command lines to get the logs with journalctl.
Perhaps also add /var/log/messages containing this line:
May 16 17:37:20 Fedora: moved to systemd logging; 'journalctl' displays log files
:), but then again it may actually be a good idea...
Yes, not the first time that's been suggested. In fact, not the first time that's been suggested _in this thread!_.
On 07/16/2013 08:27 AM, Ding Yi Chen wrote:
And you want to do the system wide search and replace all the 3rd party programs, scripts, and documents that mentioned /var/log/messages?
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
They already refer to /var/log/syslog and /var/log/daemon.log, which can also be confusing. I'm not sure if this is a significant problem.
On Tue, Jul 16, 2013 at 02:27:24AM -0400, Ding Yi Chen wrote:
And you want to do the system wide search and replace all the 3rd party programs, scripts, and documents that mentioned /var/log/messages?
http://codesearch.debian.net/search?q=%2Fvar%2Flog%2Fmessages
Bam! Done! :)
Even if you do, you cannot change the exist tutorial, blog, and forums that refer /var/log/messages.
Sure. There's a change cost.
Image the following scenario: Suppose a Fedora newbie (or linux newbie) encounters a problem, most of the search results state: Check /var/log/messages He/She will be very frustrated if he/she cannot find it.
One suggestion I've heard is to write this line:
Jan 1 00:00:00 localhost root: Run `journalctl` to see log data in /var/log/messages-compatible format
But I'm not sure that's really a good idea. The plan calls for putting a README file in /var/log.
Once upon a time, Matthew Miller mattdm@fedoraproject.org said:
One suggestion I've heard is to write this line:
Jan 1 00:00:00 localhost root: Run `journalctl` to see log data in /var/log/messages-compatible format
That would be a bad idea, as then you'd have an RPM-installed /var/log/messages that can't be used for "real" logs with rsyslog.
On Tue, Jul 16, 2013 at 09:51:08AM -0500, Chris Adams wrote:
One suggestion I've heard is to write this line: Jan 1 00:00:00 localhost root: Run `journalctl` to see log data in /var/log/messages-compatible format
That would be a bad idea, as then you'd have an RPM-installed /var/log/messages that can't be used for "real" logs with rsyslog.
Right, if this were done, it would just be written, not owned by any RPM, just as /var/log/messages is currently not owned by any package.
On 07/15/2013 08:29 PM, Lars Seipel wrote:
On Mon, Jul 15, 2013 at 02:46:27PM -0600, Eric Smith wrote:
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
If you use bash or ksh you could just replace /var/log/messages with <(journalctl) in your command line and stuff should just work (when reading). Other shells can probably do the same. It obviously depends on journalctl being able to run.
What about scripts that use /usr/bin/logger? Do messages generated by this utility end up in the journal? Or php scripts, or programs using syslog(3).
On Wed, Jul 17, 2013 at 12:31:56PM -0400, Steve Clark wrote:
On 07/15/2013 08:29 PM, Lars Seipel wrote:
On Mon, Jul 15, 2013 at 02:46:27PM -0600, Eric Smith wrote:
I don't actually care whether there's a binary journal or not, but far more of us have real usecases for /var/log/messages, so we shouldn't give up that being available by default.
If you use bash or ksh you could just replace /var/log/messages with <(journalctl) in your command line and stuff should just work (when reading). Other shells can probably do the same. It obviously depends on journalctl being able to run.
What about scripts that use /usr/bin/logger? Do messages generated by this utility end up in the journal? Or php scripts, or programs using syslog(3).
Yes. Even now, all syslog messages pass through journald.
Zbyszek
On Mon, Jul 15, 2013 at 02:28:41PM -0600, Eric Smith wrote:
One process less in the system → faster bootup.
As has been pointed out, it could easily be the systemd journal process that writes to /var/log/messages, so there wouldn't necessarily be one fewer process.
Logs not duplicated → less disk usage.
True. So keep /var/log/messages and make the binary journal optional.
Binary journal gives other features like: - additional metadata (log level, selinux context, uid, gid, capability set (coming soon), cgroups, systemd unit) - slicing and dicing of the data (reverse output, various formats, interleaving of journals from multiple machines, fast filtering by various properties) - fast access to the data, so e.g. 'systemctl status unit' can show data for that unit. With text based /var/log/messages this would be just to slow. Also TAB-completion for filters.
So removing the binary journal has huge downsides. The need for /var/log/messages filters down to wanting to use less or shell built-ins to read the data, which is a valid usecase, but not worth the overhead in 99% of cases.
Zbyszek
On Mon, Jul 15, 2013 at 2:43 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
So removing the binary journal has huge downsides.
I don't dispute that removing it would have a downside. I might dispute that the downside is "huge", but for now I'll let that pass.
The need for /var/log/messages filters down to wanting to use less or shell built-ins to read the data, which is a valid usecase, but not worth the overhead in 99% of cases.
But it's what people actually use in 99.9% of cases. 99.9% of the time I don't need the extra information in the binary journal. Making /var/log/messages unavailable by default has a huge down side.
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default. It's basically the same arguments: there's more information available; it's easier for software to parse; it can be made more reliable; special tools are OK and people don't really need to open it in a text editor. We've seen how well that works on Windows. Blech.
Eric
On Mon, 15.07.13 14:53, Eric Smith (brouhaha@fedoraproject.org) wrote:
The need for /var/log/messages filters down to wanting to use less or shell built-ins to read the data, which is a valid usecase, but not worth the overhead in 99% of cases.
But it's what people actually use in 99.9% of cases. 99.9% of the time I don't need the extra information in the binary journal. Making /var/log/messages unavailable by default has a huge down side.
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default. It's basically the same arguments: there's more information available; it's easier for software to parse; it can be made more reliable; special tools are OK and people don't really need to open it in a text editor. We've seen how well that works on Windows. Blech.
Nobody is proposing this. Slippery slope arguments seldom are particularly convincing...
And it's easy to turn this around: by following your logic we really should get rid of ELF binaries and instead write everything in some scripting language instead, since ELF binaries are, well, binary...
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora.
I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before.
Lennart
On Mon, Jul 15, 2013 at 3:06 PM, Lennart Poettering mzerqung@0pointer.de wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora.
I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before.
I have thought a lot about usecases, and it's exactly because I have to maintain other people's systems that use the default installation, and I routinely have to use more, less, cat, grep, vi, etc. on /var/log/messages that I don't want to see it go away. It's far easier for me to tell someone a grep command on the phone than to also have to tell them to run some other tool and pipe the input into grep, or into a temp file that I can have them look at in an editor. /var/log/messages is there, it works, and it is BETTER than a binary format even if it doesn't contain as much information.
Eric
On Mon, Jul 15, 2013 at 03:12:00PM -0600, Eric Smith wrote:
/var/log/messages that I don't want to see it go away. It's far easier for me to tell someone a grep command on the phone than to also have to tell them to run some other tool and pipe the input into grep,
I think actually the journalctl filtering commands are more phone-friendly than grep. This might fall under "try it -- you might like it!".
On Mon, Jul 15, 2013 at 3:06 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Mon, 15.07.13 14:53, Eric Smith (brouhaha@fedoraproject.org) wrote:
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default. It's basically the same arguments: there's more information available; it's easier for software to parse; it can be made more reliable; special tools are OK and people don't really need to open it in a text editor. We've seen how well that works on Windows. Blech.
Nobody is proposing this.
Nobody is seriously proposing it, yet no one has given much explanation of why binary-only logs are better than text logs that isn't essentially the same as the arguments for the Windows Registry.
Eric
On Mon, 15.07.13 15:14, Eric Smith (brouhaha@fedoraproject.org) wrote:
On Mon, Jul 15, 2013 at 3:06 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Mon, 15.07.13 14:53, Eric Smith (brouhaha@fedoraproject.org) wrote:
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default. It's basically the same arguments: there's more information available; it's easier for software to parse; it can be made more reliable; special tools are OK and people don't really need to open it in a text editor. We've seen how well that works on Windows. Blech.
Nobody is proposing this.
Nobody is seriously proposing it, yet no one has given much explanation of why binary-only logs are better than text logs that isn't essentially the same as the arguments for the Windows Registry.
Well, indexing, structure, sealing are some reasons for using binary journal files. The registry isn't particularly good at any of this. (But really, the comparison is just wrong, since the registry is a configuration store, and not a log store.)
But I mean, honestly, in all fairness, even if the Registry might be an awful mess, there actually are valid reasons for the existance of the registry too, regardless whether they might be convincing or not. Just because something is a reason for having a binary configuration store it doesn't invalidate the reason for having a binary index log store...
And again, nobody is talking about introducing binary configuration stores, that's another straw-man of yours.
Lennart
On Mon, Jul 15, 2013 at 3:21 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(But really, the comparison is just wrong, since the registry is a configuration store, and not a log store.)
It's not a perfect analogy, yet the arguments for both seem very similar, which is why I brought it up. We should try to learn from the mistakes made by other systems, rather than rushing to repeat them.
Microsoft uses binary logs also, and they are really awful. Maybe your binary journal is far better than MS' logs, but all the arguments I've seen for it so far make it seem like the advantages are mainly for complex use cases, and for those someone is going to do a bunch of system configuration work anyhow, so it doesn't make sense for that to be the default. The default should be simple, and the configuration should be done if you want something complex, rather than the other way around.
Eric
On Mon, 15.07.13 15:28, Eric Smith (brouhaha@fedoraproject.org) wrote:
On Mon, Jul 15, 2013 at 3:21 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(But really, the comparison is just wrong, since the registry is a configuration store, and not a log store.)
It's not a perfect analogy, yet the arguments for both seem very similar, which is why I brought it up. We should try to learn from the mistakes made by other systems, rather than rushing to repeat them.
Microsoft uses binary logs also, and they are really awful. Maybe your binary journal is far better than MS' logs, but all the arguments I've seen for it so far make it seem like the advantages are mainly for complex use cases, and for those someone is going to do a bunch of system configuration work anyhow, so it doesn't make sense for that to be the default. The default should be simple, and the configuration should be done if you want something complex, rather than the other way around.
Showing the last 10 log lines for "systemctl status" is not a "complex usecase". Quite frankly, seeing the most recent log output of a service is certainly the most relevant information when you are wondering about a service's state. There is no efficient, correct way how you could implement that on top of /var/log/messages.
journalctl makes things easier, not more complex. Sure, you have to learn a new tool, but the level is low for this one, and you will gain a lot more out of it.
Lennart
On 07/15/2013 11:28 PM, Eric Smith wrote:
Microsoft uses binary logs also, and they are really awful. Maybe your binary journal is far better than MS' logs,
IIRC, the main issue with the Windows system event files is that in order to interpret the binary data, you need DLLs provided by the application which wrote the logs. The journal provides a simple key/value map for each log entry, so it's easy for application developers to provide data in a more structured manner, while still maintaining the self-describing nature of log file entries (or perhaps increasing it because you now can add column labels because you don't feel compelled to avoid long log lines).
On 15.07.2013 23:06, Lennart Poettering wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora. I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before. Lennart
So maybe we (Fedora) should go with XML instead of binary or some dedicated RDMBS for storing system logs? I'm not against binary log format but try to understand why it's superior to text and also why something more sophisticated isn't used if we really need this additional meta data that could not be included in plain text?
Mateusz Marzantowicz
On Mon, 15.07.13 23:38, Mateusz Marzantowicz (mmarzantowicz@osdf.com.pl) wrote:
On 15.07.2013 23:06, Lennart Poettering wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora. I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before. Lennart
So maybe we (Fedora) should go with XML instead of binary or some dedicated RDMBS for storing system logs? I'm not against binary log format but try to understand why it's superior to text and also why something more sophisticated isn't used if we really need this additional meta data that could not be included in plain text?
XML and text files are not sanely indexable. Real database are large packages we cannot pull into minimal systems.
Beyond that, and that's most important: the specific requirements are different from what those systems provide. We want something minimal, C-based, that can work without any service around, that can be mmapped by multiple processes at the same time. Something that indexes implicitly by all keys without any pre-defined schema. Something that primarily append-only (for robustness reasons). Something that is indexed by time, and interleavable. Something that can store binary blobs, that can optionally compress larger blobs. Something that can be rotated. Something that supports Unix access controls natively. Something that provides us with the right complexity guarantees. And more...
If you create a domain-specific database, then you should not do that lightly. But this specific usecase certainly warranted this, and the journal does deliver the requirements we cared for.
Lennart
On Mon, Jul 15, 2013 at 3:46 PM, Lennart Poettering mzerqung@0pointer.de wrote:
But this specific usecase certainly warranted this, and the journal does deliver the requirements we cared for.
I'm sure it does. But I think those requirements mainly apply to servers and enterprise installations, not the average Fedora installation. For the most common use case, doing without /var/log/messages just makes things more complicated for the user and/or sysadmin. I'm all for being able to get fancy binary journals as a configurable option. Or even by default, as long as it doesn't take away also getting a text log by default as well.
Eric
On Mon, Jul 15, 2013 at 11:38:14PM +0200, Mateusz Marzantowicz wrote:
On 15.07.2013 23:06, Lennart Poettering wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora. I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before. Lennart
So maybe we (Fedora) should go with XML instead of binary or some dedicated RDMBS for storing system logs? I'm not against binary log format but try to understand why it's superior to text and also why something more sophisticated isn't used if we really need this additional meta data that could not be included in plain text?
With a binary format you can have an index of field -> offset, hash->offset, etc., and then you can jump to this offset and read data there. With text files, and with XML files, this is not possible, at least not in any sane way. Efficiency would be much worse too.
Zbyszek
On 15.07.2013 23:49, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jul 15, 2013 at 11:38:14PM +0200, Mateusz Marzantowicz wrote:
On 15.07.2013 23:06, Lennart Poettering wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora. I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before. Lennart
So maybe we (Fedora) should go with XML instead of binary or some dedicated RDMBS for storing system logs? I'm not against binary log format but try to understand why it's superior to text and also why something more sophisticated isn't used if we really need this additional meta data that could not be included in plain text?
With a binary format you can have an index of field -> offset, hash->offset, etc., and then you can jump to this offset and read data there. With text files, and with XML files, this is not possible, at least not in any sane way. Efficiency would be much worse too.
OK, XML is also text so my alternative is a bit illusory but .... it could be represented as DOM nodes and could be indexed easier and more efficiently. Please, see how good is XML implementation in PostgreSQL. You can say, it's too much and too complicated for stupid system logs to go with fully blown RDBMS solution, but hey, really?
Mateusz Marzantowicz
On Tue, 16.07.13 00:05, Mateusz Marzantowicz (mmarzantowicz@osdf.com.pl) wrote:
On 15.07.2013 23:49, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Jul 15, 2013 at 11:38:14PM +0200, Mateusz Marzantowicz wrote:
On 15.07.2013 23:06, Lennart Poettering wrote:
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora. I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before. Lennart
So maybe we (Fedora) should go with XML instead of binary or some dedicated RDMBS for storing system logs? I'm not against binary log format but try to understand why it's superior to text and also why something more sophisticated isn't used if we really need this additional meta data that could not be included in plain text?
With a binary format you can have an index of field -> offset, hash->offset, etc., and then you can jump to this offset and read data there. With text files, and with XML files, this is not possible, at least not in any sane way. Efficiency would be much worse too.
OK, XML is also text so my alternative is a bit illusory but .... it could be represented as DOM nodes and could be indexed easier and more efficiently. Please, see how good is XML implementation in PostgreSQL. You can say, it's too much and too complicated for stupid system logs to go with fully blown RDBMS solution, but hey, really?
Yes. Really.
Lennart
On Tue, Jul 16, 2013 at 7:06 AM, Lennart Poettering mzerqung@0pointer.dewrote:
On Mon, 15.07.13 14:53, Eric Smith (brouhaha@fedoraproject.org) wrote:
The need for /var/log/messages filters down to wanting to use less or shell built-ins to read the data, which is a valid usecase, but not worth the overhead in 99% of cases.
But it's what people actually use in 99.9% of cases. 99.9% of the time I don't need the extra information in the binary journal. Making /var/log/messages unavailable by default has a huge down side.
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default. It's basically the same arguments: there's more information available; it's easier for software to parse; it can be made more reliable; special tools are OK and people don't really need to open it in a text editor. We've seen how well that works on Windows. Blech.
Nobody is proposing this. Slippery slope arguments seldom are particularly convincing...
Slippery slope arguments is what will make Fedora stop being Linux and become a bunch of utilities to parse some binary format files on top of a Linux kernel.
And it's easy to turn this around: by following your logic we really should get rid of ELF binaries and instead write everything in some scripting language instead, since ELF binaries are, well, binary...
That's pretty silly. But as much as I'm opposed to journalctl, I would have been opposed to changing anything else which is Bash based to something that's ELF, unless there is a good argument for that (performance... or?). Compiled stuff adds many times unneeded complexity.
It's a matter of finding the right balance: i.e. what can be text files, and where we have to win more by making it binary. I am pretty sure this is a case where we win more by sticking to binary files. It's totally fine if you disagree on this, but I'd still like to ask you to think about whether your specific usecase and specific requirements are strong enough to (continue to) be the default for Fedora, instead of just being your local configuration of Fedora.
The right balance is definitely not having log files as a binary format. We don't need use cases for log files. Log files should be plain text, plain simple, portable, accessible. journalctl is doing the exact opposite and not providing much benefit. Honestly Lennart, this is something that will make people start hating Fedora. Especially people who deal with crashes, recovery and support.
I mean, you should never forget that on your own machines everything will stay as is: you will install syslog, and things will be exactly as before.
Lennart
-- Lennart Poettering - Red Hat, Inc. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Mon, Jul 15, 2013 at 02:53:06PM -0600, Eric Smith wrote:
On Mon, Jul 15, 2013 at 2:43 PM, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
So removing the binary journal has huge downsides.
I don't dispute that removing it would have a downside. I might dispute that the downside is "huge", but for now I'll let that pass.
This means that you haven't really used journalctl. It *is* much nicer. Try it, and you'll stop wanting to go back to 'less /var/log/messages' and 'tail -f /var/log/*'.
The need for /var/log/messages filters down to wanting to use less or shell built-ins to read the data, which is a valid usecase, but not worth the overhead in 99% of cases.
But it's what people actually use in 99.9% of cases. 99.9% of the time I don't need the extra information in the binary journal. Making /var/log/messages unavailable by default has a huge down side.
In one of the messages earlier in the thread (Message-ID: 20130715155534.GC4989@tango.0pointer.de) Lennart posted a list of "translations" to get the same output from journalctl... I think most use cases were covered there. I'll add one more:
"dmesg" -> "journalctl -k", except that the output is colored and doesn't wrap around.
Sure, you can't use 'for ... read' to read the journal, but as it was already pointed out, you're more likely to do that on a server, where installing rsyslog can be one of the customizations if you think you'll need it.
If we go to having only binary logs by default, maybe we should also go to having only binary configuration files by default.
[snip non-technical part]
Zbyszek
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
This means that you haven't really used journalctl. It *is* much nicer. Try it, and you'll stop wanting to go back to 'less /var/log/messages' and 'tail -f /var/log/*'.
So I went and tried journalctl, and immediately hit the same UI stupidity as systemctl. Truncated lines, auto-paging, etc., unless I pipe to something else. Significantly differing behavior between direct output and pipes is just wrong. Having to remember some double-dash long option just to get non-truncated (and non-useless) log info is highly irritating.
On Mon, 15.07.13 16:22, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
This means that you haven't really used journalctl. It *is* much nicer. Try it, and you'll stop wanting to go back to 'less /var/log/messages' and 'tail -f /var/log/*'.
So I went and tried journalctl, and immediately hit the same UI stupidity as systemctl. Truncated lines, auto-paging, etc., unless I pipe to something else. Significantly differing behavior between direct output and pipes is just wrong. Having to remember some double-dash long option just to get non-truncated (and non-useless) log info is highly irritating.
You can set SYSTEMD_PAGER to the empty string to turn off auto-paging in all systemd tools. There's a bug against bash that requests adding a (commented) line to .bash_profile, so that the auto-pager haters can easily disable this behaviour, just by uncommented that line.
But this is a really different discussion anyway. I know some people hate auto-paging, but I am pretty sure more people learned to love it since it was introduced by git. I love auto-paging for sure.
Lennart
(also: journalctl doesn't truncate lines when doing auto-paging)
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
(also: journalctl doesn't truncate lines when doing auto-paging)
It most certainly does, at least on an up-to-date F18 system. The truncation behavior is even DIFFERENT between "journalctl" and "journalctl -f" modes!
On Mon, 15.07.13 16:31, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
(also: journalctl doesn't truncate lines when doing auto-paging)
It most certainly does, at least on an up-to-date F18 system. The truncation behavior is even DIFFERENT between "journalctl" and "journalctl -f" modes!
Well, "journalctl -f" dosn't do auto-paging. "journalctl" (without -f) does. So here you go.
Lennart
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:31, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
(also: journalctl doesn't truncate lines when doing auto-paging)
It most certainly does, at least on an up-to-date F18 system. The truncation behavior is even DIFFERENT between "journalctl" and "journalctl -f" modes!
Well, "journalctl -f" dosn't do auto-paging. "journalctl" (without -f) does. So here you go.
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
On Mon, 15.07.13 16:38, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:31, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
(also: journalctl doesn't truncate lines when doing auto-paging)
It most certainly does, at least on an up-to-date F18 system. The truncation behavior is even DIFFERENT between "journalctl" and "journalctl -f" modes!
Well, "journalctl -f" dosn't do auto-paging. "journalctl" (without -f) does. So here you go.
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
You are aware that you can scroll to the right in "less"? Just press the arrow key to the left.
Lennart
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:38, Chris Adams (linux@cmadams.net) wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
You are aware that you can scroll to the right in "less"? Just press the arrow key to the left.
I assume you mean "arrow key to the right", but that doesn't work when I run "journalctl". I get "No next file" (and "No previous file" for left-arrow).
On Mon, 15.07.13 16:42, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:38, Chris Adams (linux@cmadams.net) wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
You are aware that you can scroll to the right in "less"? Just press the arrow key to the left.
I assume you mean "arrow key to the right", but that doesn't work when I run "journalctl". I get "No next file" (and "No previous file" for left-arrow).
Yes, to the right. You are using less as a pager I presume? Scrolling to the right certainly works here. Please file a bug against less if it doesn't for you.
Lennart
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:42, Chris Adams (linux@cmadams.net) wrote:
I assume you mean "arrow key to the right", but that doesn't work when I run "journalctl". I get "No next file" (and "No previous file" for left-arrow).
Yes, to the right. You are using less as a pager I presume? Scrolling to the right certainly works here. Please file a bug against less if it doesn't for you.
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
On Mon, Jul 15, 2013 at 05:43:43PM -0500, Chris Adams wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:42, Chris Adams (linux@cmadams.net) wrote:
I assume you mean "arrow key to the right", but that doesn't work when I run "journalctl". I get "No next file" (and "No previous file" for left-arrow).
Yes, to the right. You are using less as a pager I presume? Scrolling to the right certainly works here. Please file a bug against less if it doesn't for you.
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
[At least in default configuration] you can switch between "truncated" and "non-truncated" output by pressing -S<ENTER>, or view the "truncated" part by pressing right arrow. If you're seeing something different, then please file a bug (against less, because journalctl doesn't really do anything other than popen("less") here). For reference, try pressing 'h' in less, I see: ESC-) RightArrow * Left one half screen width (or N positions). ESC-( LeftArrow * Right one half screen width (or N positions). If you have something different, then it probably is a config difference.
Zbyszek
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
On Mon, Jul 15, 2013 at 05:43:43PM -0500, Chris Adams wrote:
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
[At least in default configuration] you can switch between "truncated" and "non-truncated" output by pressing -S<ENTER>, or view the "truncated" part by pressing right arrow. If you're seeing something different, then please file a bug (against less, because journalctl doesn't really do anything other than popen("less") here). For reference, try pressing 'h' in less, I see:
Actually, journalctl DOES do something other than popen("less"); it overrides the user's $LESS and sets it to "FRSXMK" (the S option tells less to chop lines). So while technically journalctl is not chopping lines, it is forcing less to do so. A user's $LESS should not be overridden, especially to chop log lines.
Another thing that I don't see in the man page is why some lines are bold/in color.
On Mon, Jul 15, 2013 at 06:39:59PM -0500, Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
On Mon, Jul 15, 2013 at 05:43:43PM -0500, Chris Adams wrote:
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
[At least in default configuration] you can switch between "truncated" and "non-truncated" output by pressing -S<ENTER>, or view the "truncated" part by pressing right arrow. If you're seeing something different, then please file a bug (against less, because journalctl doesn't really do anything other than popen("less") here). For reference, try pressing 'h' in less, I see:
Actually, journalctl DOES do something other than popen("less"); it overrides the user's $LESS and sets it to "FRSXMK" (the S option tells less to chop lines). So while technically journalctl is not chopping lines, it is forcing less to do so. A user's $LESS should not be overridden, especially to chop log lines.
Sure, those are the defaults. If you had written that you don't like the systemd defaults, instead of talking about "bugs", this whole conversation would have been much productive.
The default was picked as is, after some back-and-forth, because this way it much easier to browse through a long listing. You can always change *your* default by sticking export SYSTEMD_PAGER='less -FRX -+S' in your .bashrc or wherever.
Another thing that I don't see in the man page is why some lines are bold/in color.
error -> red, notice -> bold, etc. It's a feature you don't get traditionally because syslog drops the priority information from the on-disk format.
Zbyszek
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Sure, those are the defaults. If you had written that you don't like the systemd defaults, instead of talking about "bugs", this whole conversation would have been much productive.
When I described the behavior, I was told I was wrong and that the lines weren't chopped (which I then wondered why there was a "--full" option). Neither the documentation or the emails mentioned that journalctl overrides $LESS with the option to chop lines; I only found that out by tracing the process.
Another thing that I don't see in the man page is why some lines are bold/in color.
error -> red, notice -> bold, etc.
Which Lennart said should be documented in the man page, which is now on the to-do list.
It's a feature you don't get traditionally because syslog drops the priority information from the on-disk format.
I'd expect that if somebody thought that was an important default, the log format would have been updated years ago when rsyslog became the default.
On Mon, Jul 15, 2013 at 8:26 PM, Chris Adams linux@cmadams.net wrote:
It's a feature you don't get traditionally because syslog drops the priority information from the on-disk format.
I'd expect that if somebody thought that was an important default, the log format would have been updated years ago when rsyslog became the default.
It's damn useful.
There are tradeoffs, but you have take in the good aspects.
m -- martin.langhoff@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
On Mon, Jul 15, 2013 at 07:26:15PM -0500, Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Sure, those are the defaults. If you had written that you don't like the systemd defaults, instead of talking about "bugs", this whole conversation would have been much productive.
When I described the behavior, I was told I was wrong and that the lines weren't chopped (which I then wondered why there was a "--full" option). Neither the documentation or the emails mentioned that journalctl overrides $LESS with the option to chop lines; I only found that out by tracing the process.
The UI is to a large degree a child of the git UI, I have to admit I find it self-explanatory. But I can see how one wouldn't think of pressing the right arrow, if one was convinced that jouranlctl has truncated the lines. An explanatory paragraph has been added to journalctl(1).
Another thing that I don't see in the man page is why some lines are bold/in color.
error -> red, notice -> bold, etc.
Which Lennart said should be documented in the man page, which is now on the to-do list.
Also done. Both changes are online [1], and should be available in the next Fedora rawhide package.
It's a feature you don't get traditionally because syslog drops the priority information from the on-disk format.
I'd expect that if somebody thought that was an important default, the log format would have been updated years ago when rsyslog became the default.
Not without (a) breaking scripts, (b) annoying people, because precious screen estate would be wasted on markers for the log level.
Zbyszek
On Mon, Jul 15, 2013 at 06:39:59PM -0500, Chris Adams wrote:
Actually, journalctl DOES do something other than popen("less"); it overrides the user's $LESS and sets it to "FRSXMK" (the S option tells less to chop lines). So while technically journalctl is not chopping lines, it is forcing less to do so. A user's $LESS should not be overridden, especially to chop log lines.
Although the less man page says that lines are "chopped", they actually _should_ be available for scrolling to the right and back to the left. That is, the lines are not truncated.
I had an RFE about this in bugzilla somewhere. It's hard because some options which make sense for a general pager don't for journal paging, so it's better to set a sane set of options. You can set the systemd-specific pager variable if you think it should be different. (You can set that to `cat` and then presto, no paging.)
Another thing that I don't see in the man page is why some lines are bold/in color.
That's a documentation bug. It's not in the man page but explained here http://0pointer.de/blog/projects/journalctl.html
* Lines of error priority (and higher) will be highlighted red. * Lines of notice/warning priority will be highlighted bold.
On Mon, 15.07.13 18:39, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
On Mon, Jul 15, 2013 at 05:43:43PM -0500, Chris Adams wrote:
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
[At least in default configuration] you can switch between "truncated" and "non-truncated" output by pressing -S<ENTER>, or view the "truncated" part by pressing right arrow. If you're seeing something different, then please file a bug (against less, because journalctl doesn't really do anything other than popen("less") here). For reference, try pressing 'h' in less, I see:
Actually, journalctl DOES do something other than popen("less"); it overrides the user's $LESS and sets it to "FRSXMK" (the S option tells less to chop lines). So while technically journalctl is not chopping lines, it is forcing less to do so. A user's $LESS should not be overridden, especially to chop log lines.
Well, they arent't "chopped". You can scroll to the right and you can see them.
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
Another thing that I don't see in the man page is why some lines are bold/in color.
Log priorities. This is indeed missing in the man pages. Added to the TODO list now, to fix.
Lennart
On Tue, Jul 16, 2013 at 01:56:44AM +0200, Lennart Poettering wrote:
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
How do you turn this off? Overriding user configuration can be extremely annoying to users, but if there's an easy well-documented way to turn off overriding, it's easier to deal with.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Tue, 16.07.13 12:49, Andrew McNabb (amcnabb@mcnabbs.org) wrote:
On Tue, Jul 16, 2013 at 01:56:44AM +0200, Lennart Poettering wrote:
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
How do you turn this off? Overriding user configuration can be extremely annoying to users, but if there's an easy well-documented way to turn off overriding, it's easier to deal with.
Set $SYSTEMD_PAGE to the pager of your choice along with any command line arguments you like.
Lennart
On Tue, 16 Jul 2013 20:09:19 +0200 Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 16.07.13 12:49, Andrew McNabb (amcnabb@mcnabbs.org) wrote:
On Tue, Jul 16, 2013 at 01:56:44AM +0200, Lennart Poettering wrote:
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
How do you turn this off? Overriding user configuration can be extremely annoying to users, but if there's an easy well-documented way to turn off overriding, it's easier to deal with.
Set $SYSTEMD_PAGE to the pager of your choice along with any command line arguments you like.
This is a bit anoying for a lot of uses as permissions only allow root (or the 'adm' group), so many people would use sudo with their journal commands.
BTW, should we change 'adm' to 'wheel' to match our other admin group?
kevin
On Tue, 16.07.13 12:19, Kevin Fenzi (kevin@scrye.com) wrote:
On Tue, 16 Jul 2013 20:09:19 +0200 Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 16.07.13 12:49, Andrew McNabb (amcnabb@mcnabbs.org) wrote:
On Tue, Jul 16, 2013 at 01:56:44AM +0200, Lennart Poettering wrote:
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
How do you turn this off? Overriding user configuration can be extremely annoying to users, but if there's an easy well-documented way to turn off overriding, it's easier to deal with.
Set $SYSTEMD_PAGE to the pager of your choice along with any command line arguments you like.
This is a bit anoying for a lot of uses as permissions only allow root (or the 'adm' group), so many people would use sudo with their journal commands.
BTW, should we change 'adm' to 'wheel' to match our other admin group?
The files are owned by the group "systemd-journal" now. If you want to grant a user access to the system journals and nothing else, add him/her to this group.
Via file system ACLs the groups "adm" and "wheel" also get read access to the journal files, "adm" being more a group of "folks who can see but not do", and "wheel" being a group of folks "who can see and do". Of course, wheel and adm might enable you to see and do much more than just granting access to the system journals.
Lennart
On Tue, Jul 16, 2013 at 08:09:19PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 12:49, Andrew McNabb (amcnabb@mcnabbs.org) wrote:
On Tue, Jul 16, 2013 at 01:56:44AM +0200, Lennart Poettering wrote:
We didn't override LESS initially, but we got bugs about that, and after a while of forth and back we followed again what git does here and override it. You find the discussions in bugzilla.
How do you turn this off? Overriding user configuration can be extremely annoying to users, but if there's an easy well-documented way to turn off overriding, it's easier to deal with.
Set $SYSTEMD_PAGE to the pager of your choice along with any command line arguments you like.
The way git does it is to only set the LESS environment variable if it's not set. If the LESS environment variable is set, then git doesn't touch it. This seems like a nice balanced way to approach the situation, and it would be wonderful if systemd/journalctl did the same thing as git.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Tue, Jul 16, 2013 at 01:31:53PM -0500, Andrew McNabb wrote:
The way git does it is to only set the LESS environment variable if it's not set. If the LESS environment variable is set, then git doesn't touch it. This seems like a nice balanced way to approach the situation, and it would be wonderful if systemd/journalctl did the same thing as git.
That's actually what it did originally.
This doesn't work, because the colored output of journalctl needs -R, and several other options are important. I have my LESS variable normally set to -MX, which is great for most things but doesn't work right for this special case. I think the current approach is probably the best compromise.
Hi
On Mon, Jul 15, 2013 at 6:43 PM, Chris Adams wrote:
Once upon a time, Lennart Poettering said:
On Mon, 15.07.13 16:42, Chris Adams wrote:
I assume you mean "arrow key to the right", but that doesn't work when
I
run "journalctl". I get "No next file" (and "No previous file" for left-arrow).
Yes, to the right. You are using less as a pager I presume? Scrolling to the right certainly works here. Please file a bug against less if it doesn't for you.
No, the "bug" is journalctl, not less. If I run "journalctl | less", I get wrapped lines. If I run "journalctl", I get truncated lines (despite your repeated claims to the contrary).
Just file the bug report and it can be reassigned if necessary. No need to argue about it
Rahul
On Mon, Jul 15, 2013 at 5:38 PM, Chris Adams linux@cmadams.net wrote:
And, despite your statement to the contrary, "journalctl" (without -f)
Hey Chris!
You might be hitting a bug, have a surprising pager envvar or something. My general experience is that it does page things. I don't think there's a conspiracy against your pager.
cheers,
m -- martin.langhoff@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
Once upon a time, Martin Langhoff martin.langhoff@gmail.com said:
You might be hitting a bug, have a surprising pager envvar or something. My general experience is that it does page things. I don't think there's a conspiracy against your pager.
I do set LESS and PAGER, but I just tried without either set, and I get the same behavior.
On Mon, Jul 15, 2013 at 04:38:29PM -0500, Chris Adams wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
Ooh. Yeah, journalctl -f shouldn't do that. That makes it a lot less useful.
On Mon, Jul 15, 2013 at 05:41:38PM -0400, Matthew Miller wrote:
On Mon, Jul 15, 2013 at 04:38:29PM -0500, Chris Adams wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
Ooh. Yeah, journalctl -f shouldn't do that. That makes it a lot less useful.
https://bugzilla.redhat.com/show_bug.cgi?id=984758
On Mon, Jul 15, 2013 at 05:41:38PM -0400, Matthew Miller wrote:
On Mon, Jul 15, 2013 at 04:38:29PM -0500, Chris Adams wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
Ooh. Yeah, journalctl -f shouldn't do that. That makes it a lot less useful.
If I'm following the logs with "journalctl -f", I basically only see the time, hostname, and process name/id. Pretty much everything else is truncated. If I actually need to see the messages, is the Right Way to do this "journalctl -f |cat"?
I wouldn't mind having a "-t" option to truncate for the rare situations where I don't actually want to see log messages, but truncating by default is frustrating, particularly when resizing the window doesn't untruncate previously printed lines.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Wed, Jul 17, 2013 at 01:12:17PM -0500, Andrew McNabb wrote:
On Mon, Jul 15, 2013 at 05:41:38PM -0400, Matthew Miller wrote:
On Mon, Jul 15, 2013 at 04:38:29PM -0500, Chris Adams wrote:
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
Ooh. Yeah, journalctl -f shouldn't do that. That makes it a lot less useful.
If I'm following the logs with "journalctl -f", I basically only see the time, hostname, and process name/id. Pretty much everything else is truncated. If I actually need to see the messages, is the Right Way to do this "journalctl -f |cat"?
journalctl -fl in sufficiently new versions.
Zbyszek
On Wed, Jul 17, 2013 at 08:14:40PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
journalctl -fl in sufficiently new versions.
So that will be in Fedora 20, right? It doesn't seem to work in Fedora 19.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Wed, Jul 17, 2013 at 01:29:45PM -0500, Andrew McNabb wrote:
On Wed, Jul 17, 2013 at 08:14:40PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
journalctl -fl in sufficiently new versions.
So that will be in Fedora 20, right? It doesn't seem to work in Fedora 19.
It's just an alias for --full, which has been there for a while. Anyway, I'm pretty sure the patch can be trivially backported.
Zbyszek
Did you try the right arrow key when running "journalctl"?
----- Original Message -----
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Mon, 15.07.13 16:31, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
(also: journalctl doesn't truncate lines when doing auto-paging)
It most certainly does, at least on an up-to-date F18 system. The truncation behavior is even DIFFERENT between "journalctl" and "journalctl -f" modes!
Well, "journalctl -f" dosn't do auto-paging. "journalctl" (without -f) does. So here you go.
And, despite your statement to the contrary, "journalctl" (without -f) does truncate long lines. The difference is that "journalctl" just chops them off, while "journalctl -f" does the nutty "chop characters columns-4 to linelength-1 and replace them with dots" bit.
-- Chris Adams linux@cmadams.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Mon, Jul 15, 2013 at 11:28:48PM +0200, Lennart Poettering wrote:
But this is a really different discussion anyway. I know some people hate auto-paging, but I am pretty sure more people learned to love it since it was introduced by git. I love auto-paging for sure.
The difference between git's autopaging and journalctl is that with git log the most recent commits come first while log files have the most likely interesting stuff at the end. There's "journalctl -e" and "journalctl -r" (as well as the more specific query options, of course) so it should just take some time to get used to it.
On Mon, Jul 15, 2013 at 11:28:48PM +0200, Lennart Poettering wrote:
(also: journalctl doesn't truncate lines when doing auto-paging)
Could sane auto-paging be backported to systemctl? Things like this can make all the difference between a lovable and an unlovable tool.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Tue, 16.07.13 12:43, Andrew McNabb (amcnabb@mcnabbs.org) wrote:
On Mon, Jul 15, 2013 at 11:28:48PM +0200, Lennart Poettering wrote:
(also: journalctl doesn't truncate lines when doing auto-paging)
Could sane auto-paging be backported to systemctl? Things like this can make all the difference between a lovable and an unlovable tool.
systemctl uses pretty much the same code as journalctl these days in this regard.
Lennart
On Tue, Jul 16, 2013 at 12:43:31PM -0500, Andrew McNabb wrote:
On Mon, Jul 15, 2013 at 11:28:48PM +0200, Lennart Poettering wrote:
(also: journalctl doesn't truncate lines when doing auto-paging)
Could sane auto-paging be backported to systemctl? Things like this can make all the difference between a lovable and an unlovable tool.
Could you be more specific as to what you would like to see changed? systemctl uses the the pager in long outputs like list-units, and it is the same pager that journalctl uses.
Zbyszek
On Tue, Jul 16, 2013 at 08:09:27PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
Could you be more specific as to what you would like to see changed? systemctl uses the the pager in long outputs like list-units, and it is the same pager that journalctl uses.
On Fedora 18, I see lines truncated in the middle with "..." and it drives me crazy because there's no way to scroll to get the missing information. For example, just running "systemctl" with no arguments gives insanely truncated output. The other emails in this thread implied that journalctl doesn't do this truncation, which sounds like a huge improvement.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
Am 16.07.2013 20:37, schrieb Andrew McNabb:
On Tue, Jul 16, 2013 at 08:09:27PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
Could you be more specific as to what you would like to see changed? systemctl uses the the pager in long outputs like list-units, and it is the same pager that journalctl uses.
On Fedora 18, I see lines truncated in the middle with "..." and it drives me crazy because there's no way to scroll to get the missing information. For example, just running "systemctl" with no arguments gives insanely truncated output. The other emails in this thread implied that journalctl doesn't do this truncation, which sounds like a huge improvement
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature. These auto-pagers get out of the way immediately if you need a pipeline, so what's the harm? In fact, you can still "journalctl | less" or the like if you really want to.
-- John Florian
Am 16.07.2013 21:45, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature. These auto-pagers get out of the way immediately if you need a pipeline, so what's the harm? In fact, you can still "journalctl | less" or the like if you really want to.
you could also do alias journalctl="journalctl | less" to achive the same
hence my konsole has scrollbars, i do not like autopaging and whatever is not pure unix because there are mechs to achieve whatever you need and with GIt and systemctl/journalctl whe have a inconsistent behavior
in any case *truncate* outputs is a absolutely no-go
the ordinary user does not look at all this things and the advanced which have a reson to look get stripped informations
On Tue, 2013-07-16 at 21:50 +0200, Reindl Harald wrote:
Am 16.07.2013 21:45, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature. These auto-pagers get out of the way immediately if you need a pipeline, so what's the harm? In fact, you can still "journalctl | less" or the like if you really want to.
you could also do alias journalctl="journalctl | less" to achive the same
hence my konsole has scrollbars, i do not like autopaging and whatever is not pure unix because there are mechs to achieve whatever you need and with GIt and systemctl/journalctl whe have a inconsistent behavior
in any case *truncate* outputs is a absolutely no-go
the ordinary user does not look at all this things and the advanced which have a reson to look get stripped informations
alias journalctl='journalctl --no-pager'
an live happy
Simo.
Am 16.07.2013 22:02, schrieb Simo Sorce:
On Tue, 2013-07-16 at 21:50 +0200, Reindl Harald wrote:
Am 16.07.2013 21:45, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature. These auto-pagers get out of the way immediately if you need a pipeline, so what's the harm? In fact, you can still "journalctl | less" or the like if you really want to.
you could also do alias journalctl="journalctl | less" to achive the same
hence my konsole has scrollbars, i do not like autopaging and whatever is not pure unix because there are mechs to achieve whatever you need and with GIt and systemctl/journalctl whe have a inconsistent behavior
in any case *truncate* outputs is a absolutely no-go
the ordinary user does not look at all this things and the advanced which have a reson to look get stripped informations
alias journalctl='journalctl --no-pager' an live happy
and now explain me why i should need to set aliases for random commands to achieve the well known default which has any over decades known unix tool?
* if you want a non-standard behavior set a alias * if you want the standard behavior do nothing
what here happens is make the exception to a standard
for a consistent system behavior you would need to patch ls, cat, dir, whatever and *then* explain people this is the standard and they need aliases for all of them - the wrong direction my friend
From: h.reindl@thelounge.net
Am 16.07.2013 21:45, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the
auto-pager feature of both git and journalctl. I
love it, in fact. I'm no stranger to very long pipelines and sub-
shells but I see nothing but benefit in not
having to add "| less" routinely to things that are UI in nature.
These auto-pagers get out of the way immediately
if you need a pipeline, so what's the harm? In fact, you can
still "journalctl | less" or the like if you really
want to.
you could also do alias journalctl="journalctl | less" to achive the same
Of course I could ... just as easily as you can set the systemd/git pager env variables, or aliases to pipe thru cat.
hence my konsole has scrollbars, i do not like autopaging and whatever is not pure unix
Now see, I'd argue *pure Unix* won't have scroll bars. However, I do the same for commands that don't have an auto-pager, but I find it mildly annoying to have to reach for the mouse or wish that konsole had some feature that my pager does.
One could argue that git is not pure cvs too, but they don't because it brings a fresh take on an old problem. The authors thought they had a better approach. Many agree. Obviously some do not, but that's just a fact of life. I simply choose to adapt and enjoy the ride.
in any case *truncate* outputs is a absolutely no-go
I'm not fond of the truncation method either, but I do appreciate knowing that I'm not seeing the whole line, nor do I much care for wrapped lines with semi-structured output. So until my pager grows a throbbing indicator per line to hint there's more I find it a reasonable approach.
-- John Florian
Am 16.07.2013 22:10, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
Am 16.07.2013 21:45, schrieb John.Florian@dart.biz:
From: h.reindl@thelounge.net
i am *strictly* against all this truncate and autopaging and for me "GIT does the same" is no argument - a mistake is not better because others do the same.............
if i want paging i do " | less" or " | more" *this* is the unix way of work
but who am i...............
While I'm no more important than the next guy, I'll defend the
auto-pager feature of both git and journalctl. I
love it, in fact. I'm no stranger to very long pipelines and sub-
shells but I see nothing but benefit in not
having to add "| less" routinely to things that are UI in nature.
These auto-pagers get out of the way immediately
if you need a pipeline, so what's the harm? In fact, you can
still "journalctl | less" or the like if you really
want to.
you could also do alias journalctl="journalctl | less" to achive the same
Of course I could ... just as easily as you can set the systemd/git pager env variables, or aliases to pipe thru cat
and tomorrow a, b and c comes also with autopaging/truncate and the next day d only with trunacte but no autopaging and you have to read manpages for any random command to look how behave anything you type in a terminal the same way
shiny new world.......................
On Tue, 16.07.13 21:50, Reindl Harald (h.reindl@thelounge.net) wrote:
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature. These auto-pagers get out of the way immediately if you need a pipeline, so what's the harm? In fact, you can still "journalctl | less" or the like if you really want to.
you could also do alias journalctl="journalctl | less" to achive the same
No you can't really. The auto-paging is smarter than you might think. For example, "journalctl -e" can only really work if we spawn the pager ourselves, and there's more like that.
But anyway, the auto-paging thing is going to stay, you can talk about it as much as you want. Why? Simply because *I* love it. It's one awesome feature. You are welcome to disagree, but discussing this forth and back on fedora-devel is highly unlikely to change my mind on this. As long as I maintain it, this one feature definitely stays in. Sorry for that!
Lennart
Am 16.07.2013 22:10, schrieb Lennart Poettering:
No you can't really. The auto-paging is smarter than you might think. For example, "journalctl -e" can only really work if we spawn the pager ourselves, and there's more like that.
But anyway, the auto-paging thing is going to stay, you can talk about it as much as you want. Why? Simply because *I* love it. It's one awesome feature. You are welcome to disagree, but discussing this forth and back on fedora-devel is highly unlikely to change my mind on this. As long as I maintain it, this one feature definitely stays in. Sorry for that!
and if everybody starts to introduce random behavior instead respect well known standards becaus he loves it we end in a completly mess because *everything* behaves different
Once upon a time, John.Florian@dart.biz John.Florian@dart.biz said:
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature.
My primary objection is that I don't like things that have differing behavior between a TTY and a pipe. One problem is if you are writing a script to process output from something, you'll probably run the command in a TTY to check the output, but you'll get different results.
There are not many programs that have such TTY/pipe behavior. The only ones that come to mind are "ps" (where output is truncated at screen width) and "ls" (where "--color=tty" changes between TTY and pipe). The ls behavior makes sense (and doesn't change the information displayed, formatting, etc.); the "ps" behavior is annoying. I guess "man" has similar auto-pager behavior to systemctl/journalctl (although IMHO man is a different type of thing, since it is a documentation reader).
We already have pipes, and anyone that knows how to use a Unix shell knows how to use them. IMHO it is extra complication (and code duplication) for programs to change behavior between a TTY and a pipe, unless there are specific things that can't easily be accomplished with a simple pipe (such as "ls --color=tty"). Pagination, truncation, etc. are easy using pipes, and should not be done in regular programs.
On Tue, Jul 16, 2013 at 2:06 PM, Chris Adams linux@cmadams.net wrote:
I don't like things that have differing behavior between a TTY and a pipe.
I don't either, but that battle was lost many years ago with the 'ls' command, which defaults to columnar output to a tty but not to a pipe.
Eric
On Tue, 16.07.13 15:06, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, John.Florian@dart.biz John.Florian@dart.biz said:
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature.
My primary objection is that I don't like things that have differing behavior between a TTY and a pipe. One problem is if you are writing a script to process output from something, you'll probably run the command in a TTY to check the output, but you'll get different results.
There are not many programs that have such TTY/pipe behavior. The only ones that come to mind are "ps" (where output is truncated at screen width) and "ls" (where "--color=tty" changes between TTY and pipe). The ls behavior makes sense (and doesn't change the information displayed, formatting, etc.); the "ps" behavior is annoying. I guess "man" has similar auto-pager behavior to systemctl/journalctl (although IMHO man is a different type of thing, since it is a documentation reader).
ls, ps, man, git, wget, gcc, grep, pstree, bash, lsblk, lslocks, ...
And this is where I got bored and stopped looking.
We already have pipes, and anyone that knows how to use a Unix shell knows how to use them. IMHO it is extra complication (and code duplication) for programs to change behavior between a TTY and a pipe, unless there are specific things that can't easily be accomplished with a simple pipe (such as "ls --color=tty"). Pagination, truncation, etc. are easy using pipes, and should not be done in regular programs.
But anyway, this subthread is pointless. You won't convince me that auto-paging is awful, you can just drop the thread entirely, it's pointless.
Lennart
On Tue, 2013-07-16 at 15:06 -0500, Chris Adams wrote:
Once upon a time, John.Florian@dart.biz John.Florian@dart.biz said:
While I'm no more important than the next guy, I'll defend the auto-pager feature of both git and journalctl. I love it, in fact. I'm no stranger to very long pipelines and sub-shells but I see nothing but benefit in not having to add "| less" routinely to things that are UI in nature.
My primary objection is that I don't like things that have differing behavior between a TTY and a pipe. One problem is if you are writing a script to process output from something, you'll probably run the command in a TTY to check the output, but you'll get different results.
There are not many programs that have such TTY/pipe behavior. The only ones that come to mind are "ps" (where output is truncated at screen width)
Indeed, I've always thought it would be wonderful if "ps" could auto-page its output when run on TTY if the output is larger/longer than a screen.
It's just plain annoying to run a command, realize that the one piece of information you wanted is truncated out of the screen, and then have to run it second time with " | less" added.
On Tue, Jul 16, 2013 at 01:37:14PM -0500, Andrew McNabb wrote:
On Tue, Jul 16, 2013 at 08:09:27PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
Could you be more specific as to what you would like to see changed? systemctl uses the the pager in long outputs like list-units, and it is the same pager that journalctl uses.
On Fedora 18, I see lines truncated in the middle with "..." and it drives me crazy because there's no way to scroll to get the missing information. For example, just running "systemctl" with no arguments gives insanely truncated output. The other emails in this thread implied that journalctl doesn't do this truncation, which sounds like a huge improvement.
Yes, this is still the case, that plain 'systemctl' truncates unit names. It's a tough choice, but in this specific case it's hard to find a format that would work without truncation. There are two fields with potentially very long values (unit name and unit description). We can let the second one flow in the right side, but if we left the first one in full width, on normal terminal width of ~100 columns, even this *first* column would not fit. And the truth is that the units which are usually truncated are the device units (they tend to have the longest names), and they are not the most interesting usually. So in this case truncation makes it much nicer to look at the interesting .service and .target units. Truncation has been removed from a few other places, but in this case it just doesn't seem feasible.
Zbyszek
On Tue, Jul 16, 2013 at 10:09:19PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
Yes, this is still the case, that plain 'systemctl' truncates unit names. It's a tough choice, but in this specific case it's hard to find a format that would work without truncation. There are two fields with potentially very long values (unit name and unit description). We can let the second one flow in the right side, but if we left the first one in full width, on normal terminal width of ~100 columns, even this *first* column would not fit. And the truth is that the units which are usually truncated are the device units (they tend to have the longest names), and they are not the most interesting usually. So in this case truncation makes it much nicer to look at the interesting .service and .target units. Truncation has been removed from a few other places, but in this case it just doesn't seem feasible.
The "..." in the middle of the line makes the output completely unusable. On my current machine, I see 6 service units whose names are middle-truncated (it's not just device units). The problem is that there's no way to scroll or adjust the terminal size to get the full output. I would much rather have the description field completely omitted or at least extend off to the right (it's much less essential than the service name anyway).
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Tue, Jul 16, 2013 at 07:25:10PM -0500, Andrew McNabb wrote:
On Tue, Jul 16, 2013 at 10:09:19PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
Yes, this is still the case, that plain 'systemctl' truncates unit names. It's a tough choice, but in this specific case it's hard to find a format that would work without truncation. There are two fields with potentially very long values (unit name and unit description). We can let the second one flow in the right side, but if we left the first one in full width, on normal terminal width of ~100 columns, even this *first* column would not fit. And the truth is that the units which are usually truncated are the device units (they tend to have the longest names), and they are not the most interesting usually. So in this case truncation makes it much nicer to look at the interesting .service and .target units. Truncation has been removed from a few other places, but in this case it just doesn't seem feasible.
The "..." in the middle of the line makes the output completely unusable.
That might be a bit of an exagerration, no?
On my current machine, I see 6 service units whose names are middle-truncated (it's not just device units). The problem is that there's no way to scroll or adjust the terminal size to get the full output. I would much rather have the description field completely omitted or at least extend off to the right (it's much less essential than the service name anyway).
Hey, it's pretty straighforward code in http://cgit.freedesktop.org/systemd/systemd/tree/src/systemctl/systemctl.c#n.... If you can design a better way to split available columns, go for it.
Zbyszek
It really feels like this discussion is beyond its peak usefulness.
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
On 07/17/2013 05:40 PM, Matthias Clasen wrote:
It really feels like this discussion is beyond its peak usefulness.
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
yum whatprovides "/var/log/*" | grep Filename | wc -l 597
About that much....
JBG
On Wed, 17 Jul 2013, "Jóhann B. Guðmundsson" wrote:
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
yum whatprovides "/var/log/*" | grep Filename | wc -l 597
Most of those will be packages that log directly to files in /var/log, and so don't care whether journald or rsyslog is used, though they may assume logrotate is present to handle their logs.
Michael Young
Once upon a time, "Jóhann B. Guðmundsson" johannbg@gmail.com said:
On 07/17/2013 05:40 PM, Matthias Clasen wrote:
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
yum whatprovides "/var/log/*" | grep Filename | wc -l 597
About that much....
That doesn't have anything to do with what _looks_ for syslog logs in the usual places (the count also inflated by updates - so you have dupes).
I don't think logrotate will care if the syslog files are not there. The /etc/lograte.d/syslog is provided by rsyslog (so if there's no rsyslog, there's no corresponding logrotate config).
logwatch, fail2ban, and denyhosts should either depend on syslog or be taught to understand the journal.
systemd should NOT provide syslog, since it is not providing a "syslog" service (it is an alternate system logging setup that can receive messages via the syslog socket). That virtual provides makes it harder for people that want a syslog service.
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 05:40 PM, Matthias Clasen wrote:
It really feels like this discussion is beyond its peak usefulness.
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
yum whatprovides "/var/log/*" | grep Filename | wc -l 597
About that much....
Aren't those mostly producers of log files, not consumers?
Zbyszek
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
yum whatprovides "/var/log/*" | grep Filename | wc -l 597 About that much....
That's not so helpful. Most of these are things which provide their own log files of some sort and completely bypass syslog. In fact, we can probably _subtract_ anything on that list.
On 07/17/2013 07:05 PM, Matthew Miller wrote:
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
yum whatprovides "/var/log/*" | grep Filename | wc -l 597 About that much....
That's not so helpful. Most of these are things which provide their own log files of some sort and completely bypass syslog. In fact, we can probably _subtract_ anything on that list.
Irrelevant since it's either a tool or an admin that is looking into those logs and both come up empty handed hence it needs to be fixed one way or the other. ( sub-packaged ) The only difference is that the tools wont file bugs when they find empty files and which tools are looking into these logs are hard to find since currently the relevant components dont have dependency on it.
Like it or not when introducing features we should also fix what is revealed to be broken and do the necessary cleanup not only be focusing on the "new in shiny thing" that feature introduces and that should be part of the feature process.
Anyway this needs to be fixed regardless if we ship only the journal by "default"
JBG
Once upon a time, "Jóhann B. Guðmundsson" johannbg@gmail.com said:
On 07/17/2013 07:05 PM, Matthew Miller wrote:
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
yum whatprovides "/var/log/*" | grep Filename | wc -l 597 About that much....
That's not so helpful. Most of these are things which provide their own log files of some sort and completely bypass syslog. In fact, we can probably _subtract_ anything on that list.
Irrelevant since it's either a tool or an admin that is looking into those logs and both come up empty handed hence it needs to be fixed one way or the other. ( sub-packaged )
No, it is not irrelevant, your count is. Your search found things like httpd, which provides /var/log/httpd, and writes its logs there. The proposed syslog removal has no effect on httpd, because it doesn't use syslog for logging by default.
The same is true of all the other matches you counted. The proposed change will have ZERO effect on anything that writes directly to /var/log (other than rsyslog itself).
On Wed, Jul 17, 2013 at 07:46:01PM +0000, "Jóhann B. Guðmundsson" wrote:
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
yum whatprovides "/var/log/*" | grep Filename | wc -l 597 About that much....
That's not so helpful. Most of these are things which provide their own log files of some sort and completely bypass syslog. In fact, we can probably _subtract_ anything on that list.
Irrelevant since it's either a tool or an admin that is looking into those logs and both come up empty handed hence it needs to be fixed one way or the other. ( sub-packaged )
I'm not following you here. To pick a random example from the top of the list, "3proxy" is going to keep logging to /var/log/3proxy no matter what we do with syslog. It won't "come up empty handed" -- from its point of view, nothing has changed.
Are you suggesting we somehow need to patch all of these to use the journal instead of their own ad-hoc log file? Because that is Definitely Out of Scope for this proposal.
On 07/17/2013 08:23 PM, Matthew Miller wrote:
On Wed, Jul 17, 2013 at 07:46:01PM +0000, "Jóhann B. Guðmundsson" wrote:
On Wed, Jul 17, 2013 at 05:42:21PM +0000, "Jóhann B. Guðmundsson" wrote:
yum whatprovides "/var/log/*" | grep Filename | wc -l 597 About that much....
That's not so helpful. Most of these are things which provide their own log files of some sort and completely bypass syslog. In fact, we can probably _subtract_ anything on that list.
Irrelevant since it's either a tool or an admin that is looking into those logs and both come up empty handed hence it needs to be fixed one way or the other. ( sub-packaged )
I'm not following you here. To pick a random example from the top of the list, "3proxy" is going to keep logging to /var/log/3proxy no matter what we do with syslog. It won't "come up empty handed" -- from its point of view, nothing has changed.
Are you suggesting we somehow need to patch all of these to use the journal instead of their own ad-hoc log file? Because that is Definitely Out of Scope for this proposal.
No that's not what I'm proposing.
JBG
HI
On Wed, Jul 17, 2013 at 4:32 PM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 08:23 PM, Matthew Miller wrote:
Are you suggesting we somehow need to patch all of these to use the journal instead of their own ad-hoc log file? Because that is Definitely Out of Scope for this proposal.
No that's not what I'm proposing.
Then using yum provides to determine the count as about 597 is wrong.
Rahul
On 07/17/2013 08:41 PM, Rahul Sundaram wrote:
HI
On Wed, Jul 17, 2013 at 4:32 PM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 08:23 PM, Matthew Miller wrote: Are you suggesting we somehow need to patch all of these to use the journal instead of their own ad-hoc log file? Because that is Definitely Out of Scope for this proposal. No that's not what I'm proposing.
Then using yum provides to determine the count as about 597 is wrong.
Cut that number by half if you like if it somehow makes you feel more comfortable but the fact we have around 550 - 600 service/daemons components in the distribution and still no policy or work being done to properly package stuff that need/wants or uses rsyslog/syslog-ng/logwatch/logrotate/logchec etc...
If you really dont understand the need for us to clean things as we introduce new features that's not something I can help you with.
It's fesco duty to do the necessary research into the actual scope of this feature and let's just give them time to do that and reach conclusion.
JBG
Once upon a time, "Jóhann B. Guðmundsson" johannbg@gmail.com said:
Cut that number by half if you like if it somehow makes you feel more comfortable but the fact we have around 550 - 600 service/daemons components in the distribution and still no policy or work being done to properly package stuff that need/wants or uses rsyslog/syslog-ng/logwatch/logrotate/logchec etc...
As you have been told repeatedly, your number does not represent what you are claiming for many reasons:
- many duplicate entries, due to multiple packages providing the same file/directory, updates, etc.
- more importantly, those packages are creating their own logfiles and have nothing to do with syslog and therefore nothing to do with this proposal
On 07/17/2013 08:55 PM, Chris Adams wrote:
Once upon a time, "Jóhann B. Guðmundsson"johannbg@gmail.com said:
Cut that number by half if you like if it somehow makes you feel more comfortable but the fact we have around 550 - 600 service/daemons components in the distribution and still no policy or work being done to properly package stuff that need/wants or uses rsyslog/syslog-ng/logwatch/logrotate/logchec etc...
As you have been told repeatedly, your number does not represent what you are claiming for many reasons:
You do realize I filed for the exact same thing for F18 as is being proposed here. You do realize I asked FESCO just to disable rsyslogd even just up to beta so we in QA could catch any potential fallout this change which tools etc. You do realize I filed proposal to change the packaging guidelines so I could start working on making this transaction as painless as possible for our userbase You do realize that I'm all for this change
And you do realize I dont need your petty remarks against my findings at that time when I was doing that work but feel free to add in your own research and your own findings instead of responding like this.
Any fallout that this change will bring from my pov will be entirely the fault of FESCO and FPC since they are the once that stood in the way from me from working ahead of us for once...
JBG
On Wed, Jul 17, 2013 at 09:12:46PM +0000, "Jóhann B. Guðmundsson" wrote:
You do realize I filed for the exact same thing for F18 as is being proposed here.
I know, at least. And, as I said earlier in a different subthread (I don't think you replied), the rejection didn't seem to me (I wasn't involved at the time) to be very harsh -- just that it wasn't ready yet. So, things have changed, because a year is a long time in the development of an early-stages software project like this. And, there's the impact on the cloud images, which wasn't a consideration before. So, it seemed worth proposing again. I don't think that _either_ invalidates your earlier proposal or makes the previous FESCO decision wrong.
All of that said, I think many of are still very confused about the relevance your count of packages which have non-syslog log files. Can you help explain the relevance?
If you have constructive suggestions for how this proposal could be improved, let's hear them, please!
On 07/17/2013 10:16 PM, Matthew Miller wrote:
On Wed, Jul 17, 2013 at 09:12:46PM +0000, "Jóhann B. Guðmundsson" wrote:
You do realize I filed for the exact same thing for F18 as is being proposed here.
I know, at least. And, as I said earlier in a different subthread (I don't think you replied), the rejection didn't seem to me (I wasn't involved at the time) to be very harsh -- just that it wasn't ready yet. So, things have changed, because a year is a long time in the development of an early-stages software project like this. And, there's the impact on the cloud images, which wasn't a consideration before. So, it seemed worth proposing again. I don't think that _either_ invalidates your earlier proposal or makes the previous FESCO decision wrong.
My proposal was meant as an proposal to start identifying and shake out any deficiencies in the journal and identify which tools relied on the traditional log files in as least intrusive manner as possible which was to just disable even just up to beta which would expose it to the entire QA community during that time with the benefit of identifying which tools and which deficiency needed to be fixed to be performed before the inevitable moment came when the actual removal of rsyslog would be requested ( which Lennart is doing now and why fesco at that time denied that request even today makes absolutely no sense to me ).
All of that said, I think many of are still very confused about the relevance your count of packages which have non-syslog log files. Can you help explain the relevance?
Let me paint the picture as I saw it what those two yeas ago, the question that haunted me then and I do believe still are relevant today and is not entirely related to the removal rsyslog but rather the act of switching to alternative logger .
Now that we as an community are considering switching from traditional login in text to a binary logger we as as community should step back look at the entire logging infrastructure and solutions their packaging and implementation thereof in the project and be asking ourselves the bigger questions instead of focusing on the usability of journalctl command with a very limited administrative perspective.
First and foremost we as a community need decide which standard we as a distribution have for our "default" syslogger, which IETF and other compliance standards it must pass.
Once that has been establish we can apply that to our "default" syslogger regardless if that's traditional logger or a binary one.
The next step is to ensure that we have procedures,policy,guidelines in place to ensure that all log related components are correctly packaged and have the correct dependency against each other, which is not the case for around 550 - 600 components that ship services/daemons, the logrotion and logwatch etc and ( and whatever else I looked at at that time ) is and what I was trying to start addressing with my FPC proposal
Currently we are shipping around 550 - 600 components that ship services/daemons most but probably not all can use syslog but may not be configured to do so which may or may not be affect by the act of changing to binary logger I guess depending on which IETF syslog standards that binary logger supports?
And as we all know log files are used for audits, for evidence in legal actions, for incident response, to reduce liability, and for various legal and regulatory compliance reasons so so we need to look into log alerting and parsing tools like but not limited to...|| logwatch logcheck swatch sec gui notifications/tools etc..
Along with looking into and identify which one of those are they packaging in the distribution,how are they packaged in the distribution, are they affected by this change if so how are they affected by this change?
In addition to the data retention tools I mentioned there above we need to also look into which HIDS,NIDS and Common IDSs like but not limited to...| |IPQ DBD Fail2ban Denyhosts OSSEC OpenVas Tiger Chrootkit rkhunter tripwire osiris samhain etc
All of which might be depended on traditional log files
In addition to that we might also need to also check various monitoring tools like but not limited to munin nagios zabbix ximon etc...
Then there are probably few applications out there we ship which do not fall into the above category but are crawling into those traditional log files.
Now the main argument of removing rsyslog all that I mentioned above that depends on traditional requires administrative installation and configuration thus it should not be a problem for administrator to install a tradition logger be rsyslog or syslog-ng at the same time they install and configure any of the previously mentioned stuff.
And that argument stands by itself but from my pov we should not only be focusing on that but rather be addressing the whole bigger picture.
That's the relevance
JBG
Once upon a time, "Jóhann B. Guðmundsson" johannbg@gmail.com said:
Currently we are shipping around 550 - 600 components that ship services/daemons most but probably not all can use syslog but may not be configured to do so which may or may not be affect by the act of changing to binary logger I guess depending on which IETF syslog standards that binary logger supports?
Lots of programs don't use syslog because it isn't sufficient for their needs. For some, there is not liable to be any common logging setup that will work for them.
However, as I've said repeatedly, your "yum whatprovides" check is flat wrong, and so is your repeated 550-600 components claim. If you look at the number of packages that provide something in /var/log (rather than your bogus "number of entries under /var/log" check), it comes to a much smaller number.
I come up with 216 packages (in F18) that put files under /var/log. However, even that number is inflated; some of those are not an issue. A few examples:
- setup: has /var/log/lastlog - util-linux: also has /var/log/lastlog - initscripts: has /var/log/{w,b}tmp - pam: has /var/log/tallylog - ntp: puts stats in /var/log/ntpstats - sendmail: puts stats in /var/log/mail
That's just a few I recognize and/or am familiar with. I'm sure there are others that provide something under /var/log that have absolutely no issue related to logging (/var/log is sometimes used as a catch-all for "things that change a lot").
Please stop with the 600 package "scare" number.
And as we all know log files are used for audits, for evidence in legal actions, for incident response, to reduce liability, and for various legal and regulatory compliance reasons so so we need to look into log alerting and parsing tools like but not limited to...||
That is a completely different requirement; if you want to look at auditable logging, that is way outside the scope of rsyslog vs. journald (since neither is any different with respect to security). Bringing that into a discussion of whether to remove syslog is far more off-topic than bikeshedding about the journalctl output, options, etc.
On 07/18/2013 01:30 PM, Chris Adams wrote:
However, as I've said repeatedly, your "yum whatprovides" check is flat wrong, and so is your repeated 550-600 components claim. If you look at the number of packages that provide something in /var/log (rather than your bogus "number of entries under /var/log" check), it comes to a much smaller number.
I dont know why you continue to claim that I'm dealing with bullshit numbers
F18 units, sys V initscripts thus services/daemons
$ repoquery --whatprovides '/usr/lib/systemd/system/*' --qf "%{name}" | sort -u | wc -l 569
Legacy sysv initscrips
$ repoquery --whatprovides '/etc/rc.d/init.d/*' --qf "%{name}" | sort -u | egrep -v '(-sysvinit|-initscript|-sysv)$' | wc -l 166
Total number of units/sysV initscripts/daemon in F18 = 762
You running....
$ repoquery --whatprovides '/var/log/*' --qf "%{name}" | sort -u | wc -l 219
Only proves that out of that 762 only 219 of those might be providing files /var/log ( while in fact that number ain't accurate in relation to unit,services and daemons )
Let's shave off 200 units due to them not being type service units and multiple units or legacy sysv inscription might be shipped in the same package which gives you 550 - 600 components range I was talking about...
Go through those 550 - 600 components and see how many of those for example are shipping log files, logrotation and logwatch files when they should or should not...
JBG
----- Original Message -----
On 07/17/2013 08:55 PM, Chris Adams wrote:
Once upon a time, "Jóhann B. Guðmundsson"johannbg@gmail.com said:
Cut that number by half if you like if it somehow makes you feel more comfortable but the fact we have around 550 - 600 service/daemons components in the distribution and still no policy or work being done to properly package stuff that need/wants or uses rsyslog/syslog-ng/logwatch/logrotate/logchec etc...
As you have been told repeatedly, your number does not represent what you are claiming for many reasons:
You do realize I filed for the exact same thing for F18 as is being proposed here. You do realize I asked FESCO just to disable rsyslogd even just up to beta so we in QA could catch any potential fallout this change which tools etc. You do realize I filed proposal to change the packaging guidelines so I could start working on making this transaction as painless as possible for our userbase You do realize that I'm all for this change
I do realize that you put a lot of effort, and I do realize you do not need to do that much effort if you keep rsyslogd as default.
On Wed, Jul 17, 2013 at 01:40:27PM -0400, Matthias Clasen wrote:
It really feels like this discussion is beyond its peak usefulness.
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
fail2ban no: journald support has been merged, so it's a matter of waiting for the upstream release and packaging that [1].
logrotate also no: no logs → nothing to rotate → no problem.
[1] https://github.com/fail2ban/fail2ban/commit/5ca6a9aeb6fd8e
Zbyszek
On Wed, Jul 17, 2013 at 01:40:27PM -0400, Matthias Clasen wrote:
It really feels like this discussion is beyond its peak usefulness.
Maybe we can give another shot of relevance by collecting a list of packages that depend on syslog (or are useless without /var/log/messages or other log files in /var/log) ?
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
Based on looking at http://codesearch.debian.net/, with both /var/log/messages and /var/log/syslog (debian's default), I come with a handful.
logcheck. Matthias Runge was maintaining this fairly actively about 2.5 years ago and then kind of trailed off. This is a shell script.
pcp (Performance Co-Pilot). Haven't really investigated.
nova-manage (from openstack) has an awful script to extract log info from log files (tries /var/log/syslog, then falls back to /var/log/messages). This would actually be significantly nicer using journalctl.
lvmdump, which makes a tarball of lvm diagnostic reports. Currently includes "recent entries from /var/log/messages".
hplip (HP Linux Imaging and Printing Project) - includes a helper script called "logcapture" intended to search log files for printer messages for diagnostic purposes. This looks in /var/log/syslog, /var/log/messages, and /var/log/cups/error_log.
There's also epylog (maybe not in debian?)
And really not much else other than documentation and examples -- which like as not _already_ aren't applicable. For example, the sudo man page references /var/log/syslog, as does the logfilter example in the rsync documentation.
On Wed, Jul 17, 2013 at 01:40:27PM -0400, Matthias Clasen wrote:
I've heard logwatch, logrotate and fail2ban mentioned. Are there others ?
Also worth noting that at least for fail2ban, the earlier change in Arch Linux (where they are, for better or for worse, leading the way) means that the next version of fail2ban already addresses this. https://github.com/fail2ban/fail2ban/pull/82
and now https://bugzilla.redhat.com/show_bug.cgi?id=985567
On Wed, Jul 17, 2013 at 04:24:01AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
The "..." in the middle of the line makes the output completely unusable.
That might be a bit of an exagerration, no?
If I run "systemctl" and see that "console-k...tem-start.service" or "fedora-st...init-late.service" exited, I have no way to use that information. If I try to run "systemctl status console-k...tem-start.service", I get "Loaded: error (Reason: No such file or directory)". I've also had many times where I tried to search (with "/somestring") for a service I knew part of the name of and failed to find it because the middle of the service name was truncated. I now completely avoid running "systemctl" without arguments because the truncation has made the command, to me, unreliable and useless. So, no, it's not an exaggeration.
Which isn't to say that systemctl is unusable, just systemctl without arguments.
Hey, it's pretty straighforward code in http://cgit.freedesktop.org/systemd/systemd/tree/src/systemctl/systemctl.c#n.... If you can design a better way to split available columns, go for it.
The output of "systemctl list-unit-files" is very usable, with no truncation of the most important text.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Wed, Jul 17, 2013 at 12:03:04PM -0500, Andrew McNabb wrote:
On Wed, Jul 17, 2013 at 04:24:01AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
The "..." in the middle of the line makes the output completely unusable.
That might be a bit of an exagerration, no?
If I run "systemctl" and see that "console-k...tem-start.service" or "fedora-st...init-late.service" exited, I have no way to use that information. If I try to run "systemctl status console-k...tem-start.service", I get "Loaded: error (Reason: No such file or directory)". I've also had many times where I tried to search (with "/somestring") for a service I knew part of the name of and failed to find it because the middle of the service name was truncated. I now completely avoid running "systemctl" without arguments because the truncation has made the command, to me, unreliable and useless. So, no, it's not an exaggeration.
Well, "unusable" is still an exaggertion.
But you've got a point here: I'd prefer for systmectl to avoid ellipsization when not on a tty. IIRC, this idea was rejected, because it also is confusing when the output changes significantly depending on how you use it. Maybe it is time to revisit this decision.
Zbyszek
On Wed, Jul 17, 2013 at 07:14:32PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
But you've got a point here: I'd prefer for systmectl to avoid ellipsization when not on a tty. IIRC, this idea was rejected, because it also is confusing when the output changes significantly depending on how you use it. Maybe it is time to revisit this decision.
Maybe some of the columns could be less verbose? "loaded" and "active" take up a lot of space.
On Wed, Jul 17, 2013 at 07:14:32PM +0200, Zbigniew Jędrzejewski-Szmek wrote:
But you've got a point here: I'd prefer for systmectl to avoid ellipsization when not on a tty. IIRC, this idea was rejected, because it also is confusing when the output changes significantly depending on how you use it. Maybe it is time to revisit this decision.
Part of what I was trying to say is that the ellipsization is harmful on a tty, too. If the actual unit names are printed without truncation, then the output can be the same whether on a tty or not.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Mon, Jul 15, 2013 at 04:22:43PM -0500, Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
This means that you haven't really used journalctl. It *is* much nicer. Try it, and you'll stop wanting to go back to 'less /var/log/messages' and 'tail -f /var/log/*'.
So I went and tried journalctl, and immediately hit the same UI stupidity as systemctl. Truncated lines, auto-paging, etc., unless I pipe to something else. Significantly differing behavior between direct output and pipes is just wrong. Having to remember some double-dash long option just to get non-truncated (and non-useless) log info is highly irritating.
What version are you looking at? Current version should not truncate lines.
I think that's now fixed across all systemd commands.
Once upon a time, Matthew Miller mattdm@fedoraproject.org said:
On Mon, Jul 15, 2013 at 04:22:43PM -0500, Chris Adams wrote:
So I went and tried journalctl, and immediately hit the same UI stupidity as systemctl. Truncated lines, auto-paging, etc., unless I pipe to something else. Significantly differing behavior between direct output and pipes is just wrong. Having to remember some double-dash long option just to get non-truncated (and non-useless) log info is highly irritating.
What version are you looking at? Current version should not truncate lines.
I think that's now fixed across all systemd commands.
Current up-to-date F18, which is systemd-201-2.fc18.7.x86_64.
On Mon, Jul 15, 2013 at 02:53:06PM -0600, Eric Smith wrote:
But it's what people actually use in 99.9% of cases. 99.9% of the time I don't need the extra information in the binary journal. Making /var/log/messages unavailable by default has a huge down side.
We should probably refrain from hyperbole on either side. It has
1) a training/learning cost because the tools are different 2) an inconvenience when the tools aren't easily available 3) a rare real case where no _tools_ are available but the system is partially live and you have no-off-system logging and you can't reboot into a diagnostic tools environment or take the disk offline 4) lack of current tools for attempting to recover a possibly scrambled file 5) early-adopter risk that the code is more fragile than expected and has unknown serious corruption cases
Have I missed something?
I'd rather see a structured text format, but I understand that that's computationally expensive. At least the binary format is a) simple and b) documented. http://www.freedesktop.org/wiki/Software/systemd/journal-files/
On Mon, 15.07.13 17:31, Matthew Miller (mattdm@fedoraproject.org) wrote:
- lack of current tools for attempting to recover a possibly scrambled file
Well, let's note on this one that the journal is fine with the tail-corrupted files with its default tool set. However, we have no nice tools currently for recovering for "middle" corrupted files, which however are far less likely.
Lennart
On Mon, Jul 15, 2013 at 2:01 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(Just to mention this: we are neither the pioneer on the no-default-syslog feature nor on no-default-sendmail... A lot of other
As a cross-distro chap, I can attest to this. Specially with sendmail.
Everytime I spot sendmail in a Fedora/RHEL/CentOS box I wonder when will it happen. Together with sudo-by-default (and root logins disabled & discouraged by default).
Back to the topic, what happens when I boot from a LiveCD to diagnose a borked machine? Some FAQs that come to mind that google doesn't answer:
- The easy part: I run journalctl -r /path/to/rootpart/ -- what if /var/lib/ is a mountpoint? Can I point directly to the 'database' file?
- What guarantees of completeness/consistency can we expect in practice from the journal in a hard poweroff situation? What is the price in terms of fsync() calls?
- What if the database file is corrupt? On occasion, I have read partially corrupted logfiles with less. Sure, there was a chunk of crud in the middle, but I could read past it. While the un-corrupted bits were far from reliable, they did provide helpful info.
This last item is my main worry. Perhaps working on prototype XOs, trying to sort out kernel/driver issues is not a mainstream use case...
cheers,
m -- martin.langhoff@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
On Mon, 15.07.13 14:21, Martin Langhoff (martin.langhoff@gmail.com) wrote:
On Mon, Jul 15, 2013 at 2:01 PM, Lennart Poettering mzerqung@0pointer.de wrote:
(Just to mention this: we are neither the pioneer on the no-default-syslog feature nor on no-default-sendmail... A lot of other
As a cross-distro chap, I can attest to this. Specially with sendmail.
Everytime I spot sendmail in a Fedora/RHEL/CentOS box I wonder when will it happen. Together with sudo-by-default (and root logins disabled & discouraged by default).
Back to the topic, what happens when I boot from a LiveCD to diagnose a borked machine? Some FAQs that come to mind that google doesn't answer:
- The easy part: I run journalctl -r /path/to/rootpart/ -- what if
/var/lib/ is a mountpoint? Can I point directly to the 'database' file?
You'd usually mount your journal files dir, then direct "journalctl -D /path/to/the/journal/files/dir" to it. It will then collect all files in that dir, interleave them and present them to you.
- What guarantees of completeness/consistency can we expect in
practice from the journal in a hard poweroff situation? What is the price in terms of fsync() calls?
We sync() all journal files to disk 5min after each write the latest (and also mark the files as "offline" when this happens, as inverse "dirty" flag). This timeout is just a default you can configure in with SyncIntervalSec= in journald.conf. The value is relatively high by default, but it's difficult to find a value that works for everybody well, since you need to balance out power management issues against safety guarantees.
In general, the journal file is written in a scheme where we first append things to the end, and then only update a few pointers in front. This scheme should be fairly safe regarding incomplete writes, as it is hard to corrupt log lines you already successfully wrote. If you lose something you will only lose something at the end (which is not too different from classic syslog).
When the journal starts up and finds a journal file is not marked "offline" it will immediately rotate the file and start a new one, in order to make sure we never fiddle with files that have incomplete transactions in them.
journalctl will read all files during runtime, regardless if online, rotated or offline, and will interleave them, trying to read as much of them as it can.
So, we try hard to make sure the files stay in a complete synced/offline state, and when that's not possible we at least try to minimize the chance of corruption. Furthermore when reading we make the best out of what we have.
With the default settings, in the worst case you might lose 4:59s of logs, but even that is very unlikely.
Of course, things are different if some other form of corruption takes place, i.e. where the fs gets corrupted so badly due to some external condition that the middle of the journal files is bad. We do not provide any recovery tools for that case currently.
- What if the database file is corrupt? On occasion, I have read
partially corrupted logfiles with less. Sure, there was a chunk of crud in the middle, but I could read past it. While the un-corrupted bits were far from reliable, they did provide helpful info.
With MaxFileSec= or SystemMaxFileSize= you control how much log messages we write into a journal file before we rotate and start with the next one. If you are afraid of "middle corruptions", then I'd recommend lowering these for your case, so that you might lose a rotated file here or there, but all the ones that are still fine will just work.
Lennart
On Mon, Jul 15, 2013 at 2:45 PM, Lennart Poettering mzerqung@0pointer.de wrote:
You'd usually mount your journal files dir, then direct "journalctl -D /path/to/the/journal/files/dir" to it. It will then collect all files in that dir, interleave them and present them to you.
Thanks! And that -D would be to the dir normally under /var/log/journal AIUI. My reference earlier to /var/lib/systemd/catalog was wrong I believe.
=>> - What guarantees of completeness/consistency can we expect in
practice from the journal in a hard poweroff situation? What is the price in terms of fsync() calls?
We sync() all journal files to disk 5min after each write the latest
that's new and interesting. AIUI pre-journal, the default was to sync() on every message for some logs (and overrriden with a hyphen for some logs). Am I wrong?
Heading offtopic: Are there ways to control sync on a per-service basis? Or to trigger a sync on things like authentication failures, OOM shootouts, or kernel oopses?
IOWs, logging potentially catastophic event events is always without guarantees, but when it works, it logs damn useful info.
(snipped lots of good info, thanks!)
When the journal starts up and finds a journal file is not marked "offline" it will immediately rotate the file and start a new one, in order to make sure we never fiddle with files that have incomplete transactions in them.
OK. That's sane and failsafe.
With the default settings, in the worst case you might lose 4:59s of logs, but even that is very unlikely.
The trick with this is... you usually have a short window between the catastrophic event and the problem.
Sometimes the problem cripples the system irretrievably. Can't win in that case. But there are plenty of "in the middle" cases where if logs written out in time can help diagnose issues.
Of course, things are different if some other form of corruption takes place, i.e. where the fs gets corrupted so badly due to some external condition that the middle of the journal files is bad. We do not provide any recovery tools for that case currently.
It's gonna happen at some point :-) Anyway, if the file-format is append-only and indexes and such are rebuildable, then it's within reach.
Still, not a happy prospect for the first admin to need it.
cheers,
m -- martin.langhoff@gmail.com - ask interesting questions - don't get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff
On Mon, Jul 15, 2013 at 03:07:43PM -0400, Martin Langhoff wrote:
that's new and interesting. AIUI pre-journal, the default was to sync() on every message for some logs (and overrriden with a hyphen for some logs). Am I wrong?
Yes. Or rather, your are not wrong, but this ship left the harbor with the switch to rsyslog, which doesn't sync unless $ActionFileEnableSync is on -- and we ship with it at the default, which is off. (And, I believe that it doesn't sync at all; the journald default is more conservative.)
Heading offtopic: Are there ways to control sync on a per-service basis? Or to trigger a sync on things like authentication failures, OOM shootouts, or kernel oopses?
On the other hand, rsyslog _does_ have the ability to turn it on in a fine grained way like this, and that does seem like a useful feature.
On Mon, 15.07.13 15:07, Martin Langhoff (martin.langhoff@gmail.com) wrote:
On Mon, Jul 15, 2013 at 2:45 PM, Lennart Poettering mzerqung@0pointer.de wrote:
You'd usually mount your journal files dir, then direct "journalctl -D /path/to/the/journal/files/dir" to it. It will then collect all files in that dir, interleave them and present them to you.
Thanks! And that -D would be to the dir normally under /var/log/journal AIUI.
Correct.
My reference earlier to /var/lib/systemd/catalog was wrong I believe.
That's the message catalog, which is useful to augment log messages with longer explanatory messages (accessible with journalctl -x). However, as the texts in the message catalog are not stellar yet we don't advertise it big time yet.
=>> - What guarantees of completeness/consistency can we expect in
practice from the journal in a hard poweroff situation? What is the price in terms of fsync() calls?
We sync() all journal files to disk 5min after each write the latest
that's new and interesting. AIUI pre-journal, the default was to sync() on every message for some logs (and overrriden with a hyphen for some logs). Am I wrong?
Yes, correct.
Heading offtopic: Are there ways to control sync on a per-service basis? Or to trigger a sync on things like authentication failures, OOM shootouts, or kernel oopses?
No. this is currently not implemented, but certainly sounds like worthwile additions. I think the best approach is to do this for log messages with certain log levels and higher, and make that configurable, and then simply make sure that OOM events and kernel oops set approrpiately high log levels. Added to the TODO list.
Of course, things are different if some other form of corruption takes place, i.e. where the fs gets corrupted so badly due to some external condition that the middle of the journal files is bad. We do not provide any recovery tools for that case currently.
It's gonna happen at some point :-) Anyway, if the file-format is append-only and indexes and such are rebuildable, then it's within reach.
Yes, writing such a tool should not be too difficult. We could even extend the logic and from time to time write resynchronization tag into the stream that is useful for resyncing access if the journal files get corrupted somewhere in the front. (But this is harder than it sounds, as we need to find a way to make sure that these tags cannot be part of the normal payload, so that the resyncing logic cannot easily be confused.)
Note that the journal files are basically just a stream of objects. There are objects that actually carry contents, and objects that are just useful for indexing things. When recovering a corrupted journal file it is sufficient to recover the former (and just skip over the rest which is the majority), the indexing can be regenerated easily.
Lennart
On Jul 15, 2013, at 5:11, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Jon.
On Mon, 15.07.13 17:26, Jonathan Masters (jcm@redhat.com) wrote:
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
I figure by "making it optional" you actually mean "not installing it by default"?
What kind of logic is this? So everything that we don't install by default dies? That is a very weird idea. There are tons of packages in Fedora that are not installed by default and very healthy.
I mean, what's next, you suggest to install Apache or MariaDB by default because otherwise "they die"?
Lennart
On 07/15/2013 09:26 PM, Jonathan Masters wrote:
On Jul 15, 2013, at 5:11, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Has syslog-ng entirely died since rsyslogd has been the default?
Anyway there really is no point in installing and running two loggers on embed/server/desktop wasting ram,diskspace and cpu cycles *for everybody* instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
Nobody is talking about removing from the distribution entirely and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision and actually with my QA hat on we could have been better prepared for this and worked to our advantage ( instead of be dealing with potential breakage afterwards) and would have done so when I submitted [1] but apparently such work flow are forbidden by FESCO/FPC in the project.
JBG
On Mon, Jul 15, 2013 at 09:53:56PM +0000, "Jóhann B. Guðmundsson" wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision and actually with my QA hat on we could have been better prepared for this and worked to our advantage ( instead of be dealing with potential breakage afterwards) and would have done so when I submitted [1] but apparently such work flow are forbidden by FESCO/FPC in the project.
I'm not sure what you're talking about with all of that. The entire discussion of the previous proposal is at http://meetbot.fedoraproject.org/fedora-meeting/2012-03-19/fesco.2012-03-19-... and I'm not seeing anything about a forbidden workflow.
I see
- journald doesn't cover all rsyslog functionality; this time around the proposal talks about that as on purpose - Something about "project lumberjack", but that's based around a government project which is now dead: http://cee.mitre.org/ And https://lists.fedorahosted.org/pipermail/lumberjack-developers/ is likewise stagnant. - impact of change not explained thoroughly enough; hopefully that too is better now - "not thrilled" about binary log - "don't think it's worth the effort at this point. maybe in the future..." - and "yeah, -1 for now. it just doesn't appear to be really ready yet."
I also had previously started a discussion about reasonable blockers for making journald the default, and there was good progress on that (with many of the things which came out as crucial crossed off the list). So the situation is different from how it was, plus of course the code has had some time time mature, all of which is why I agreed to sign on. (In addition to it being useful for the cloud image use case.)
On 07/15/2013 10:16 PM, Matthew Miller wrote:
On Mon, Jul 15, 2013 at 09:53:56PM +0000, "Jóhann B. Guðmundsson" wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision and actually with my QA hat on we could have been better prepared for this and worked to our advantage ( instead of be dealing with potential breakage afterwards) and would have done so when I submitted [1] but apparently such work flow are forbidden by FESCO/FPC in the project.
I'm not sure what you're talking about with all of that. The entire discussion of the previous proposal is at http://meetbot.fedoraproject.org/fedora-meeting/2012-03-19/fesco.2012-03-19-... and I'm not seeing anything about a forbidden workflow.
I see
- journald doesn't cover all rsyslog functionality; this time around the proposal talks about that as on purpose
- Something about "project lumberjack", but that's based around a government project which is now dead: http://cee.mitre.org/ And https://lists.fedorahosted.org/pipermail/lumberjack-developers/ is likewise stagnant.
- impact of change not explained thoroughly enough; hopefully that too is better now
- "not thrilled" about binary log
- "don't think it's worth the effort at this point. maybe in the future..."
- and "yeah, -1 for now. it just doesn't appear to be really ready yet."
This was just about disabling it even just up to beta so we could catch fallouts early on then fix then switch as opposed to switch try deal with breakage just before release and FESCO did not approve that which is that "forbidden workflow" and obviously that only applies to the FESCO at that time and the same members of FESCO then and now which I would be surprised if they have changed their opinions which I myself will be very interested to see why.
I also had previously started a discussion about reasonable blockers for making journald the default, and there was good progress on that (with many of the things which came out as crucial crossed off the list). So the situation is different from how it was, plus of course the code has had some time time mature, all of which is why I agreed to sign on. (In addition to it being useful for the cloud image use case.)
Here [1] FESCO points to FPC <---> FPC points to FESCO
Ironically this should be fixed regardless if the journal is default/only installed or not and actually from my pov this just makes it harder for FESCO to make that decision but hey FPC greatest thing since sliced bread
The fact is the our implementation of rsyslog or syslog-ng logwatch etc. in the distribution is not good, even the proper dependency between those components is missing.
JBG
On Mon, Jul 15, 2013 at 3:53 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision
Some of us would like to think that FESCO members might be influenced by arguments made on the development list. Maybe that's just wishful thinking.
Eric
On 16.07.2013 00:21, Eric Smith wrote:
On Mon, Jul 15, 2013 at 3:53 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision
Some of us would like to think that FESCO members might be influenced by arguments made on the development list. Maybe that's just wishful thinking.
Eric
I don't know how it goes with FESCO but once, there was a community voice that was heard by developers about Anaconda not hiding password characters during installation. Don't lose your faith.
Mateusz Marzantowicz
On Mon, 15 Jul 2013 16:21:57 -0600 Eric Smith brouhaha@fedoraproject.org wrote:
On Mon, Jul 15, 2013 at 3:53 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision
Some of us would like to think that FESCO members might be influenced by arguments made on the development list. Maybe that's just wishful thinking.
I can't speak for anyone else in FESCo, but I do read everything posted here and try and keep an open mind. Of course arguments with facts and logic are much more likely to sway my opinion than ones with "I don't like this", "+1", or folks who repeat themselves without adding anything new to the discussion.
kevin
On 07/15/2013 10:21 PM, Eric Smith wrote:
On Mon, Jul 15, 2013 at 3:53 PM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision
Some of us would like to think that FESCO members might be influenced by arguments made on the development list. Maybe that's just wishful thinking.
I would assume they take into account hard valid technical arguments not change in workflow ( which any change might bring ) or people pluses or minuses but maybe that's just my wishful thinking.
JBG
On Mon, Jul 15, 2013 at 11:02:09PM +0000, "Jóhann B. Guðmundsson" wrote:
I would assume they take into account hard valid technical arguments not change in workflow ( which any change might bring ) or people pluses or minuses but maybe that's just my wishful thinking.
"This has a change in workflow" _is_ hard, valid technical argument. Any change needs to overcome that in one of two ways: 1) minimize the pain of transition with compatibility paths, documentation, and serious thought about the user experience for current users; and 2) bring significant new value that weighs against it. The best changes work on both sides of the equation, of course.
On 07/15/2013 11:15 PM, Matthew Miller wrote:
On Mon, Jul 15, 2013 at 11:02:09PM +0000, "Jóhann B. Guðmundsson" wrote:
I would assume they take into account hard valid technical arguments not change in workflow ( which any change might bring ) or people pluses or minuses but maybe that's just my wishful thinking.
"This has a change in workflow" _is_ hard, valid technical argument. Any change needs to overcome that in one of two ways: 1) minimize the pain of transition with compatibility paths, documentation, and serious thought about the user experience for current users; and 2) bring significant new value that weighs against it. The best changes work on both sides of the equation, of course.
Yes we followed that rule of thumb when we introduced those changes with anaconda/systemd/gnome3 etc ;)
Anyway awhile back I did not file that ticket with FPC rsyslog proposal just because, with my QA hat on I can say daemon/service solution that depend on text files something like fail2ban etc. is what we need to be watching out for and that's where the breakage will be and afaik nothing has been done to
a) figure out which components those are ( and it will be hard to detect them all due to components not properly depend on rsyslog syslog-ng etc or even logwatch for that matter ) b) fix it ( which my FPC proposal was aiming for, with me actually doing the fixing part ). c) not seen maintainers that do maintain components that rely on files in /var/log chime in
But fesco get's to do all that research to properly determine the actual cause and effects and the scope of removing rsyslog...
JBG
On Mon, Jul 15, 2013 at 04:21:57PM -0600, Eric Smith wrote:
And honestly I dont understand why people are ack/nack-ing this since this is FESCO decision
Some of us would like to think that FESCO members might be influenced by arguments made on the development list. Maybe that's just wishful thinking.
If you mean arguments in the constructive sense, then, yes. That's the whole point of this process where the changes are posted for discussion.
On Mon, 15 Jul 2013 21:53:56 +0000 "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Nobody is talking about removing from the distribution entirely and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
I like journalctl, it gives good info.
but as a test do: echo | journalctl -x --no-pager --since-today | mailx -s "Today's Journal" user (real isp email )
then do: cat /var/log/messages | mailx -s "/var/log/messages" user (real isp email )
for me it was: Journalctl in calws-mail inbox: 5.47mb /var/log/messages in claws-mail inbox .58mb
On Tue, 16.07.13 08:34, Frank Murphy (frankly3d@gmail.com) wrote:
On Mon, 15 Jul 2013 21:53:56 +0000 "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Nobody is talking about removing from the distribution entirely and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
I like journalctl, it gives good info.
but as a test do: echo | journalctl -x --no-pager --since-today | mailx -s "Today's Journal" user (real isp email )
What's the "echo |" supposed to do?
Note that --no-pager is implied if you pipe the command.
Also note that it's --since=today, not --since-today.
then do: cat /var/log/messages | mailx -s "/var/log/messages" user (real isp email )
for me it was: Journalctl in calws-mail inbox: 5.47mb /var/log/messages in claws-mail inbox .58mb
Well, you added "-x" to the command line, so you get the message catalogue entries added in. Don't do that, and it is going to be smaller. Also, note that it's not defined where /var/log/messages is rotated, but "journalctl --since=today" will give you everything of today, regardless whether rotated or not.
So, you are comparing *very* different things.
Lennart
On Tue, 16 Jul 2013 13:22:07 +0200 Lennart Poettering mzerqung@0pointer.de wrote:
but as a test do: echo | journalctl -x --no-pager --since-today | mailx -s "Today's Journal" user (real isp email )
What's the "echo |" supposed to do?
For some reason first few emails, without it had nothing. Could be I am using rawhide on this.
Note that --no-pager is implied if you pipe the command.
Did not know this.
Also note that it's --since=today, not --since-today.
typo on my part.
Well, you added "-x" to the command line, so you get the message catalogue entries added in. Don't do that, and it is going to be smaller
But -x is one of the good benefits. Giving explanation.
Also, note that it's not defined where /var/log/messages is rotated, but "journalctl --since=today" will give you everything of today, regardless whether rotated or not.
Agreed
So, you are comparing *very* different things.
True,
But once a week my Fedora using children email me their messages, (in the believe "a stitch in time http://www.phrases.org.uk/meanings/a-stitch-in-time.html)
size matters, when on 3G-internet. They have maybe 5-10gb p\m. depending on plan.
Lot of non-sysadmin people using Fedora. Because they can do their work with it. Which may not even be in I.T. field.
I am not knocking journalctl, as I stated I like it "with -x" But, believe keep rsyslog installed by default for Desktops. Until such time, as there is a gui, can be installed or run from a live CD\DVD in rescue case where such buttons\titles as auto-search for journal \ compress journal for email,
I'm not against change.
On Tue, 16.07.13 13:11, Frank Murphy (frankly3d@gmail.com) wrote:
You do understand that this:
But -x is one of the good benefits. Giving explanation.
and this:
size matters, when on 3G-internet. They have maybe 5-10gb p\m. depending on plan.
are directly contradicting: you first pump up the log size with "-x", but you actually want less data to transfer.
You cannot have both. Classic syslog didn't have "-x", only journalctl has. If you use it your log output becomes larger, but you think that's bad and prefer the size of classic syslog there? Then don't use this new feature!
Until such time, as there is a gui, can be installed or run from a live CD\DVD in rescue case
You can invoke it easily from a livecd. Just mount the journal directory, and invoke journalctl -D on it, and you see its contents, nicely interleaved.
where such buttons\titles as auto-search for journal \
auto-search for journal? What is that supposed to be?
compress journal for email,
Hmm? You can easily compress it on your own, exactly the same as for classic syslog.
Lennart
On Tue, 16 Jul 2013 14:26:48 +0200 Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 16.07.13 13:11, Frank Murphy (frankly3d@gmail.com) wrote:
You do understand that this:
But -x is one of the good benefits. Giving explanation.
and this:
size matters, when on 3G-internet. They have maybe 5-10gb p\m. depending on plan.
are directly contradicting: you first pump up the log size with "-x", but you actually want less data to transfer.
It sounds so, but without -x, journalctl is no better than rsyslog,
You cannot have both. Classic syslog didn't have "-x", only journalctl has. If you use it your log output becomes larger, but you think that's bad and prefer the size of classic syslog there? Then don't use this new feature!
Until such time, as there is a gui, can be installed or run from a live CD\DVD in rescue case
You can invoke it easily from a livecd. Just mount the journal directory, and invoke journalctl -D on it, and you see its contents, nicely interleaved.
explain that to my daughter, who want to drive the car, not fix it.
where such buttons\titles as auto-search for journal \
auto-search for journal? What is that supposed to be?
it is for people to whom Terminal is where you get a train. Who can click a button and find it. (even young Linux users, can be point and click) My daughter is dyslexic, and blind in one eye. Explaining *.journal is beyond 55e13f347adb48f9b1089695da5080a7 is difficult at best.
I know that is a personal thing, but I have to think on behalf of other similar users.
compress journal for email,
Hmm? You can easily compress it on your own, exactly the same as for classic syslog.
Lennart
Again back to my daughter. Who does not live with us. I tried explaining xarchiver :( Uncompressed logs come in fine.
(I still have people asking me what a browser is, even on Win7\8)
On 07/15/2013 11:53 PM, "Jóhann B. Guðmundsson" wrote:
On 07/15/2013 09:26 PM, Jonathan Masters wrote:
On Jul 15, 2013, at 5:11, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Has syslog-ng entirely died since rsyslogd has been the default?
Anyway there really is no point in installing and running two loggers on embed/server/desktop wasting ram, diskspace and cpu cycles *for everybody*
There is, and it was pointed out already several times on this thread: because admins need text logs in /var/log/messages
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
False argument. People (on this thread) aren't complaining about journactl being a bad thing. They are complaining about /var/log/messages disappearing.
On 07/17/2013 12:36 PM, Denys Vlasenko wrote:
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Well my job description for the last 10 years and my pay check says otherwise so I dont know what you are getting at...
JBG
On 07/17/2013 02:41 PM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:36 PM, Denys Vlasenko wrote:
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Well my job description for the last 10 years and my pay check says otherwise so I dont know what you are getting at...
I'm saying that systemd developers are not full-time admins who troubleshoot and repair other people's Linux installations remotely.
If they were, they would see why "just install rsyslogd" isn't a solution.
On 07/17/2013 12:50 PM, Denys Vlasenko wrote:
On 07/17/2013 02:41 PM, "Jóhann B. Guðmundsson" wrote:
On 07/17/2013 12:36 PM, Denys Vlasenko wrote:
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Well my job description for the last 10 years and my pay check says otherwise so I dont know what you are getting at...
I'm saying that systemd developers are not full-time admins who troubleshoot and repair other people's Linux installations remotely.
Nope that's why they are developers...
If they were, they would see why "just install rsyslogd" isn't a solution.
Well I as an administrator and speaking as such am not seeing how the journal or systemd for that matter is preventing that from still taking place either by using the tools that systemd have to offers or how it's preventing administrators from installing rsyslogd to continue to use it if they want to do so.
I as an administrator actually love many of the plethora of administrator features systemd has to offer and I as an administrator do not have any trouble installing rsyslog or syslog-ng and configure it to be used in various infrastructures for clients.
JBG
On Wed, Jul 17, 2013 at 2:36 PM, Denys Vlasenko dvlasenk@redhat.com wrote:
On 07/15/2013 11:53 PM, "Jóhann B. Guðmundsson" wrote:
On 07/15/2013 09:26 PM, Jonathan Masters wrote:
On Jul 15, 2013, at 5:11, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Has syslog-ng entirely died since rsyslogd has been the default?
Anyway there really is no point in installing and running two loggers on embed/server/desktop wasting ram, diskspace and cpu cycles *for everybody*
There is, and it was pointed out already several times on this thread: because admins need text logs in /var/log/messages
No they don't. The admins needs logs to find out what happened. It does not matter which format they are stored in and where they are stored in.
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You can use grep / sed / awk on the jounrnal output as well.
On 07/17/2013 02:45 PM, drago01 wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You can use grep / sed / awk on the jounrnal output as well.
Please do read the text you are replying to.
On Wed, Jul 17, 2013 at 2:57 PM, Denys Vlasenko dvlasenk@redhat.com wrote:
On 07/17/2013 02:45 PM, drago01 wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You can use grep / sed / awk on the jounrnal output as well.
Please do read the text you are replying to.
I did.
On Wed, 17.07.13 14:36, Denys Vlasenko (dvlasenk@redhat.com) wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Again:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
And if you really need it as a file, you can do "journalctl > /var/log/messages", and have it in a file. And if that doesn't cut it and you want something that is "living", then install rsyslog and you got the real /var/log/messages back.
and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
False argument. People (on this thread) aren't complaining about journactl being a bad thing. They are complaining about /var/log/messages disappearing.
It's only disappearing as a file, it is not disappearing as a text format. "journalctl" has that, and thanks to the power of unix pipelines you can make use of that pretty much in the sam ways in grep/sed/awk as the text file itself.
Lennart
On 07/17/2013 03:39 PM, Lennart Poettering wrote:
On Wed, 17.07.13 14:36, Denys Vlasenko (dvlasenk@redhat.com) wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Again:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
We understand your point. We are not THAT stupid, you know.
Try to understand ours.
Text log was used for troubleshooting for decades already. In an emergency, text log is more easily accessible - in some severe cases, "recoverable" is a better word - than a binary one.
I don't understand why you are on a crusade to remove stuff which works, even after people conceded to your desire to have binary logs.
Now you are trying to push people to have ONLY binary logs.
I don't understand why you are on a crusade to remove stuff which works, even after people conceded to your desire to have binary logs.
Because that is how progress happens? Of backwards compatibility was held on forever it would be insane... Eventually things change and as Lennart pointed out Arch already does this...
Now you are trying to push people to have ONLY binary logs.
Way to say things that aren't there... He has categorically stated the opposite of this...
You already customise Austen's fur clients right? You don't just install default fedora and run right?
Just install rsyslogd as part of that process...
On 07/17/2013 03:51 PM, James Hogarth wrote:
I don't understand why you are on a crusade to remove stuff which works, even after people conceded to your desire to have binary logs.
Because that is how progress happens?
Progress happens by removing stuff which works? Interesting...
On Wed, Jul 17, 2013 at 3:58 PM, Denys Vlasenko dvlasenk@redhat.com wrote:
On 07/17/2013 03:51 PM, James Hogarth wrote:
I don't understand why you are on a crusade to remove stuff which works, even after people conceded to your desire to have binary logs.
Because that is how progress happens?
Progress happens by removing stuff which works?
Yes and replacing it by stuff that's better. You know people know use cars rather then horses. Not because horses stopped working but because cars turned to be better at the job.
On Wed, Jul 17, 2013 at 8:32 AM, drago01 drago01@gmail.com wrote:
On Wed, Jul 17, 2013 at 3:58 PM, Denys Vlasenko dvlasenk@redhat.com wrote:
Progress happens by removing stuff which works?
Yes and replacing it by stuff that's better. You know people know use cars rather then horses. Not because horses stopped working but because cars turned to be better at the job.
That happened because "end users" chose that for themselves, not because a few elite people said "cars are better, so we're going to make people jump through extra hoops if they want to keep using the horses they already have.
Eric
On 07/17/2013 03:51 PM, James Hogarth wrote:
I don't understand why you are on a crusade to remove stuff which works, even after people conceded to your desire to have binary logs.
Because that is how progress happens?
Progress? My feel is a new generation of programmers is repeating the mistakes their predecessors had understood.
Ralf
On Wednesday 17 July 2013 15:39:23 Lennart Poettering wrote:
On Wed, 17.07.13 14:36, Denys Vlasenko (dvlasenk@redhat.com) wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Again:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
And if you really need it as a file, you can do "journalctl > /var/log/messages", and have it in a file. And if that doesn't cut it and you want something that is "living", then install rsyslog and you got the real /var/log/messages back.
and quite frankly administrators that complain about journal have not actually tried it and experienced the flexibility the journactl gives them it truly is not as bad as some people are trying to make it out to be.
False argument. People (on this thread) aren't complaining about journactl being a bad thing. They are complaining about /var/log/messages disappearing.
It's only disappearing as a file, it is not disappearing as a text format. "journalctl" has that, and thanks to the power of unix pipelines you can make use of that pretty much in the sam ways in grep/sed/awk as the text file itself.
Lennart
Again with this?
The problem is that we need another tool (journalctl) in the middle of that process.
Not to mention the problems (in the form of bugs/incompatibilities) that such tool can introduce in the job ( text files and sed,grep... have decades of refinement).
Regards,
Marc Deop
On Wed, 17 Jul 2013, Lennart Poettering wrote:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
One thing you have missed is how you edit the log file. There may be cases where you want to strip out log entries, eg. when a process has gone wild and swamped the useful messages with useless ones and you want to keep the useful ones and throw away the useless ones.
Michael Young
From: m.a.young@durham.ac.uk
On Wed, 17 Jul 2013, Lennart Poettering wrote:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
One thing you have missed is how you edit the log file. There may be
cases
where you want to strip out log entries, eg. when a process has gone
wild
and swamped the useful messages with useless ones and you want to keep
the
useful ones and throw away the useless ones.
I used to do something like this with vim ":g/NOISE/d" until I could see the detail I wanted when the alternations for grep would have been tremendously long. With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise. However, all use cases differ, so if you must, you can: "journalctl | vim -". YMMV with other editors though.
-- John Florian
From: John.Florian@dart.biz With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise.
I do wish journalctl had a -C option (like ps) to be equivalent to _COMM= much like -u equates to _SYSTEMD_UNIT. I realize if all the fields were thusly accessible it would result in option bloat quickly, but I'd think filtering on _COMM is quite routine -- it is for me at least.
-- John Florian
On Wed, Jul 17, 2013 at 10:48:55AM -0400, John.Florian@dart.biz wrote:
From: John.Florian@dart.biz With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise.
I do wish journalctl had a -C option (like ps) to be equivalent to _COMM= much like -u equates to _SYSTEMD_UNIT. I realize if all the fields were thusly accessible it would result in option bloat quickly, but I'd think filtering on _COMM is quite routine -- it is for me at least.
You can provide binary path (_EXE=) by ”journalctl /usr/sbin/sshd”.
From: tomek@pipebreaker.pl
On Wed, Jul 17, 2013 at 10:48:55AM -0400, John.Florian@dart.biz wrote:
From: John.Florian@dart.biz With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise.
I do wish journalctl had a -C option (like ps) to be equivalent to
_COMM=
much like -u equates to _SYSTEMD_UNIT. I realize if all the fields
were
thusly accessible it would result in option bloat quickly, but I'd
think
filtering on _COMM is quite routine -- it is for me at least.
You can provide binary path (_EXE=) by ”journalctl /usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
-- John Florian
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl /usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
journalctl _COMM=<blah> works for me on F19.
Bill
From: notting@redhat.com
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this, much like -u equates to _SYSTEMD_UNIT.
-- John Florian
John.Florian@dart.biz (John.Florian@dart.biz) said:
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this, much like -u equates to _SYSTEMD_UNIT.
It's available. File an RFE, if you haven't already?
Bill
From: notting@redhat.com
John.Florian@dart.biz (John.Florian@dart.biz) said:
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just
this,
much like -u equates to _SYSTEMD_UNIT.
It's available. File an RFE, if you haven't already?
Yeah, I was hoping Lennart would just pop in with "good idea, added to the TODO" so I could avoid the sluggishness that is BZ. However, I didn't see that happen and feel no right in complaining without a BZ, so here we go :-)
https://bugzilla.redhat.com/show_bug.cgi?id=985548 [RFE: improve journalctl filtering by _COMM]
-- John Florian
On Wed, Jul 17, 2013 at 12:09:14PM -0400, John.Florian@dart.biz wrote:
From: notting@redhat.com
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this,
This surely could be done. But maybe it would be better to make 'journalctl /path/to/program' smarter, so that it would look at _COMM when program is not an executable. This way things would work automagically.
much like -u equates to _SYSTEMD_UNIT.
Just to clear up potential confusion: -u is *not* equivalent to _SYSTEMD_UNIT. You can show the match by looking at the (stricly unofficial) debug output: $ SYSTEMD_LOG_LEVEL=debug journalctl -u XXXXX |& grep 'Journal filter'
Journal filter: ((OBJECT_SYSTEMD_UNIT=XXXXX.service AND _UID=0) OR (UNIT=XXXXX.service AND _PID=1) OR (COREDUMP_UNIT=XXXXX.service AND _UID=0 AND MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1) OR _SYSTEMD_UNIT=XXXXX.service)
The output is still authoritative, but you get more than just messages originating from the unit.
Zbyszek
From: zbyszek@in.waw.pl
much like -u equates to _SYSTEMD_UNIT.
Just to clear up potential confusion: -u is *not* equivalent to
_SYSTEMD_UNIT.
You can show the match by looking at the (stricly unofficial) debug
output:
$ SYSTEMD_LOG_LEVEL=debug journalctl -u XXXXX |& grep 'Journal filter'
Journal filter: ((OBJECT_SYSTEMD_UNIT=XXXXX.service AND _UID=0) OR (UNIT=XXXXX.service AND _PID=1) OR (COREDUMP_UNIT=XXXXX.service AND _UID=0 AND MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1) OR _SYSTEMD_UNIT=XXXXX.service)
The output is still authoritative, but you get more than just messages originating from the unit.
I just *knew* I was going to be corrected on that. Thanks for the explanation though.
-- John Florian
From: zbyszek@in.waw.pl
On Wed, Jul 17, 2013 at 12:09:14PM -0400, John.Florian@dart.biz wrote:
From: notting@redhat.com
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted
languages (e.g., python). I want to match on the name of the
python
program, not python itself.
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just
this,
This surely could be done. But maybe it would be better to make 'journalctl /path/to/program' smarter, so that it would look at _COMM
when
program is not an executable. This way things would work automagically.
That would be suitable too, if not more so. Also, for whatever reason, I've noticed that "journalctl blah" is much, much slower than "journalctl _EXE=bla". Is that a bug or are they not exactly equivalents?
-- John Florian
On Wed, Jul 17, 2013 at 01:39:30PM -0400, John.Florian@dart.biz wrote:
From: zbyszek@in.waw.pl On Wed, Jul 17, 2013 at 12:09:14PM -0400, John.Florian@dart.biz wrote:
From: notting@redhat.com John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted
languages (e.g., python). I want to match on the name of the
python
program, not python itself.
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just
this,
This surely could be done. But maybe it would be better to make 'journalctl /path/to/program' smarter, so that it would look at _COMM
when
program is not an executable. This way things would work automagically.
That would be suitable too, if not more so. Also, for whatever reason, I've noticed that "journalctl blah" is much, much slower than "journalctl _EXE=bla". Is that a bug or are they not exactly equivalents?
It only does an extra stat on the file do termine its kind, and then adds "_EXE=..." match. There shouldn't be any speed difference.
Zbyszek
From: zbyszek@in.waw.pl It only does an extra stat on the file do termine its kind, and then adds "_EXE=..." match. There shouldn't be any speed difference.
Hmmm... I cannot reproduce it now. It must have been something else. Please disregard and my apologies for the noise.
-- John Florian
On Wed, 17.07.13 18:18, Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) wrote:
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this,
This surely could be done. But maybe it would be better to make 'journalctl /path/to/program' smarter, so that it would look at _COMM when program is not an executable. This way things would work automagically.
I must say, I don't like ambiguities. The thing about /path/to/program is that just by looking at the thing we can identify this as file system path, because it starts with a slash. comm names otoh are not that easily recognizable, there's no specific char they start with.
Lennart
On Thu, Jul 18, 2013 at 12:28:10AM +0200, Lennart Poettering wrote:
On Wed, 17.07.13 18:18, Zbigniew Jędrzejewski-Szmek (zbyszek@in.waw.pl) wrote:
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this,
This surely could be done. But maybe it would be better to make 'journalctl /path/to/program' smarter, so that it would look at _COMM when program is not an executable. This way things would work automagically.
I must say, I don't like ambiguities. The thing about /path/to/program is that just by looking at the thing we can identify this as file system path, because it starts with a slash. comm names otoh are not that easily recognizable, there's no specific char they start with.
I mean that the other way around actually: if I say
journalctl /usr/bin/yum-builddep
where yum-buildep is a python script, journalctl would look at the file type (looking for a #! line or whatever the way to detect that is), and use _COMM=yum-builddep instead of _EXE=/usr/bin/yum-builddep, because _EXE won't match anything in this context.
Zbyszek
On Wed, 17.07.13 12:09, John.Florian@dart.biz (John.Florian@dart.biz) wrote:
From: notting@redhat.com
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
journalctl _COMM=<blah> works for me on F19.
As it does for me, but somewhere it got clipped that what I was asking/wishing for was a convenient -C option (like ps) to do just this, much like -u equates to _SYSTEMD_UNIT.
The kernel clips that. The process name field in the kernel is limited to 16 chars.
Note that there is Tab completion for _COMM, hence I don't think the truncation thing is really that annoying: after all the shell will auto-complete stuff to the right number of chars anyway if you just type in a few.
I am a bit reluctant to adding more and more high-level options like "-u" unless there's some major advantage in it. For "-u" there is, because the filter will also actually look for a unit name in quite a few fields, not just in _SYSTEMD_UNIT= (see Zbyszek's mail about that). Also, "-u" will do unit name normalization, i.e. automatically append the ".service" suffix for you if you omit and stuff.
If you can suggest something similarly powerful that -C would to, then I am all ears, but the truncation thing doesn't really convince me too much I must say...
Lennart
On Wed, 17.07.13 11:48, Bill Nottingham (notting@redhat.com) wrote:
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl /usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python program, not python itself.
journalctl _COMM=<blah> works for me on F19.
Also, to mention this explicitly: there is command line completion for this. I.e. type "journalctl _CO", press Tab, and it will autocomplete to "journalctl _COMM=", then press Tab Tab and it shows you a list of all values that so far have been recorded.
This is realy, really useful actually.
Lennart
From: mzerqung@0pointer.de
On Wed, 17.07.13 11:48, Bill Nottingham (notting@redhat.com) wrote:
John.Florian@dart.biz (John.Florian@dart.biz) said:
You can provide binary path (_EXE=) by ”journalctl
/usr/sbin/sshd”.
Yes, but that's of little help with applications using interpreted languages (e.g., python). I want to match on the name of the python
program, not python itself.
journalctl _COMM=<blah> works for me on F19.
Also, to mention this explicitly: there is command line completion for this. I.e. type "journalctl _CO", press Tab, and it will autocomplete to "journalctl _COMM=", then press Tab Tab and it shows you a list of all values that so far have been recorded.
This is realy, really useful actually.
Huh. Given how it seems that I hit Tab nearly every other keystroke I'm surprised I hadn't stumbled onto that yet. Probably because I haven't gotten much into the habit of filtering with these terms. I'm still so pleased just having -u (and the wonderful extra info conveyed via color) that it's hard to grumble about much else. :-)
I may have avoided these largely because I knew it would take time to learn the various names. But now I know I can "journalctl _ <Tab><Tab>" and have an instant refresher. Thanks for such a well-done tool.
-- John Florian
On Wed, 17 Jul 2013, John.Florian@dart.biz wrote:
From: m.a.young@durham.ac.uk
On Wed, 17 Jul 2013, Lennart Poettering wrote:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
One thing you have missed is how you edit the log file. There may be cases where you want to strip out log entries, eg. when a process has gone wild and swamped the useful messages with useless ones and you want to keep the useful ones and throw away the useless ones.
I used to do something like this with vim ":g/NOISE/d" until I could see the detail I wanted when the alternations for grep would have been tremendously long. With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise. However, all use cases differ, so if you must, you can: "journalctl | vim -". YMMV with other editors though.
That isn't a complete solution though because you may want to remove the bad logs completely to free up the space they are taking up. Of course you may have already lost all the interesting logs by this point with journald anyway because they have been overwritten.
That leads me to ask another question, how well does journald cope with keeping certain logs long term? The classic syslog way of doing this is to send them to a separate file, then use logrotate to compress them once they have been rotated. Is there any equivalent with journald? Compressing may be necessary due to the quantity of logs required.
Michael Young
On Wed, Jul 17, 2013 at 04:37:10PM +0100, M A Young wrote:
That leads me to ask another question, how well does journald cope with keeping certain logs long term? The classic syslog way of doing this is to send them to a separate file, then use logrotate to compress them once they have been rotated. Is there any equivalent with journald? Compressing may be necessary due to the quantity of logs required.
This is mentioned in the feature proposal -- if you have particular data retention and lifecycle policies (common in enterprise deployments), you will want to add rsyslog. This shouldn't be a very big change in practice, since you will then also need to _configure_ rsyslog for your particular case.
As we eventually get applications which are able to use the more rich logging capabilities of the journal, we'll also need something to deal with this. However, I don't think doing it at the per-system journal level is appropriate -- it's better to send the logs to a log processing / management application.
On Wed, 17.07.13 16:37, M A Young (m.a.young@durham.ac.uk) wrote:
I used to do something like this with vim ":g/NOISE/d" until I could see the detail I wanted when the alternations for grep would have been tremendously long. With journalctl's built-in filtering capabilities I'm glad I don't have to do that anymore; it's way more concise. However, all use cases differ, so if you must, you can: "journalctl | vim -". YMMV with other editors though.
That isn't a complete solution though because you may want to remove the bad logs completely to free up the space they are taking up. Of course you may have already lost all the interesting logs by this point with journald anyway because they have been overwritten.
That leads me to ask another question, how well does journald cope with keeping certain logs long term? The classic syslog way of doing this is to send them to a separate file, then use logrotate to compress them once they have been rotated. Is there any equivalent with journald? Compressing may be necessary due to the quantity of logs required.
Regarding retention policies we currently only have a max retention time limit which you can configure globally.
There are plans to add a bit of code to allow rotation that throws away lower priority messages earlier than high priority messages during rotation and then add individual retention time limits for that too.
However, I am not keen on adding a complex language that allows matching of specific clients or messages and then apply specific retention times to that. For that please use rsyslog which supports such a language.
Lennart
On 07/17/2013 02:04 PM, M A Young wrote:
One thing you have missed is how you edit the log file. There may be cases where you want to strip out log entries, eg. when a process has gone wild and swamped the useful messages with useless ones and you want to keep the useful ones and throw away the useless ones.
Allowing editing of log files is a pure security risk...
JBG
On Wed, Jul 17, 2013 at 8:39 AM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Allowing editing of log files is a pure security risk...
So is giving a sysadmin the root password, but we do it.
I generally make a copy of a log file and edit the copy, but I'd oppose anything that took away the ability for log files to be edited.
Eric
On 07/17/2013 04:52 PM, Eric Smith wrote:
On Wed, Jul 17, 2013 at 8:39 AM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Allowing editing of log files is a pure security risk...
So is giving a sysadmin the root password, but we do it.
I generally make a copy of a log file and edit the copy, but I'd oppose anything that took away the ability for log files to be edited.
Nobody is taking that functionality as has been pointed out several times already you simple pipe the output from the journal to a separated file heck you can even filter that immediately then and there with the journalctl command
JBG
On Wed, 17 Jul 2013, Eric Smith wrote:
On Wed, Jul 17, 2013 at 8:39 AM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Allowing editing of log files is a pure security risk...
So is giving a sysadmin the root password, but we do it.
I generally make a copy of a log file and edit the copy, but I'd oppose anything that took away the ability for log files to be edited.
Another reason why you might to edit the journal is when you have to keep logs for a precise time for regulatory reasons. This isn't a problem under the classic logging and rotation.
Michael Young
On Wed, Jul 17, 2013 at 06:58:33PM +0100, M A Young wrote:
I generally make a copy of a log file and edit the copy, but I'd oppose anything that took away the ability for log files to be edited.
Another reason why you might to edit the journal is when you have to keep logs for a precise time for regulatory reasons. This isn't a problem under the classic logging and rotation.
Gah! Why would you edit the journal for that? I think that's very much the wrong approach. Right now, the solution there is to set the journal to the longest common acceptable lifetime, and then use rsyslog-based logging for policy implementation. As discussed in another subthread, this requires configuration and so does not affect the question of the defaults.
I think this is probably the right thing to do; in a large-scale environment, one doesn't really want any long-term logs on the individual systems at all if possible, right?
(Note that time-based limits _at all_ came up in the previous round of discussions on what it would take to make journal the default logging solution, and they were added to match the that request.)
On Wed, 17.07.13 18:58, M A Young (m.a.young@durham.ac.uk) wrote:
On Wed, 17 Jul 2013, Eric Smith wrote:
On Wed, Jul 17, 2013 at 8:39 AM, "Jóhann B. Guðmundsson" johannbg@gmail.com wrote:
Allowing editing of log files is a pure security risk...
So is giving a sysadmin the root password, but we do it.
I generally make a copy of a log file and edit the copy, but I'd oppose anything that took away the ability for log files to be edited.
Another reason why you might to edit the journal is when you have to keep logs for a precise time for regulatory reasons. This isn't a problem under the classic logging and rotation.
To implement retention policy please use the retention features, i.e. "MaxRetentionSec=" in journald.conf.
"MaxRetentionSec=2months" will set the retention time to 2months, and we will synchronously delete all older messages as soon as that time is hit.
(And in case you wonder, the "Sec" suffix just indicates the default time unit if you don't specify any. In the above line we specified "month", hence the time unit is months)
Lennart
On Wed, 17.07.13 15:04, M A Young (m.a.young@durham.ac.uk) wrote:
On Wed, 17 Jul 2013, Lennart Poettering wrote:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
One thing you have missed is how you edit the log file. There may be cases where you want to strip out log entries, eg. when a process has gone wild and swamped the useful messages with useless ones and you want to keep the useful ones and throw away the useless ones.
Well, with stuff like FSS we actually provide you with extra features to track log file manipulations. We try to make it harder to manipulate log files and easier to detect manipulated log files.
We also provide you with tools to effectively counter misbehaving log clients: by default there's a per-service rate limiting logic, that is slightly modulated by the available disk space (the more space is available the later we start rate limiting log output) and by the log level (to ensure that errors still are captured at a point where debug messages are already dropped because of a flood). Together this should be a pretty effective way to counter log floods automatically without admin intervention, in a way that one misbehaving service cannot cause loss of messages of other services, and in a way that the important stuff still gets through if there's a flood.
Lennart
----- Original Message -----
On Wed, 17.07.13 14:36, Denys Vlasenko (dvlasenk@redhat.com) wrote:
instead of administrators simply adding rsyslog or syslog-ng manually at install time or to their ks snippets.
And this too was answered several times already. The machine in question may be already borked. Our support people will need to figure out - over the phone or email! - what has happened on client's installation, and having traditional grep/sed/awk recipes not working anymore because /var/log/messages is not there anymore is an extremely unwelcome discovery in an emergency.
You guys aren't administrators who are dealing with these problems every day. You don't feel the pain you create for other people.
Again:
"cat /var/log/messages" becomes "journalctl" "tail -f /var/log/messages" becomes "journalctl -f" "tail -n100 /var/log/messages" becomes "journalctl -n100" "grep foobar /var/log/messages" becomes "journalctl | grep foobar"
This isn't complex. You can grep/sed/awk as much as you want. You just do it over the output of journalctl rather than teh file. That's not that big a difference.
And if you really need it as a file, you can do "journalctl > /var/log/messages", and have it in a file. And if that doesn't cut it and you want something that is "living", then install rsyslog and you got the real /var/log/messages back.
1) yum remove rsyslog
2) a line in your kickstart file -rsyslog
3) uncheck the checkbox of rsyslog in anaconda or yumex
All three are not complex either. :-)
By letting rsyslog the default, you make the one that don't embrace the change happy, because they can continue to do what they always used to do.
And for the innovation and aware seekers like yourself, you know exact what you want and don't want, can safely remove the package. A simple change from your side is just a tiny price to pay.
This should be simpler than forcing those stubborn mind (such as me) to change, No?
On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me) to change, No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
Lennart
On 07/18/2013 08:09 AM, Lennart Poettering wrote:
On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me) to change, No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
Lennart
Or you can de-install rsyslog and have everything as you love it.
On Thu, Jul 18, 2013 at 9:59 AM, Steve Clark sclark@netwolves.com wrote:
On 07/18/2013 08:09 AM, Lennart Poettering wrote:
On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me) to change, No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
Lennart
Or you can de-install rsyslog and have everything as you love it.
+1
At long last, someone has come up with the most concise, sane solution to this. Thanks Steve.
Someone should really document how to uninstall syslog. Knowing how could have saved a lot of people a lot of time....
From: sclark@netwolves.com
On 07/18/2013 08:09 AM, Lennart Poettering wrote: On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me)to
change,
No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
Lennart
Or you can de-install rsyslog and have everything as you love it.
Which makes more sense: take a default and modify it via composition ... or take a default and modify it via decomposition?
I'd always choose the former, regardless of the case or how convenient it was to me.
-- John Florian
John.Florian@dart.biz (John.Florian@dart.biz) said:
Or you can de-install rsyslog and have everything as you love it.
Which makes more sense: take a default and modify it via composition ... or take a default and modify it via decomposition?
I'd always choose the former, regardless of the case or how convenient it was to me.
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer beware' or 'you get both pieces' operation.
(Also, I do wonder if those who suggest unchecking rsyslog *in anaconda* do regular interactive installs. That's not been offered for quite some time now outside of kickstart.)
Bill
----- Original Message -----
John.Florian@dart.biz (John.Florian@dart.biz) said:
Or you can de-install rsyslog and have everything as you love it.
Which makes more sense: take a default and modify it via composition ... or take a default and modify it via decomposition?
I'd always choose the former, regardless of the case or how convenient it was to me.
Taking your words seriously, the most sensible default is only install the core, nothing else.
You want network? install network drivers, GUI? install desktop environment/WM. Show your languages? Install language support.
For Next->Next->Ok type of users, this "default" will lead them just a .. core. No pretty GUI. :-)
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer beware' or 'you get both pieces' operation.
Didn't Jesse Keating said something like we don't offer minimal install other than uncheck the all boxes?
Default is just an environment that most people expected to have, its much bigger that the minimal install.
(Also, I do wonder if those who suggest unchecking rsyslog *in anaconda* do regular interactive installs. That's not been offered for quite some time now outside of kickstart.)
That's because it is in core, which is always hidden from users. You can move it to standard and make it default.
On Thu, 2013-07-18 at 21:37 -0400, Ding Yi Chen wrote:
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer beware' or 'you get both pieces' operation.
Didn't Jesse Keating said something like we don't offer minimal install other than uncheck the all boxes?
I doubt Jesse would say that, and if he did, he'd be wrong: it's a primary group in the package selection spoke of an equal status with GNOME, KDE etc.
On Fri, Jul 19, 2013 at 10:37 AM, Adam Williamson awilliam@redhat.comwrote:
On Thu, 2013-07-18 at 21:37 -0400, Ding Yi Chen wrote:
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer
beware'
or 'you get both pieces' operation.
Didn't Jesse Keating said something like we don't offer minimal install other than uncheck the all boxes?
I doubt Jesse would say that, and if he did, he'd be wrong: it's a primary group in the package selection spoke of an equal status with GNOME, KDE etc.
Perhaps there should be a "Least Possible Bootable" install for situations like this. I would agree Syslog should be missing from such an install. Just not from Default -- Not until journalctl and systemd attain ubiquity..
Billy Crook (billycrook@gmail.com) said:
Perhaps there should be a "Least Possible Bootable" install for situations like this. I would agree Syslog should be missing from such an install. Just not from Default -- Not until journalctl and systemd attain ubiquity..
The installation only has a default because of user requests to not require anaconda to go into that menu. It's still a choice of one of the installation options presented (GNOME, KDE, minimal, infrastructure server, etc.). Minimal is definitely a *subset* of all the others, but that doesn't make it a 'default'.
Bill
----- Original Message -----
On Thu, 2013-07-18 at 21:37 -0400, Ding Yi Chen wrote:
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer beware' or 'you get both pieces' operation.
Didn't Jesse Keating said something like we don't offer minimal install other than uncheck the all boxes?
I doubt Jesse would say that, and if he did, he'd be wrong: it's a
He did, see this and the thread Regarding install options: http://www.redhat.com/archives/rhl-devel-list/2008-October/msg01277.html
primary group in the package selection spoke of an equal status with GNOME, KDE etc.
Well, nowadays we even have yum in the core. :-)
-- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Sun, 2013-07-21 at 20:46 -0400, Ding Yi Chen wrote:
----- Original Message -----
On Thu, 2013-07-18 at 21:37 -0400, Ding Yi Chen wrote:
Exactly - adding to the minimal install is generally always a supported operation. Removing from the minimal install is always a 'buyer beware' or 'you get both pieces' operation.
Didn't Jesse Keating said something like we don't offer minimal install other than uncheck the all boxes?
I doubt Jesse would say that, and if he did, he'd be wrong: it's a
He did, see this and the thread Regarding install options: http://www.redhat.com/archives/rhl-devel-list/2008-October/msg01277.html
Um. You could have qualified your statement with 'in 2008'. That was the old anaconda UI. We have been using a new one for two releases.
On Thu, 2013-07-18 at 10:59 -0400, Steve Clark wrote:
On 07/18/2013 08:09 AM, Lennart Poettering wrote:
On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me) to change, No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
Lennart
Or you can de-install rsyslog and have everything as you love it.
Yes, that is indeed the decision we are trying to make. The point is that for _either_ side to describe the _other_ side as trying to 'force' anything on anyone is disingenuous.
----- Original Message -----
On Wed, 17.07.13 22:35, Ding Yi Chen (dchen@redhat.com) wrote:
This should be simpler than forcing those stubborn mind (such as me) to change, No?
We don't force anyone. You can just install rsyslog and you have everything as you love it.
And we don't force anyone to keep rsyslog, you can just remove rsyslog and you have everything as you love it.
On Mon, Jul 15, 2013 at 05:26:10PM -0400, Jonathan Masters wrote:
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Jon, I'm not worried about this. From a presentation (http://www.youtube.com/watch?v=GTS7EuSdFKE) by Rainer Gerhards, the rsyslog author/maintainer:
- So we re-focussed the project * Previously low-end and enterprise needs were equal peers * Now strong focus on enterprise
- The logging world at large got benefit as suddenly everyone was interested in logging – which also helps rsyslog!
Jon.
devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On 07/15/2013 04:26 PM, Jonathan Masters wrote:
On Jul 15, 2013, at 5:11, Miroslav Suchý msuchy@redhat.com wrote:
On 07/15/2013 10:44 AM, Jaroslav Reznik wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
My voice may be one of thousands, but I'm saying: I want to have traditional syslog service as default and have journal from systemd as option.
I concur. I have systems that live in a heterogeneous environment and need traditional syslog. By making it optional, it will ultimately die, forcing journal as the only viable option in a Fedora environment. This is IMO not net beneficial for downstream use cases later on either.
Jon.
I've read though most of the email on this thread, and would just like to put in my $.02. I think this "System Wide Change" is 1 to 2 releases too early. I'm not totally against it. But I live with one foot in RHEL land, and one foot in Fedora. It takes me a while to incorporate these new features. I need more transition time.
Just taking it out of @core instead of both would ease the transition.
Troy
On 07/17/2013 09:04 PM, Troy Dawson wrote:
I've read though most of the email on this thread, and would just like to put in my $.02. I think this "System Wide Change" is 1 to 2 releases too early. I'm not totally against it. But I live with one foot in RHEL land, and one foot in Fedora. It takes me a while to incorporate these new features. I need more transition time.
Just taking it out of @core instead of both would ease the transition.
If you got your one food in RHEL land you should already be familiar with how to manually install package and add it to your ks file
JBG
On Mon, 15 Jul 2013 10:44:44 +0200 Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
Can systemd journal be picked up by logwatch?, or similarly be emailed on a daily basis, without user creating a bagful of scripts?
On 07/15/2013 11:22 AM, Frank Murphy wrote:
On Mon, 15 Jul 2013 10:44:44 +0200 Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
Can systemd journal be picked up by logwatch?, or similarly be emailed on a daily basis, without user creating a bagful of scripts?
No, logwatch works only with traditional text-based logs. There is this RFE filed [1], but that will probably not happen as it would require a total rewrite. It would make more sense to write a new logwatch-like utility from scratch, that handles only the journal. It's somewhere on my todo list, but has quite a low priority:(
[1] https://bugzilla.redhat.com/show_bug.cgi?id=864872
On Mon, 15.07.13 12:44, Jan Synacek (jsynacek@redhat.com) wrote:
On 07/15/2013 11:22 AM, Frank Murphy wrote:
On Mon, 15 Jul 2013 10:44:44 +0200 Jaroslav Reznik jreznik@redhat.com wrote:
= Proposed System Wide Change: No Default Syslog = https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
Change owner(s): Lennart Poettering <lennart at poettering net>, Matthew Miller <mattdm at fedoraproject org>
No longer install a traditional syslog service by default. (Specifically, remove rsyslog from the @core or @standard groups in comps.)
The systemd journal will be the default logging solution. Rsyslog, Syslog-NG, and even traditional sysklogd will continue to cover use cases outside of the default.
Can systemd journal be picked up by logwatch?, or similarly be emailed on a daily basis, without user creating a bagful of scripts?
No, logwatch works only with traditional text-based logs. There is this RFE filed [1], but that will probably not happen as it would require a total rewrite. It would make more sense to write a new logwatch-like utility from scratch, that handles only the journal. It's somewhere on my todo list, but has quite a low priority:(
Such a change should be relatively easy. "journalctl" provides you with an output that is pretty much identical to what /var/log/messages contains. Hence, you can replace this:
f = fopen("/var/log/messages", "r");
by this:
f = popen("journalctl", "r");
And everything should just continue to work, the stream you read from that is pretty much identical.
Using the structured journal contacts of course is also possible and desirable, but certainly more work. There are very good Python bindings for the journal, but I am not aware of any Perl bindings so far.
Lennart
On Mon, Jul 15, 2013 at 01:31:44PM +0200, Lennart Poettering wrote:
On Mon, 15.07.13 12:44, Jan Synacek (jsynacek@redhat.com) wrote:
On 07/15/2013 11:22 AM, Frank Murphy wrote:
No, logwatch works only with traditional text-based logs. There is this RFE filed [1], but that will probably not happen as it would require a total rewrite. It would make more sense to write a new logwatch-like utility from scratch, that handles only the journal. It's somewhere on my todo list, but has quite a low priority:(
Such a change should be relatively easy. "journalctl" provides you with an output that is pretty much identical to what /var/log/messages contains. Hence, you can replace this:
f = fopen("/var/log/messages", "r");
by this:
f = popen("journalctl", "r");
And everything should just continue to work, the stream you read from that is pretty much identical.
At least for logcheck this won't work, because needs to get access to only the log entries that changed since the last run. For this it uses logtail whick keeps track for the inode and line offset on each invocation. Therefore it seems like it would somehow like this:
previous_run = "$(cat last_run.txt)" current_run = "$(date --iso-8601=seconds --date '1 second ago' | head -c 19 | tr "T" " ") $log_data = $("journalctl --since $last_run --until $current_run) echo "$current_run" > last_run.txt
But there seems to be the problem that journalctl does not support timezones/UTC time for since/until according to journalctl(1), therefore there does not seem to be a safe way to specify since/until that will even work when the current local UTC offset changes (daylight saving time). Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
Regards Till
On Mon, 15.07.13 20:09, Till Maas (opensource@till.name) wrote:
On Mon, Jul 15, 2013 at 01:31:44PM +0200, Lennart Poettering wrote:
On Mon, 15.07.13 12:44, Jan Synacek (jsynacek@redhat.com) wrote:
On 07/15/2013 11:22 AM, Frank Murphy wrote:
No, logwatch works only with traditional text-based logs. There is this RFE filed [1], but that will probably not happen as it would require a total rewrite. It would make more sense to write a new logwatch-like utility from scratch, that handles only the journal. It's somewhere on my todo list, but has quite a low priority:(
Such a change should be relatively easy. "journalctl" provides you with an output that is pretty much identical to what /var/log/messages contains. Hence, you can replace this:
f = fopen("/var/log/messages", "r");
by this:
f = popen("journalctl", "r");
And everything should just continue to work, the stream you read from that is pretty much identical.
At least for logcheck this won't work, because needs to get access to only the log entries that changed since the last run. For this it uses logtail whick keeps track for the inode and line offset on each invocation. Therefore it seems like it would somehow like this:
previous_run = "$(cat last_run.txt)" current_run = "$(date --iso-8601=seconds --date '1 second ago' | head -c 19 | tr "T" " ") $log_data = $("journalctl --since $last_run --until $current_run) echo "$current_run" > last_run.txt
But there seems to be the problem that journalctl does not support timezones/UTC time for since/until according to journalctl(1), therefore there does not seem to be a safe way to specify since/until that will even work when the current local UTC offset changes (daylight saving time). Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
glibc is actually smart enough to determine DST from the specified time itself (of course, this might guess incorrectly since during the switch there might be times that happen 'twice'), so this should be a non-issue. journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC, in particular because the time zone names are not unique...
While it will be hard to support arbitrary timezone specs in --since= it should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
Note that the journal actually knows a concept called "cursors". The cursors are a way to refer to a specific log line (or the closest available one) in a stable way. by using this you can make a logic like the above work nicely, and even remove any inaccuracy regarding timestamps...
Lennart
On Mon, Jul 15, 2013 at 08:18:34PM +0200, Lennart Poettering wrote:
But there seems to be the problem that journalctl does not support timezones/UTC time for since/until according to journalctl(1), therefore there does not seem to be a safe way to specify since/until that will even work when the current local UTC offset changes (daylight saving time). Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
glibc is actually smart enough to determine DST from the specified time itself (of course, this might guess incorrectly since during the switch there might be times that happen 'twice'), so this should be a non-issue.
Why is it a non-issue? It exactly breaks above code when the same local time occurs twice during the switch. Here are corresponding times for two UTC offsets during the switch: 2013-10-27 1:30+0200 2013-10-27 0:30+0100 2013-10-27 2:25+0200 2013-10-27 1:25+0100 2013-10-27 3:20+0200 2013-10-27 2:20+0100 2013-10-27 4:15+0200 2013-10-27 3:15+0100
You will get these calls --since 1:30 --until 2:25 (UTC+2) normal report --since 2:25 --until 2:20 (UTC+1) no report --since 2:20 --until 3:15 (UTC+1) partly duplicate report, 1 hour delayed
journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC, in particular because the time zone names are not unique... While it will be hard to support arbitrary timezone specs in --since= it should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
You only need to add or subtract hours and minutes from the local time, because ISO 8601 dates contain the UTC offset:
| $ date --iso-8601=seconds | 2013-07-15T22:37:04+0200
Note that the journal actually knows a concept called "cursors". The cursors are a way to refer to a specific log line (or the closest available one) in a stable way. by using this you can make a logic like the above work nicely, and even remove any inaccuracy regarding timestamps...
The manpage only mentions how to specify which cursor to use but not how to get the cursor of the last read line.
Regards Till
On Tue, 16.07.13 09:42, Till Maas (opensource@till.name) wrote:
journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC, in particular because the time zone names are not unique... While it will be hard to support arbitrary timezone specs in --since= it should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
You only need to add or subtract hours and minutes from the local time, because ISO 8601 dates contain the UTC offset:
| $ date --iso-8601=seconds | 2013-07-15T22:37:04+0200
Well, we can certainly add support for such numeric timezone specs (added to the TODO list), but I have my doubts that this is actually what people want to use in their day-to-day use, given that the numbers are hard to remember.
I am pretty sure that most folks would like to specify symbolic timezone names, but that's hard to do due to lack of APIs for that, and the non-uniqueness of the names.
Note that the journal actually knows a concept called "cursors". The cursors are a way to refer to a specific log line (or the closest available one) in a stable way. by using this you can make a logic like the above work nicely, and even remove any inaccuracy regarding timestamps...
The manpage only mentions how to specify which cursor to use but not how to get the cursor of the last read line.
journalctl -o verbose will give you that (and so will -o json, -o export and others), but the rest of the output is very low-level then. Maybe we can add a switch that prints the final cursor as last line if you specify some switch.
If you use the C API you can easily query the cursor string for any line you like (see sd_journal_get_cursor(3) for details).
Lennart
On Tue, Jul 16, 2013 at 01:43:04PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 09:42, Till Maas (opensource@till.name) wrote:
journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC, in particular because the time zone names are not unique... While it will be hard to support arbitrary timezone specs in --since= it should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
You only need to add or subtract hours and minutes from the local time, because ISO 8601 dates contain the UTC offset:
| $ date --iso-8601=seconds | 2013-07-15T22:37:04+0200
Well, we can certainly add support for such numeric timezone specs (added to the TODO list), but I have my doubts that this is actually what people want to use in their day-to-day use, given that the numbers are hard to remember.
Thank you.
I am pretty sure that most folks would like to specify symbolic timezone names, but that's hard to do due to lack of APIs for that, and the non-uniqueness of the names.
I guess for most use cases using the local time zone is enough.
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
Note that the journal actually knows a concept called "cursors". The cursors are a way to refer to a specific log line (or the closest available one) in a stable way. by using this you can make a logic like the above work nicely, and even remove any inaccuracy regarding timestamps...
The manpage only mentions how to specify which cursor to use but not how to get the cursor of the last read line.
journalctl -o verbose will give you that (and so will -o json, -o export and others), but the rest of the output is very low-level then. Maybe we can add a switch that prints the final cursor as last line if you specify some switch.
The extra switch sounds useful.
Regards Till
On Tue, Jul 16, 2013 at 06:09:54PM +0200, Till Maas wrote:
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
That would be nice, ”journalctl” output would match rsyslog:
% tail -1 /var/log/messages 2013-07-16T17:57:27.903228+02:00 mother postfix/smtpd[3452]: DC6A1600B6: client=bastion01.fedoraproject.org[209.132.181.2]
journalctl -n 1 ?
On Tue, Jul 16, 2013 at 6:26 PM, Tomasz Torcz tomek@pipebreaker.pl wrote:
On Tue, Jul 16, 2013 at 06:09:54PM +0200, Till Maas wrote:
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
That would be nice, ”journalctl” output would match rsyslog:
% tail -1 /var/log/messages 2013-07-16T17:57:27.903228+02:00 mother postfix/smtpd[3452]: DC6A1600B6: client=bastion01.fedoraproject.org[209.132.181.2]
-- Tomasz Torcz "God, root, what's the difference?" xmpp: zdzichubg@chrome.pl "God is more forgiving."
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, Jul 16, 2013 at 06:33:23PM +0200, drago01 wrote:
On Tue, Jul 16, 2013 at 6:26 PM, Tomasz Torcz tomek@pipebreaker.pl wrote:
On Tue, Jul 16, 2013 at 06:09:54PM +0200, Till Maas wrote:
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
That would be nice, ”journalctl” output would match rsyslog:
% tail -1 /var/log/messages 2013-07-16T17:57:27.903228+02:00 mother postfix/smtpd[3452]: DC6A1600B6: client=bastion01.fedoraproject.org[209.132.181.2]
journalctl -n 1 ?
No, it's the timestamp which is imporant; speaking of time, did anyone mention that journal is quite slow on rotating HDDs?
% time journalctl -u postfix -n 1 -- Logs begin at Sat 2012-12-08 19:45:43 CET, end at Tue 2013-07-16 18:34:45 CEST. -- Jul 16 18:34:45 mother.pipebreaker.pl postfix/qmgr[1675]: 0F945601C0: removed
real 0m24.873s user 0m0.008s sys 0m0.201s
I know that SSD are current storage technology, but there are some people with HDDs left, and they will complain. They already complain “systemctl status foo” takes half a minute to pull logs back.
(above ”time journalctl …” comes from system with Intel Sandy Bridge CPU, ext4 over dm-crypt over 4xSATA 7200 mdraid).
On Tue, 16.07.13 18:26, Tomasz Torcz (tomek@pipebreaker.pl) wrote:
On Tue, Jul 16, 2013 at 06:09:54PM +0200, Till Maas wrote:
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
That would be nice, ”journalctl” output would match rsyslog:
% tail -1 /var/log/messages 2013-07-16T17:57:27.903228+02:00 mother postfix/smtpd[3452]: DC6A1600B6: client=bastion01.fedoraproject.org[209.132.181.2]
That's not the default output we ship, is it? Or did rsyslog defaults change recently? Classic /var/log/messages output does not include ISO dates.
Lennart
On Tue, Jul 16, 2013 at 06:39:53PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 18:26, Tomasz Torcz (tomek@pipebreaker.pl) wrote:
On Tue, Jul 16, 2013 at 06:09:54PM +0200, Till Maas wrote:
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
That would be nice, ”journalctl” output would match rsyslog:
% tail -1 /var/log/messages 2013-07-16T17:57:27.903228+02:00 mother postfix/smtpd[3452]: DC6A1600B6: client=bastion01.fedoraproject.org[209.132.181.2]
That's not the default output we ship, is it? Or did rsyslog defaults change recently? Classic /var/log/messages output does not include ISO dates.
That's after adding one ”#” in rsyslog.conf we ship. I don't know how ”classic” matches latest Syslog RFC document in case of timestamp.
On Tue, 16.07.13 18:09, Till Maas (opensource@till.name) wrote:
On Tue, Jul 16, 2013 at 01:43:04PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 09:42, Till Maas (opensource@till.name) wrote:
journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC, in particular because the time zone names are not unique... While it will be hard to support arbitrary timezone specs in --since= it should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
You only need to add or subtract hours and minutes from the local time, because ISO 8601 dates contain the UTC offset:
| $ date --iso-8601=seconds | 2013-07-15T22:37:04+0200
Well, we can certainly add support for such numeric timezone specs (added to the TODO list), but I have my doubts that this is actually what people want to use in their day-to-day use, given that the numbers are hard to remember.
Thank you.
I am pretty sure that most folks would like to specify symbolic timezone names, but that's hard to do due to lack of APIs for that, and the non-uniqueness of the names.
I guess for most use cases using the local time zone is enough.
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
In the default output we stay true to the formatting that has been used in /var/log/messages since always.
We currently do not use the ISO date format anywhere, it's not really readable, I think. Great way to serialize dates, recurring and duration events to ascii strings for computers to read, but not really for humans.
Note that in all other places we tend to use date format like this: "Tue 2013-07-16 18:41:57 CEST" Which is close to ISO, but not ISO.
Lennart
Le Mar 16 juillet 2013 18:42, Lennart Poettering a écrit :
On Tue, 16.07.13 18:09, Till Maas (opensource@till.name) wrote:
On Tue, Jul 16, 2013 at 01:43:04PM +0200, Lennart Poettering wrote:
On Tue, 16.07.13 09:42, Till Maas (opensource@till.name) wrote:
journalctl only supports local time specifications when you specify calendar times. Unfortunately there's no nice API to map calendar times that include time zone specifications back to UTC,
in
particular because the time zone names are not unique... While it will be hard to support arbitrary timezone specs in
--since= it
should be relatively easy to support UTC in addition to the local timezone. Added this to the TODO list.
You only need to add or subtract hours and minutes from the local
time,
because ISO 8601 dates contain the UTC offset:
| $ date --iso-8601=seconds | 2013-07-15T22:37:04+0200
Well, we can certainly add support for such numeric timezone specs (added to the TODO list), but I have my doubts that this is actually what people want to use in their day-to-day use, given that the
numbers
are hard to remember.
Thank you.
I am pretty sure that most folks would like to specify symbolic
timezone
names, but that's hard to do due to lack of APIs for that, and the non-uniqueness of the names.
I guess for most use cases using the local time zone is enough.
Btw. can journalctl output ISO 8601 dates instead of the US formated date without a year? I really expected journalctl to cleanup this as well.
In the default output we stay true to the formatting that has been used in /var/log/messages since always.
We currently do not use the ISO date format anywhere, it's not really readable, I think.
It's not really readable because you're not used to seeing it. You're not used to seeing it because you (and others) have invented custom alternatives. Seems a self-inflicted problem to me (just like most of the "UTF-8 is too complex, it's simpler to use _pet-endoding_" were)
Great way to serialize dates, recurring and duration events to ascii strings for computers to read, but not really for humans.
Note that in all other places we tend to use date format like this: "Tue 2013-07-16 18:41:57 CEST" Which is close to ISO, but not ISO.
While replacing T with space is common (even though it will break field tokenization in many apps) Tue is a pure Englicism (useless in most parts of the world) and CEST will break interoperability (since people love to invent new names for time zones. That's why the ISO numeric adjustment is way saner).
Also "tend to" means "I'm not consistent and other apps and people can not rely on any specific convention when dealing with me"
On Thu, 18.07.13 14:56, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
In the default output we stay true to the formatting that has been used in /var/log/messages since always.
We currently do not use the ISO date format anywhere, it's not really readable, I think.
It's not really readable because you're not used to seeing it. You're not used to seeing it because you (and others) have invented custom alternatives. Seems a self-inflicted problem to me (just like most of the "UTF-8 is too complex, it's simpler to use _pet-endoding_" were)
No, it's not readable, because it's just not readable. It doesn't include whitespace as separators, which could help people to read the string. It also doesn't include day of week information which is highly interesting to most people.
But anyway, I understand you like ISO, and think it is readable. I don't agree, but we just have to agree to disagree on this one. It might thrill you though to learn that I just commited a patch by Tomasz that adds an ISO output mode to journalctl. "journalctl -o short-iso" is your friend.
And the nice thing: you can actually switch forth and back with this at display time! Instead of choosing your date format before the log files are written, you can actually choose at display time! How awesome is that?
Great way to serialize dates, recurring and duration events to ascii strings for computers to read, but not really for humans.
Note that in all other places we tend to use date format like this: "Tue 2013-07-16 18:41:57 CEST" Which is close to ISO, but not ISO.
While replacing T with space is common (even though it will break field tokenization in many apps) Tue is a pure Englicism (useless in most parts of the world) and CEST will break interoperability (since people love to invent new names for time zones. That's why the ISO numeric adjustment is way saner).
The "Tue" is actually in your local language based on system locale. Hence for you "mar." or so.
Also "tend to" means "I'm not consistent and other apps and people can not rely on any specific convention when dealing with me"
Well, if you think we are generally confused and unreliable folks, then go ahead and think that.
Lennart
Le Jeu 18 juillet 2013 16:08, Lennart Poettering a écrit :
But anyway, I understand you like ISO, and think it is readable. I don't agree, but we just have to agree to disagree on this one. It might thrill you though to learn that I just commited a patch by Tomasz that adds an ISO output mode to journalctl. "journalctl -o short-iso" is your friend.
Thank you very much (and Tomasz of course), that's awesome turnaround.
Regards,
Le Lun 15 juillet 2013 20:09, Till Maas a écrit :
Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
+100 not using iso 8601 by default nowadays is insane, I've been slowly moving all the bits I could to iso 8601 in the past years and it's the same breath of fresh air as UTF-8 was compared to the legacy all-incompatible-with-each-other 8 bit encodings
(and I won't write here my opinion of the glibc maintainers that have persistently refused to make it easy to select iso-8601 dates by default over existing locales. See also “English in Denmark” — which besides being ridiculous only helps English speakers)
On Tue, 16.07.13 11:49, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 20:09, Till Maas a écrit :
Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
+100 not using iso 8601 by default nowadays is insane, I've been slowly
Thanks for calling me insane.
moving all the bits I could to iso 8601 in the past years and it's the same breath of fresh air as UTF-8 was compared to the legacy all-incompatible-with-each-other 8 bit encodings
We looked into using ISO dates, especially for denoting repetition events, but quite frankly they are just not simple and intuitive to write. Expressions such as "P1Y2M10DT2H30M/2008-05-11T15:30:00Z" are not nice to write, nor even to read.
I think the ISO dates are great for serializing timestamp and calendar events to ASCII strings for being parsed again by computers, but they just suck as user interface.
The reason it took so long for systemd to gain support for calendar events was primarily because this question is so messy. We wanted to stick to a standardised language for date, but quite frankly there was none that was convincing as user interface.
You might have noticed that most graphical programs (such as email programs) that want to show you a time will not do so in the ISO format either, but in a more local, user friendly presentation. That's because the ISO format is just not suitable for user presentation.
Lennart
Le Mar 16 juillet 2013 13:54, Lennart Poettering a écrit :
On Tue, 16.07.13 11:49, Nicolas Mailhot (nicolas.mailhot@laposte.net) wrote:
Le Lun 15 juillet 2013 20:09, Till Maas a écrit :
Also it is sad that journalctl does not directly accept ISO 8601 time specifications (I can open a bug if there is a changes it will be implemented).
+100 not using iso 8601 by default nowadays is insane, I've been slowly
Thanks for calling me insane.
I did not call anyone insane, I called a use insane, this is not the same thing.
moving all the bits I could to iso 8601 in the past years and it's the same breath of fresh air as UTF-8 was compared to the legacy all-incompatible-with-each-other 8 bit encodings
We looked into using ISO dates, especially for denoting repetition events, but quite frankly they are just not simple and intuitive to write. Expressions such as "P1Y2M10DT2H30M/2008-05-11T15:30:00Z" are not nice to write, nor even to read.
Even though some parts of ISO 8601 may seem less natural than others, every time I had to dig a particular datetime problem it turned out the standard was actually well thought and the "better" custom alternatives all had nasty failure points (the biggest being plugging two apps that used a custom alternative always resulted in bugs that always appears at period limits no one tested completely for).
While I don't suppose I'll be able to convince you not to try to invent something better, please at least make it possible for those of us who have been burned by those experiments to ignore yours and be able to work with systemd in iso 8601 mode.
You might have noticed that most graphical programs (such as email programs) that want to show you a time will not do so in the ISO format either, but in a more local, user friendly presentation. That's because the ISO format is just not suitable for user presentation.
The problem of those "user friendly" presentations is that not two of them are similar, so as soon as you're unlucky enough not to use an en_US locale (that they all fallback too) you'll get presentation discrepancies or even outright incompatibilities, bugs, or even indefinite behaviour when the software writer assumed you knew his pet format and didn't bother making explicit what his conventions were.
I'll take iso 8601 standard over this user "friendliness" any time, thank you very much