On Tue, Jan 05, 2021 at 01:05:01PM -0500, Ben Cotton wrote: This is for _every file_ in every RPM? Or some files in some RPMs? -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader

On 1/7/21 10:41 AM, Panu Matilainen wrote: Isn't there a cost in the fileystem too? Adding more and more additional file attributes increases space overhead and pushes people to ugly "DB in a file" "packs", while we know that small files is the right way to do many things (e.g. "*.d" directories for confs). Can't this stuff be moved out of the main rpm and used only by whoever cares about signatures? Like debuginfo has been addressed (for sure an immensely more useful thing to have). Regards. -- Roberto Ragusa mail at robertoragusa.it

On Tue, Jan 05, 2021 at 01:38:48PM -0500, Neal Gompa wrote: Why? It gets us nothing really... adds complexity and issues. We would definiltey need to improve dnf's handling of signed repos before we did at least. kevin

On Tue, Jan 5, 2021 at 6:59 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: Well that's one of the things we're aiming to do here is to improve the tools and the process and the documentation.

On Tue, Jan 5, 2021 at 6:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: This is completely unrelated please don't try and derail the topic.

* Ben Cotton: What is the impact on RPM database size? Will GPLv3 packages be excluded, or will the signing keys be provided upon request? Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

On Tue, Jan 5, 2021 at 6:41 PM Florian Weimer <fweimer(a)redhat.com&gt; wrote: They're stored in xattr so it shouldn't have any noticeable impact, although Patrick can confirm the details of that. The public key?

On Tue, Jan 5, 2021, at 3:19 PM, Florian Weimer wrote: This is a complex and highly nuanced topic because IMA is both a mechanism and a set of potential *policies* that one can use, and a whole lot depends on the exact policy in use. Like SELinux in that it's a highly flexible thing; and related to that because IMA isn't the default in any mainstream distribution (that I can think of) it ends up being a "secondary" mechanism. There have definitely been multiple cases in the past where SELinux policy stops a chain of compromise - a great example is https://seclists.org/oss-sec/2019/q1/119 (The ostree read-only bind mount over /usr also stopped it, a great example of defense-in-depth) And one can definitely construct scenarios where an enforced IMA policy like `ima_tcb` would stop a similar chain - in the case of that runc flaw the attacker wouldn't have access to the signing key and so couldn't write a new binary that could be executed by root. But an ima_tcb policy would also e.g. break tooling like Ansible which is all about copying dynamic Python code from a remote host and executing it as root. And really that's a root problem in all security systems like this: it's just really hard to usefully distinguish between "code injected by a compromised privileged process" and "change the admin wants to make to their own computer". I think the Change authors here trying to make it easier to enable IMA without the really awful hack of "boot up your installed system and run these shell scripts to sign", which is a laudable goal. Having pre-signed OS binaries would definitely help, but...in any kind of general case users are going to want to run code *not* from the distribution, so the tooling and docs need to generalize to user-owned keys. Also there's the inverse problem in that a lot of people will want to use IMA as a form of "application whitelisting" but there's...a lot of RPMs in Fedora. Less of a concern for RHEL at least. Anyways again I think a much more flexible user story is: "As a Fedora IoT user I want to provide a key to use for IMA signatures of the operating system (rpm-ostree) (and potentially container images, etc.) and have the first-boot process handle IMA signing for me and automatically apply this across upgrades too." The key would e.g. be stored in a TPM on device. A simple way to think of this is as a parallel with TPM-bound-LUKS (clevis) that we landed in Ignition - make it as easy as possible for admins to enable. Having pre-signed binaries isn't strictly necessary for that. The generalization of course of this is the model mentioned in the rpm-ostree issue where rather than doing signatures per-device you perform centralized image builds and seal /etc too, but as noted it's a rather large change in systems management (notable security benefits, breaks a lot of management tooling and greatly hampers one's ability to go in and test a change on a system).

Hi, I did some experimentation with the size differences of both the binary RPM packages and the RPMDB if the packages would contain IMA signatures. I took all the packages in the Fedora base container image (registry.fedoraproject.org/fedora:latest), removed the Fedora signatures, and then got two copies of that: one with just GPG signed, one with IMA signatures and then GPG signed. The size of the binary RPMs: 77412 ima-signed-pkgs 76632 rawpkgs 75660 signedpkgs So on the binary RPMs, I am seeing a 2% increase. After that, I created two containers, one where I reinstalled all the packages with the resigned versions and one where I reinstalled all the packages with the IMA resigned versions, and then looked at the RPMDB sizes: Resigned: 14732 /var/lib/rpm IMA + Resigned: 16332 /var/lib/rpm So in the RPMDB, this comes down to a 11% increase. However, to be noted here is that this is 15 or 16MB on a 584MB base image, so it's about a 0.3% increase of size compared to the base image itself. Note that if you do not install the "rpm-plugin-ima", that is the only impact: RPM will only actually put the extended attributes in place once you install that package. For more notes on how I got these results, see my notes at https://puiterwijk.fedorapeople.org/ima-size-research . The default IMA policy in the kernel measures and attests nothing. You would need to manually deploy an IMA policy with a line like "audit uid=0" for any file signature verification to be performed (in this case, on files opened as root). After you have enabled policy lines like these, it's impossible to remove them without rebooting, but the system starts without a policy. So you would need to manually deploy the policy, after which systemd loads it as one of the very first steps of the boot process and it takes effect. Regards, Patrick

On Tue, 2021-01-05 at 13:05 -0500, Ben Cotton wrote: Is there any relation between this and fapolicyd, that seems to be developed mostly by Red Hat employees? https://github.com/linux-application-whitelisting/fapolicyd -- Michel Alexandre Salim profile: https://keyoxide.org/michel@michel-slm.name chat via email: https://delta.chat/ GPG key: 5DCE 2E7E 9C3B 1CFF D335 C1D7 8B22 9D2F 7CCC 04F2

Ben Cotton wrote: I am opposed to this Change, because it increases the file size of all RPMs and the size of the RPM database (and hence, of all installed systems, including, but not limited to, the live images) to implement what basically amounts to "Treacherous Computing" [ https://www.gnu.org/philosophy/can-you-trust.en.html ]. Neither do I consider it acceptable to ban execution of non-centrally-signed binaries, nor do I consider it acceptable to bloat all our packages with per-file signatures that are ultimately redundant with the package signatures that already guarantee the integrity of all files in the package. Kevin Kofler

Zbigniew Jędrzejewski-Szmek wrote: As I understand it, yes. Huge, see Panu Matilainen's comment in this thread. If it is locally centrally managed, that means people are using their own signatures and don't need Fedora to put them into the RPMs. I guess it is to comply with some standard that absolutely needs per-file signatures. rpm -V can already verify the integrity of each file by checking the file's cryptographic checksum that is signed (as a part of the package contents) with the package signature. Kevin Kofler

On Tue, Jan 05, 2021 at 01:05:01PM -0500, Ben Cotton wrote: Who is going to use this feature? My guess is a very limited set of users, so it seems unfair to dramatically increase the size of their downloads and install footprint to support something they don't use. Can't they be shipped on the side? An rpm of signatures that's optionally installed would be more user friendly. Also, I (being unfamiliar with IMA), don't see how this is any better than trusting the file hash signed by the fedora keys that we currently have. Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart

On Tue, Jan 12, 2021 at 8:21 PM Brian C. Lane <bcl(a)redhat.com&gt; wrote: I wasn't aware the current rpm has functionality integrated with kernel/tpm and other functionality to do it at runtime access of the files, I thought it was more an audit style use case such as rpm -V to see if something had changed. IMA is more about being able to check at runtime and setting policies of what to do if something has changed.

On Di, 12.01.21 12:20, Brian C. Lane (bcl(a)redhat.com) wrote: So, my question would be similar to the above: what is the attack scenario you (as in feature submitters) want to address with this? I mean, this is a security mechanism afaiu, so you must have thought about an attack scenario. What is it? Can you elaborate on it? The feature page is irritatingly silent about it, but it's really what you should start with. Moreover, it appears that there was not much time spent in finding alternatives to what you want to do and figuring out if they wouldn't solve your problem better. Now, given that you never state the problem, I can only guess that what you want to attack is the problem of "offline security", i.e. you want that changes made offline to your file system are recognized on next boot? If offline security is what you are striving for, one would have expected that you list at least dm-verity and dm-integrity in your "Why not…?" section in the feature page. Now, my guess is that dm-verity is probably not interesting to you, since you want to keep RPM working, i.e. the immutability property of dm-verity is not a feature for you but a disqualifying problem. That however leaves dm-integrity (used with a HMAC hash function), so that local changes to the file systems can be made, but offline ones cannot – as long as the secret key for the HMAC function is known to the local system. Much like IMA dm-integrity/HMAC stuff exists already in the kernel too, so why IMA instead of such an approach? To me IMA appears quite a bogus concept tbh, and in particular deploying it without a policy that actually makes use of this sounds entirely pointless to me. Design your features to the end, implement them bit by bit — that would be an OK approach, but your appraoch appears to be, design one step without thinking ahead, in the vague hope it can be useful later? What I'd criticize about IMA is that it is inherently per-file. It's not clear to me what it does to protect the combination of files, or that even the metadata is properly secured too, for example the file system path used to find a file. i.e. if /usr/bin/true is fine to invoke, is /usr/foo/true too, if i copy the file there? And even more, if I copy the file to /usr/bin/false, is it still OK to execute then? If so, how is this stuff useful at all? what kind of offline, malicious changes do you intend to detect with this and refuse, and which kind of changes are you willing to accept? There's also the problem that Linux file system implementations are generally assumed not to be safe against rogue file system images, i.e. the kernel devs make clear that they do not give guarantees and do not systematically test their implementations against rogue file system images, and that those cannot exploit your kernel. Any security tech on top of a file system leaves the kernel vulnerable to that. (The dm-integrity+HMAC approach I outline above, does protect you against this btw). Also, what about TPM? It is my understanding that the IMA policy needs to be protected by TPM or such to be useful. Do you intend to work on that too? If not, what's the point? I could just change the IMA policy and you won't notice... Then, what about containers? They are kinda popular these days: if I untar them onto one of such systems, what happens to IMA? Is there any protection? Or is it hunting season for anyone? THen, when I tar up an image generated with these IMA xattrs, and untar them elsewhere, what happens? Do they still make sense there? What about performance? If I understand correct IMA calculates a hash some over the whole file and stores that away, is that correct? If you have a huge binary and you validate on start you thus have this huge latency first, to read the whole thing into memory and verify it? This sounds like it could be pretty awful, in particular on simpler, embedded machines. i.e. think firefox big binaries, or anything go or rust compiled where they prefer statically linking the whole world. (dm-verity + dm-integrity don't suffer by this: they have hashes in smaller data blocks, and thus only have to verify shorter blocks when they are actually accessed) To me, this appears be not thought to the end. This appears to be a project for its own sake, not because it would deliver anything truly useful. Hence, let me ask you: what is the attack scenario for this? Given that you do not intend to ship a policy by default, what are your next steps on all of this? Seriously puzzled about this feature, Lennart -- Lennart Poettering, Berlin

On 21. 01. 21 0:29, Patrick マルタインアンドレアス Uiterwijk wrote: Thanks! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

On Wed, Jan 20, 2021 at 11:29:55PM -0000, Patrick マルタインアンドレアス Uiterwijk wrote: Maybe worth sending to devel-announce again? -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader

On Thu, Jan 21, 2021 at 10:50:52AM +0100, Kevin Kofler via devel wrote: Quoting size of the RPMDB in isolation is a misleading way to describe the real world disk space impact. The RPMDB is only responsible for a small % of the overall install footprint. This is clear in the docs which describe how the 20% RPMDB size increase, amounts to a mere 0.3% increase in size of the overall VM disk image. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Jan 21, 2021 at 9:52 AM Kevin Kofler via devel < devel(a)lists.fedoraproject.org&gt; wrote: Keylime cannot be used to enforce anything, it will only monitor and there are already many tools that already do the same thing (aide, rpm -v etc). There are zero GPLv3 violations here. Any violation would need to be enacted outside of the project.

On Tue, Jan 26, 2021 at 1:45 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote: Given the schedule, it seems like this should be retargeted for F35. I'm not sure if your list has that assumption in mind. I'm talking to the RHEL maintainers about getting this into RHEL 8. I don't want to make promises on their behalf, but I think we'll be okay in that regard. You're right that RHEL 7 seems very unlikely. If we consider it very critical, I can have that conversation, too. -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis

On Tue, Jan 26, 2021 at 10:44:05AM -0800, Kevin Fenzi wrote: Is this an immediate issue given that it only affects systems where the plugin is enabled? Very unlikely with where RHEL 7 is in its maintenance lifecycle. -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader

On Tue, Jan 26, 2021 at 03:46:49PM -0500, Neal Gompa wrote: "This" is referring to the point I'm quoting about the signatures taking up "tons of room". -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader

On Tue, Jan 26, 2021, 3:47 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: Very unlikely in RHEL 7. Take that to mean that you cannot count on it and if it happens it's a bonus. 8 is doable. The concern over impacts to older RHEL releases is appreciated for sure, but I am not sure that's a deal breaker for this feature. I'd rather see it go through Fedora to get exactly the kind of feedback that is being given here rather than have this show up in a future RHEL release first. Let's try to avoid doing things backwards if we can. josh

On Wed, Jan 27, 2021 at 2:14 AM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote: The openSUSE Build Service runs on SUSE Linux Enterprise 15, which is afflicted in the same way that RHEL 8 is, since they have the same version of RPM. -- 真実はいつも一つ！/ Always, there's only one truth!

On 1/26/21 8:44 PM, Kevin Fenzi wrote: That'd be rpm upstream work. On my F33 laptop, there are 331284 rpm-installed files. The IMA signature as proposed is apparently 162 bytes per file in the hex-encoded format, this makes for approximately 51 megabytes of data. My rpmdb is about 115 megabytes. That'd be almost 45% increase in size! And this would be on EVERYBODY's database whether you use the feature or not, also slowing down every single rpm query somewhat as a whole lot more data has to be pulled from disk, and there's no way to get rid of the weight once its there. The height of the insult is that the data is essentially useless in the rpmdb, it's only relevant during installation, for the (presumably few) people who actually enable the feature. And of course that extra weight in every single package is carried around in mirrors and each and every package download too, again whether you use the feature or not. What the IMA feature really needs is a redesign to avoid inflicting this cost on everybody whether you use the feature or not, but the low-hanging fruit is the encoding: the hex encoding is just about the most stupid format there is for such a purpose, when base64 encoded the same data is ~38% of the size of the hex encoding, which brings down the IMA data size in the above figures to ~19 megabytes and ~17% increase in rpmdb size, which is a lot of data still but a lot less anyhow. The existing hex encoded format needs to continue to be supported both on the install and signing side as it has been in several releases now, but there's no reason both can't be supported. Patches welcome. - Panu -

On Wed, Jan 27, 2021 at 10:48:46AM +0200, Panu Matilainen wrote: SO, I don't really understand... Patrick says in the Change: "The size of the rpmdb increases from 22952 to 28416 bytes, a 20% increase. This is on an install size of 1.7GB in total, so this 5MB increase is a 0.3% size increase on the final installed system." Is that just because he used the server install with fewer files? Or is your or his math wrong here? kevin

On 1/27/21 8:30 PM, Kevin Fenzi wrote: As directories are not signed, in a smaller installation the directory vs file ratio could be different, making the overhead smaller than in a larger install. Looking at the F33 server edition install, there are 59246 files total, 22837 of which are directories. That's a very different ratio to my laptop install where there are 402158 files total of which 70874 are directories. So it's a case of "it depends" and it depends quite a lot. By no means the overhead is always 45% but neither is it always 20% - depending on the exact package set it can be even be quite a bit more or less than either figure. You're free to check the math [1]. As said above, the size of an IMA signature is 162 bytes per file, which you can also see for yourself by looking at a signed package. Multiply that by the number of files installed. The real overhead from the string array structure is more than that, but just the signature data is 162 bytes per non-directory file entry. Whether the database file size increases by that exact amount depends as a database can preallocate space etc, but there literally is that much more data to store, and otherwise haul around even on unrelated queries. [1] I'm known to get it wrong on occasion, https://github.com/rpm-software-management/rpm/pull/1252 - Panu -

On 1/28/21 12:21 PM, Roberto Ragusa wrote: That'd be the total number of file entries, including directories. To exclude directories, try rpm -qalv|grep -v ^d|wc -l For statistics, you might want to compact the db first: rpmdb --rebuilddb Although that clearly is one *big* installation so dunno how much air there might be (or not) in that number. - Panu -

Panu Matilainen wrote: IMHO, this overhead is entirely unacceptable. Even using base64 would still be too expensive. This Change should just be permanently rejected (not just for F34 as it already was). I disagree that centrally signed individual files are a desirable feature at all. It is already clear that the vast majority of users will have no use for this feature and will not have it enabled. Hence, I do not see why we should be paying for it with any kind of overhead. Not even if it were only the overhead of infrastructure having to sign all those files and mirrors having to carry an external database. Kevin Kofler

On Thu, Jan 21, 2021 at 11:25:30AM +0100, Roberto Ragusa wrote: The signatures are stored in file extended attributes. Some file systems inline short extended attributes into i-node data structure. It could happen then the signature was placed into alignment gaps of the inode table resulting into no increase of the occupied blocks. But a simpler explanation could be that the used file system forgets to count extended attributes into stat.st_blocks value. We really need to know the details. -- Petr

On Thu, Jan 21, 2021 at 03:16:47PM -0800, Kevin Fenzi wrote: Oh! I hadn't caught that in the original description (and it's much more clear now in the revised change proposal). That very much lessens the impact of this change! -- Matthew Miller <mattdm(a)fedoraproject.org&gt; Fedora Project Leader

* Patrick マルタインアンドレアス Uiterwijk: I still have questions. Is it possible to enrol more than one signing key? Do the signatures cover path and package names. If they do, how do they work with hard links? How does the system ensure that no unwanted package has been installed? How does this system handle revocation? Can we mark individual builds as untrusted, or do we have to perform a key rollover and resign the release? Apart from the free software issue, my main concern is that the trust model is so unclear. Just because something has been signed by Fedora (because it went into some compose) doesn't been it will result in a secure system if it's installed as /bin/bash. The trust model appears to assume that. Is there a documentation of the trust model? Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

On Wed, Jan 20, 2021 at 11:29:55PM -0000, Patrick マルタインアンドレアス Uiterwijk wrote: Thanks. Typo in "uid=1000"? Also, executed, or read? What is "appraise_type=imasig"? I'm asking all those basic questions because the project documentation under https://sourceforge.net/p/linux-ima/wiki/Home/ is incredibly bad. It seems pretty clear nobody loves it, it's full of TLAs and formatting errors (e.g. every use of a <placeholder> is accompanied by </placeholder> at the end of the paragraph, sic). It also doesn't explain much except the obvious parts. It's also full of stuff like "'make modules_install install' to install the kernel", which I don't think we want to point our users to. (BTW the lwn article [1] linked below is *way* more approachable the the "documentation".) Is there any actual example of policy that makes use of this on a Fedora-type system? I'm asking because IMA has been around for for 15 years [1], and I think it's fair to say that it hasn't exactly taken the Linux world over by storm. I can see how IMA can be used to attest binaries in a custom-purpose system with a fixed and small set of binaries, but it seems much harder in a general purpose system with thousands of config files not distributed as rpm contents, user scripts, generated unit files, hwdb, etc. The docs that are available online seem to be toy examples only. Such an example is not a requirement, we may enable this based on just a hope that some real use will be found in the future, but it would certainly help to have such an example when evaluating this. [1] https://lwn.net/Articles/137306/ Either the measurement doesn't take xattrs into account, or maybe the explanation is that the attributes fit in preallocated space for xattrs? Either way, it seems that this cost is small enough and is not a barrier to adopting this. What about runtime costs? Is the installation with rpm-plugin-ima slower than without? Broken link. Zbyszek