On Mon, 11 Jan 2016 15:46:27 +0100 Jan Kurik <jkurik(a)redhat.com&gt; wrote: > = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = > https://fedoraproject.org/wiki/Changes/NewRpmDBFormat > > Change owner(s): > * Florian Festi < ffesti AT redhat DOT com > > > > Change format of the RPM Database from Berkeley DB to RPM's own > format. > > == Detailed Description == > The current implementation of the RPM Database is based on Berkeley > DB. There are doubts about the its future and level of maintenance. In > addition rpm's use of the database has multiple issues on its own. As > a result RPM upstream is working to replace the database format with a > new implementation. does it mean rpm is changing from Berkeley DB library to another library or to a completely new implementation of a database engine?

On 13.1.2016 13:48, Florian Festi wrote: > On 01/11/2016 03:57 PM, Dan Horák wrote: >> On Mon, 11 Jan 2016 15:46:27 +0100 >> Jan Kurik <jkurik(a)redhat.com&gt; wrote: >> >>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = >>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat >>> >>> Change owner(s): >>> * Florian Festi < ffesti AT redhat DOT com > >>> >>> >>> Change format of the RPM Database from Berkeley DB to RPM's own >>> format. >>> >>> == Detailed Description == >>> The current implementation of the RPM Database is based on Berkeley >>> DB. There are doubts about the its future and level of maintenance. In >>> addition rpm's use of the database has multiple issues on its own. As >>> a result RPM upstream is working to replace the database format with a >>> new implementation. >> >> does it mean rpm is changing from Berkeley DB library to another >> library or to a completely new implementation of a database engine? > > We change to our own format. One of the problem is the bit special > requirements of rpm where you want to have non privileged readers that > must not have any write access - which is required for most databases > for locking. I'm curious! Would it be possible to elaborate on reasons why no existing DB was good enough for RPM? https://bugzilla.redhat.com/show_bug.cgi?id=1086784#c8 indicates that this locking requirement is not a problem/obstacle for using LMDB.

On 01/14/2016 01:08 PM, Richard W.M. Jones wrote: > Yup - I'm curious about this as well. Using sqlite [for example] > would solve the libguestfs issues I outlined in this thread, as well > as using a format which is robust and proven rather than some > half-baked homebrew thing. Well, we actually deleted a sqlite backend a few years ago. But even with a more relational DB layout performance is about 2 times worse than the BDB.

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = https://fedoraproject.org/wiki/Changes/NewRpmDBFormat Change owner(s): * Florian Festi < ffesti AT redhat DOT com > Change format of the RPM Database from Berkeley DB to RPM's own format. == Detailed Description == The current implementation of the RPM Database is based on Berkeley DB. There are doubts about the its future and level of maintenance. In addition rpm's use of the database has multiple issues on its own. As a result RPM upstream is working to replace the database format with a new implementation.

Am 11.01.2016 um 17:26 schrieb Kalev Lember: > > On 01/11/2016 03:46 PM, Jan Kurik wrote: >> >> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = >> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat >> >> Change owner(s): >> * Florian Festi < ffesti AT redhat DOT com > >> >> >> Change format of the RPM Database from Berkeley DB to RPM's own format. >> >> == Detailed Description == >> The current implementation of the RPM Database is based on Berkeley >> DB. There are doubts about the its future and level of maintenance. In >> addition rpm's use of the database has multiple issues on its own. As >> a result RPM upstream is working to replace the database format with a >> new implementation. > > > Is the new database going to be able to support yumdb use cases as well? > Might be a good time to get rid of separate rpmdb and yumdb and merge > them together. please don't do that the yumdb is bloat because it contains a endless history and that's growing on machines after 10, 20 or more dist-upgrades while currently it's easy to get rid of that bload by just rm -rf /var/lib/yum/* and/or rm -rf /var/lib/dnf/*

On 01/11/2016 03:46 PM, Jan Kurik wrote: > = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = > https://fedoraproject.org/wiki/Changes/NewRpmDBFormat > > Change owner(s): > * Florian Festi < ffesti AT redhat DOT com > > > > Change format of the RPM Database from Berkeley DB to RPM's own format. > > == Detailed Description == > The current implementation of the RPM Database is based on Berkeley > DB. There are doubts about the its future and level of maintenance. In > addition rpm's use of the database has multiple issues on its own. As > a result RPM upstream is working to replace the database format with a > new implementation. Is the new database going to be able to support yumdb use cases as well? Might be a good time to get rid of separate rpmdb and yumdb and merge them together.

On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote: > = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = > https://fedoraproject.org/wiki/Changes/NewRpmDBFormat Details of the format? What forward and backward compatibility guarantees are there?

libguestfs reads the rpmdb of guests (of any version, new or old) in order to get the list of packages installed in a guest. Remember that the host and guest may be very different versions, eg. a RHEL 9 host accessing a Fedora 24 guest. Or vice versa.

Currently we use the (BDB-specific obviously) db_dump tool for this purpose. Other distros such as Debian as much simpler in this regard since they expose the package data as plain text files. https://github.com/libguestfs/libguestfs/blob/master/src/inspect-apps.c

so there is no justification to declare one need to install from scratch just because rpm which works for many years fine changes it's storage format

Am 13.01.2016 um 14:30 schrieb Richard Hughes: > On 13 January 2016 at 13:13, Reindl Harald <h.reindl(a)thelounge.net&gt; > wrote: >> so there is no justification to declare one need to install from scratch >> just because rpm which works for many years fine changes it's storage >> format > > I don't think anyone said there was a need to reinstall from scratch so how do you translate "clearly not forward compatible"?

On Wed, 13 Jan 2016 14:38:08 +0100 Florian Festi <ffesti(a)redhat.com&gt; wrote: > On 01/13/2016 02:36 PM, Reindl Harald wrote: > > > > > > Am 13.01.2016 um 14:30 schrieb Richard Hughes: > >> On 13 January 2016 at 13:13, Reindl Harald > >> <h.reindl(a)thelounge.net&gt; wrote: > >>> so there is no justification to declare one need to install > >>> from scratch just because rpm which works for many years fine > >>> changes it's storage format > >> > >> I don't think anyone said there was a need to reinstall from > >> scratch > > > > so how do you translate "clearly not forward compatible"? > > "forward compatible" means the old version of a program being able > to read/process newer version data. > > The current rpm versions will not be able to read the new database > format. I tend to use systemd-nspawn containers for building rpms. So for example, I have a Fedora 24 system and use its dnf to create e.g. Centos 7 container root and then build Centos rpms from within that container. If I understand the change correctly, this is going to break since the Centos 7 rpm-build will not be able to read the database created by the Fedora 24 dnf. I know more people using dnf/rpm to "manage" the containers and this is somewhat a regression for us. I'm not sure there is a way to prevent this breakage... So just FYI. :)

On Fri, 15 Jan 2016 09:24:36 +0100 Tomáš Smetana <tsmetana(a)redhat.com&gt; wrote: > On Wed, 13 Jan 2016 14:38:08 +0100 Florian Festi <ffesti(a)redhat.com&gt; wrote: > I tend to use systemd-nspawn containers for building rpms. So for > example, I have a Fedora 24 system and use its dnf to create e.g. > Centos 7 container root and then build Centos rpms from within that > container. If I understand the change correctly, this is going to > break since the Centos 7 rpm-build will not be able to read the > database created by the Fedora 24 dnf. > > I know more people using dnf/rpm to "manage" the containers and this > is somewhat a regression for us. I'm not sure there is a way to > prevent this breakage... So just FYI. :) won't regular mock chroot have the same problem?

On 13 January 2016 at 13:13, Reindl Harald <h.reindl(a)thelounge.net&gt; wrote: > so there is no justification to declare one need to install from scratch > just because rpm which works for many years fine changes it's storage format I don't think anyone said there was a need to reinstall from scratch.

Daniel P. Berrange wrote: > Well the feature writeup is rather fuzzy on this. It says that in Fedora > 24 rpm will be able to read both old and new format, but it also says > that future RPM versions will drop support for the old format. So unless > there is a mandatory data format conversion during some Fedora upgrade, > then at some point RPM will cease to be able to read existing installs > with the old format which could imply a need to reinstall. Will rpm --rebuilddb convert the database?

On 01/11/2016 09:06 PM, Richard W.M. Jones wrote: > On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote: >> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = >> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat > > Details of the format? > > What forward and backward compatibility guarantees are there? RPM will keep support for BDB for now. But to get rid of the dependency it will be dropped at some point in the future. So it will stay backward compatible.

> Currently we use the (BDB-specific obviously) db_dump tool for this > purpose. Other distros such as Debian as much simpler in this regard > since they expose the package data as plain text files. > > https://github.com/libguestfs/libguestfs/blob/master/src/inspect-apps.c I would advice to change these over to using librpm or one of the rpm cli tools. If there are any tools missing, please let me know and we will try to come up with them. If possible the rpm installation of the system examined should be used.

If this cannot be done you might need a new version of rpm on the host system.

On 01/13/2016 03:07 PM, Richard W.M. Jones wrote: > > Say, for example, that Fedora 24 moves to the new format. Will Fedora 34 > be able to read Fedora 24 RPM databases? Fedora 25 will. Fedora 34 will probably not. If it turns out that reading 10 releases old database is really, really important. We will either keep the bdb support or offer some tools to basically do a rpmdb --exportdb which can be used to create a modern rpm database. Who is maintaining the BDB needed for this in 2021 is not a question I am going to answer, though.

On 01/29/2016 05:02 PM, Richard W.M. Jones wrote: > I'm still unclear why you don't just use sqlite 3. Among other things it is about 2 times lower than the current BDB implementation.

I wonder if the MDB (or LMDB) database from the openldap team was evaluated as a possible BDB replacement?

My understanding of the problem is that it's less about the "doubts about its future" and more about the fact Oracle inexplicably changed the license with BDB 6.x to AGPLv3. Berkley DB 5.3 is old, and no one has forked it and maintained it.

5) BerkeleyDB lives under a licensing cloud. Since 2008 Oracle lawyers were contacting commercial OpenLDAP users and demanding license fees from them, even though BerkeleyDB is expressly licensed for free use in open source software (such as OpenLDAP).

On 01/31/2016 04:17 PM, Jonathan Underwood wrote: > On 29 Jan 2016 10:03 pm, "Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote: >> >> >> FWIW I found the new database backend (not mentioned anywhere in the >> original submission). It is here: >> >> http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb >> >> Since this change has (rashly) been approved by FESCO, I guess we're >> going to be copying this code into some libguestfs tools, and we'll >> have to keep up with changes to RPM. It's a drag compared to using a >> real key-value store. >> >> > > I wonder if the MDB (or LMDB) database from the openldap team was evaluated > as a possible BDB replacement? LMDB seems a poor match because it requires the administrator to set a maximum database size, its key length is limited compared to Berkeley DB (which supports multi-megabyte keys). This means integrating it takes more than just rewriting the API calls. Florian

FWIW I found the new database backend (not mentioned anywhere in the original submission). It is here: http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb Since this change has (rashly) been approved by FESCO, I guess we're going to be copying this code into some libguestfs tools, and we'll have to keep up with changes to RPM. It's a drag compared to using a real key-value store.

Even if the RPM database is only accessed via librpm, it's still important that the most central database present on every Fedora system is reliable, well-tested and flexible. Sqlite is a highly regarded piece of software, which runs on billions of Android phones.

On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote: > On 02/01/2016 10:59 AM, Richard W.M. Jones wrote: > > > Even if the RPM database is only accessed via librpm, it's still > > important that the most central database present on every Fedora > > system is reliable, well-tested and flexible. Sqlite is a highly > > regarded piece of software, which runs on billions of Android phones. > > That's a strange argument because SQLite is optimized for the mobile use > case, where durability is not a concern: > > <http://article.gmane.org/gmane.comp.db.sqlite.general/99223>

On 02/01/2016 11:26 AM, Richard W.M. Jones wrote: > On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote: >> On 02/01/2016 10:59 AM, Richard W.M. Jones wrote: >> >>> Even if the RPM database is only accessed via librpm, it's still >>> important that the most central database present on every Fedora >>> system is reliable, well-tested and flexible. Sqlite is a highly >>> regarded piece of software, which runs on billions of Android phones. >> >> That's a strange argument because SQLite is optimized for the mobile use >> case, where durability is not a concern: >> >> <http://article.gmane.org/gmane.comp.db.sqlite.general/99223> > > I don't see that this posting backs up what you are saying. Of course > databases don't perform the impossible - you have to wait for the > commit to succeed before the data is persisted, anything else requires > backup power supplies and redundant disks. But sqlite is much more > likely to be durable than some hand-written brand new code. Please read the thread. SQLite does not persist the unlink operation on the journal, and an existing valid journal file triggers transaction rollback.

On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote: > On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote: > > FWIW I found the new database backend (not mentioned anywhere in the > > > > original submission). It is here: > > http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb > > > > Since this change has (rashly) been approved by FESCO, I guess we're > > going to be copying this code into some libguestfs tools, and we'll > > have to keep up with changes to RPM. It's a drag compared to using a > > real key-value store. > > May I ask why? Accessing rpm database in a different way than through > librpm API is highly discouraged even now. It's discussed in this thread already. Please see: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/ message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/ https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org /message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/ Even if the RPM database is only accessed via librpm, it's still important that the most central database present on every Fedora system is reliable, well-tested and flexible. Sqlite is a highly regarded piece of software, which runs on billions of Android phones.

On Mon, Feb 01, 2016 at 12:51:47PM +0100, Jan Zelený wrote: > On 1. 2. 2016 at 09:59:23, Richard W.M. Jones wrote: > > On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote: > > > On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote: > > > > FWIW I found the new database backend (not mentioned anywhere in the > > > > > > > > original submission). It is here: > > > > http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb > > > > > > > > Since this change has (rashly) been approved by FESCO, I guess we're > > > > going to be copying this code into some libguestfs tools, and we'll > > > > have to keep up with changes to RPM. It's a drag compared to using > > > > a > > > > real key-value store. > > > > > > May I ask why? Accessing rpm database in a different way than through > > > librpm API is highly discouraged even now. > > > > It's discussed in this thread already. Please see: > > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject. > > org/ message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/ > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject. > > org > > /message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/ > > > > Even if the RPM database is only accessed via librpm, it's still > > important that the most central database present on every Fedora > > system is reliable, well-tested and flexible. Sqlite is a highly > > regarded piece of software, which runs on billions of Android phones. > > Fair enough but that doesn't answer my question. What is the reason for > copying rpm code in libguestfs instead of using librpm? I don't have a clear answer on that yet, because I'm still waiting to find out if the database format will change incompatibly in future. If it changes incompatibly in future, and if librpm won't maintain backwards compatibility with old databases, then we'll need to take snapshots of the librpm code to ensure we can read old RPM databases out of guests. Notes: - I'm not talking about the BDB format - I understand the licensing issues which require a clean break. - I couldn't find a way in the the current librpm code to open the RPM database from a different location - obviously required because we need to open the database using our own trusted code, *not* the guest's untrusted code as was proposed elsewhere in this thread. However that's not a reason not to use librpm, since we could add APIs to do that.

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = https://fedoraproject.org/wiki/Changes/NewRpmDBFormat Change owner(s): * Florian Festi < ffesti AT redhat DOT com > Change format of the RPM Database from Berkeley DB to RPM's own format. == Detailed Description == The current implementation of the RPM Database is based on Berkeley DB. There are doubts about the its future and level of maintenance. In addition rpm's use of the database has multiple issues on its own. As a result RPM upstream is working to replace the database format with a new implementation. == Scope == Proposal owners: * Test and release the already implemented database format. * Make sure it is supported by all important tools - Tools using librpm should not be affected - Libsolv is the most prominent tool accessing the rpmdb directly and needs adjustment *Tooling for converting existing databases.

On Monday, January 11, 2016 03:46:27 PM Jan Kurik wrote: > = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat = > https://fedoraproject.org/wiki/Changes/NewRpmDBFormat > > Change owner(s): > * Florian Festi < ffesti AT redhat DOT com > > > > Change format of the RPM Database from Berkeley DB to RPM's own format. > > == Detailed Description == > The current implementation of the RPM Database is based on Berkeley > DB. There are doubts about the its future and level of maintenance. In > addition rpm's use of the database has multiple issues on its own. As > a result RPM upstream is working to replace the database format with a > new implementation. How will this work in situations like the builders. Rpm on f23 makes the f24 chroots. I am guessing that will be mostly ok, the db will just have the old format. But what about when we move the builders to f24 and use f24 and rpm with the new db format to make rhel 5, 6, 7 and Fedora, 22 and 23 buildroots. Is there the potential that things will break horribly?