-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I figured I should forward on this thread from the Fedora-devel mailing list, as a community member raises some interesting ideas for the OpenLMI project here. I'm not sure I understand it well enough yet to say if this is a good fit, but it's worth sharing with the wider OpenLMI community at least. Please continue discussion in this thread on devel(a)lists.fedoraproject.org, since that's where it started. - -------- Original Message -------- Subject: Re: Inter-WG coordination: Stable application runtimes Date: Sun, 12 Jan 2014 22:34:19 +0200 From: Alek Paunov <alex(a)declera.com&gt; Reply-To: Development discussions related to Fedora <devel(a)lists.fedoraproject.org&gt; To: Development discussions related to Fedora <devel(a)lists.fedoraproject.org&gt; On 10.01.2014 21:12, Matthew Miller wrote: Your expression " ... tools aware of the native packaging system" and the Andrew comment about the pip behaviors above in the thread, encouraged me to share my "big hammer of the DB plumber style" :-) opinion on the problem. TL;DR: What follows is SCC: An idea for optional DB and service which caches bits from YumDB, the local state (RPMDB and /etc) plus the "native" systems like NPM in the unified way for the purposes of planning, resolving and performing system state transitions. Initially I meant to say just few words to mark the possibility, but obviously my English and talent for easy for reading and well focused messages are both far from good, so the whole thing become too long and I am going to split some additional notes into self replays - Excuses! RPMDB and YumDB are two rich datasets present on every Fedora instance representing respectively the local state and distribution+ state of the package universe. '+' because +Fusion*, +COPRs, +LocalOrg, etc. Unfortunately they are somewhat hidden in the dark, because lack of interfaces - we even missing SVG or other explorer for the YumDB graph. The (let think of them as "virtual") Maven DB, PyPI DB, NPM DB, LuaRocks DB, etc, technically are subsets of YumDB (in sense richness of encoded logical relations between the DB nodes used in their schemes - e.g. PyPI DB, before the pip egg, do not knows which file from which module comes, nor have the concept of higher than package NV granularity of the requirement points - Provides and Requires in YumDB). Also, the depsolving of the "native" packaging systems is less sophisticated than both current yum and hawkey ones. These observation naturally lead to the idea of SCC: System Configuration Cache DB representing the merge of RPMDB, YumDB and e.g. local pip egs, PyPI DB (if e.g. additional python modules/versions are needed for the given deployment) where the depsolving could be shifted, somewhat in the same fashion as Data warehouse solutions are used in the enterprises for merging the significant datasets from various ERP systems into single DB for interactive time reporting and decision making. I am using the term SCC (vs. e.g. UPMC: Unified Package Metadata Cache) in attempt to cover a (paraphrased) Mirek question from the beginning of the thread - "OK, we have Fedora and PyPI integrated, at one point of the time for a given instance, the Fedora packaged module has been chosen. What happens if we upgrade Fedora along with am incompatible version this Python module for given installed service" - obviously we need to register in the SCC the dependencies of the installed machine roles with the same effect like Require clauses in the our packages, so the SCC machinery to validate (with negative result in this described case) the yum upgrade or to resolve the upgrade including installation of newest available compatible version from PyPI as an alternative provision during upgrade preparation. Further, I think that ideally SCC should parse/absorb as much as possible system object properties and relations from /etc (plus /lib and /var configuration areas) to allow sysadmins and devops to inject rules for effective use of these sets latter in the depsolving (and other DB functionality). That said, the integration with OpenLMI, or even implementing the whole thing under the OpenLMI umbrella, both seems natural. So, finally on that road we have: - - choice for good enough DB engine for SCC, query language, compiler [*], etc. design decisions like sync protocol and plugable data sources. - - Yum/RPM datasets: optional rpm, yum, hawkey hooks for syncing their DBs, alternatively just sync tools. - - optional pip, npm, luarocks, etc. hooks for the same, alternatively just sync tools. - - OpenLMI integration for absorbing system configuration, alternatively just Augeas import + transformation rules to sync the DB representation of the system objects. - - sccd capable to: - depsolving (on top of cumulative - YumDB + native managers DBs, preferably providing interfaces to the system libsolv for using SCC DB in place of the libsolv own filestores. - invoke package operations with yum/dnf + "native" managers - provides sysadmins, developers and devops with the capability of defining system overlays - services/applications trees and their relations with the rest of the DB. - supports live textual views (e.g. trough FUSEs) for cases when local preferred deployment style is ansible, salt, etc. based. As alternative/complementing the own daemon approach the above above functionality might be implemented in the form of OpenLMI providers (or existing provider extensions). - - sccd-web: WebUI exposing full functionality, alternatively Cockpit (OpenLMI WebUI) extension. - - scc [**] command line shortcut for few simple tasks. - - providing pip, luarocks, etc patches for sccd integration if present, alternatively "native" managers command lines/api implementations on top of sccd; providing yum and hawkey plugins for "native" packages conflicts handling - - NTH: remote SCC DB for the instance, - - NTH: SCC local state for multiple instances (e.g. deployment nodes or local containers) kept in the same SCC DB - - NTH: SCC local state inheritance between instances Kind Regards, Alek [*] Crucial aspect of any sophisticated data management system is the data query and manipulation language. Unfortunately the choices are rather limited - Imperative approaches (recently resurrected by some NoSQL DBs) are weak and error prone; SQL and few more "text prose" languages have proven their incompatibility with the vast majority of the developers (these without years of specific experience around the data volumes processing). The predominant workaround seems to be ORMs, but ORMs and "sophisticated/fast" should not be mixed in same project :-). [**] scc resolve python module hgsubversion output: Found 2 resolutions, Transaction IDs: 1921, 1922 output: Recommended: http://scc-geteway/machine-id/transaction/1921 user: browsing, thinking ... (hgsubversion comes from PyPI), but happily spotting that subvertpy dependency is satisfied by fedora provided package - it is linked with the system subversion. scc transaction 1921 add resolve luajit 2.0 module zmq min-version 0.3.3 output: Found 17 resolutions, Transaction IDs: 1921, 1924, ... output: Recommended: http://scc-geteway/machine-id/transaction/1921 user: browsing, ... lua zmq will use the system zmq library, which is the in the desired version range - OK scc transaction 1921 apply etc ... - -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct