-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going. First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over. Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion. Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram(a)atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite. Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion. The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now. Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ). Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!

Everybody was bitching about CVS for years,

On Friday 30 of July 2010 05:55:09 Jesse Keating wrote: > ... Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ). Thanks for your hard work! Could you describe this in more details: OLD CVS | NEW GIT | Notes make tag | N/A | Explicitly tagging source states for package builds is no longer necessary. how this exactly works? What will happen in following cases? : 1) commit some changes with release number change, commit another changes, build

2) commit some changes with release number change, scratch build, commit rest of the changes, build

3) commit some changes with release number change, someone else starts build, commit rest of the changes, build

4) commit some changes with release number change, build - fails because of typo/missing updated sources, commit fix, build

Michal

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxTDI0ACgkQ4v2HLvE71NW+iACgkE2cIycWxBxbhVWia1ynIrKT 5xAAoKfpYdgxwlF3P3yx62SVccAMNVgW =E9/3 -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This evening we opened up dist-git for business.  Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files.  This has been a long time coming and a massive effort.  I want to take a little time here to outline what we've done and where we are going. First a brief outline of how our CVS system worked.  CVS is a daemon of sorts, and all repos typically live within a single CVSROOT.  Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT.  Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch".  CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content.  Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over. Now we look at git.  Git is for the most part a daemonless system.  Each repository is completely stand alone and generally does not require any other infrastructure to be useful.  You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git).  Generally there is no running daemon to connect to and authenticate with.  Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level.  One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases.  In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion. Enter gitolite.  Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more!  By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers.  The gitolite upstream ( Sitaram Chamarty, sitaram(a)atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique.  We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050).  The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly.  Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box.  Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys.  Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties.  Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix.  At this point our production server is a whopping one line different from current gitolite upstream.  This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite. Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what.  I had never been a fan of Make, it was entirely too difficult to modify and innovate with.  Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python.  The fedpkg tool was born and took off.  fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code.  While far from complete, it is a great start (if I do say so myself!) at replacing the make system.  Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions.  More will come as it starts to be more widely used.  The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide.  I'll spare you the details but I'm fairly happy with what we have.  The end result should be far fewer misfires and end user confusion. The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories.  I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others.  In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories.  There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them.  The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this.  All but the kernel repo  seems to have converted successfully which is a pretty good success rate in my book.  We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now. Without the help of many others, this project would never have gotten done.  Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way.  Things will be bumpy over the next few weeks as we really start putting distgit to the test.  No amount of staging and testing can really replace production use.  There will be many more updates to fedpkg as bugs are found and fixed and features are contributed.  Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ). Once again I want to thank everybody who helped out and for all the (continued) patience!  I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues.  Look for Oxf13 on freenode.  Happy gitting!

On Fri, 2010-07-30 at 09:49 +0100, pbrobinson(a)gmail.com wrote: > Excellent news! Thanks for all the work. > > One thing I've never got around to working out how to do in git which > is different from previous is dealing with branches. Where previously > it was as simple as changing directories to deal with the various > fedora releases obviously with real branches now we need to do > something a little differnet. Could someone update the docs with > details how to do this? I retried "fedpkg switch-branch f-13" (and > various other possible branch names) and using a "git branch" didn't > give me any branches other than master. Could we also add some extra > branch related commands to indicate things like listing the current > branch, a list of all branches, and how we would commit a new spec to > more than one branch. > I'm not sure how it works if you do "fedpkg clone -B libfoo", but probably similar to the CVS setup. If you do just "fedpkg clone" the branches are managed in git only so the "fedpkg switch-branch f13" indeed switches you to f13 branch even though it looks like master ;-) Personally I use "git merge <branch-to-merge-from>" to sync the branches (if I have same spec for more than one branch).

Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).

2010/7/30 Martin Sourada <martin.sourada(a)gmail.com&gt;: > On Thu, 2010-07-29 at 20:55 -0700, Jesse Keating wrote: > <snip> >> Without the help of many others, this project would never have gotten >> done. Folks helped out with Koji modifications, with fedpkg >> contributions, with repeated testing of attempted conversions, with >> logic checking of my plans, of helping me understand more of git >> internals and deciphering error output, and most of all with being >> patient while we worked through the transition and very positive along >> the way. Things will be bumpy over the next few weeks as we really >> start putting distgit to the test. No amount of staging and testing can >> really replace production use. There will be many more updates to >> fedpkg as bugs are found and fixed and features are contributed. Wiki >> pages will get filled out as knowledge of how to interact with dist-git >> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a >> good start ). > > Ok, so trying to update a package (libass) I've noticed a tiny problem: > the fedpkg switch-branch does not seem to set up proper tracking of > remote branches which then results in (uhm, I'm using git commit -a and > git push, so you maybe handle this in fedpkg commit -p): > > pyfedpkg.FedpkgError: There are unpushed changes in your repo > > fedora-packager-0.5.0.1-3.fc12.noarch > > Martin > > -- I now use 'git checkout -t origin/f14/master' instead of 'fedpkg switch-branch f14' temporarily.

Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p): pyfedpkg.FedpkgError: There are unpushed changes in your repo fedora-packager-0.5.0.1-3.fc12.noarch

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxTDxwACgkQ4v2HLvE71NUAiACcCK8OsJWHs2y43ikmiIwMrEt3 vJcAoKYqDJz9MeEDB1ocPPl8aG7X9Lvy =xKCB -----END PGP SIGNATURE-----

 On 07/30/2010 02:44 PM, Christof Damian wrote: > Thank you for all the work. > > I am currently in the process of convincing my employer to change from > svn to git and know how much work this is. Running git-svn clone alone > takes a few weeks for us. FYI,  git-svn is for users who want to continue to use Subversion on the server but interact using Git from clients.  It is not a tool for conversion.  It is a common mistake.

I think you need to get the latest package from koji so it points to the live git repos and not the test ones.

I had similar issues when I tested it first up but fedora-packager-0.5.1.0-1 seems to have fixed it.

starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).

Excerpts from Christof Damian's message of Fri Jul 30 16:10:44 +0200 2010: > I am trying to update one of my packages with git for the first time. > Works good, but "fedpkg build" fails for me. > > It complains that it can't retrieve the sources: > http://koji.stg.fedoraproject.org/koji/getfile?taskID=2239266&name=ro... Your fedora-packager package is too old. Install version 0.5.x (from koji, but should be available in the repos now too I believe)

fedpkg build fedpkg build Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 297, in build mymodule.init_koji(args.user, kojiconfig) File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line 1102, in init_koji defaults['serverca']) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request response = h.getresponse(buffering=True) AttributeError: PlgHTTPS instance has no attribute 'getresponse' [Exit 1] Seems to be broken in Rawhide. rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc14.noarch

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxTFj0ACgkQ4v2HLvE71NVI8wCeP1Dlv4oVg6fM6lCsKwIiS6+z cLcAoJ1ArFu/cICQzh9uxzCZIJcT/yV0 =St51 -----END PGP SIGNATURE-----

On 07/30/2010 08:13 PM, Jesse Keating wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/30/2010 08:52 AM, Daniel J Walsh wrote: >> >> fedpkg build >> fedpkg build >> Traceback (most recent call last): >> File "/usr/bin/fedpkg", line 959, in<module> >> args.command(args) >> File "/usr/bin/fedpkg", line 297, in build >> mymodule.init_koji(args.user, kojiconfig) >> File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line >> 1102, in init_koji >> defaults['serverca']) >> File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, >> in ssl_login >> sinfo = self.callMethod('sslLogin', proxyuser) >> File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, >> in callMethod >> return self._callMethod(name, args, opts) >> File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, >> in _callMethod >> return proxy.__getattr__(name)(*args) >> File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ >> return self.__send(self.__name, args) >> File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request >> verbose=self.__verbose >> File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request >> return self.single_request(host, handler, request_body, verbose) >> File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request >> response = h.getresponse(buffering=True) >> AttributeError: PlgHTTPS instance has no attribute 'getresponse' >> [Exit 1] >> >> Seems to be broken in Rawhide. >> >> rpm -q fedora-packager >> fedora-packager-0.5.1.0-1.fc14.noarch > > Yeah, this is unfortunate, but something is broken in the xmlrpc stuff > in or around koji. > Any workaround, fix for that?

Thanks for dist-git -- one step closer to banishing CVS from my life! :) A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config: [remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too. Thanks! Josh

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxTFroACgkQ4v2HLvE71NXnlgCfQc2y0CIS5Y2F8AWFYZFXBv1+ DwsAn23WIs7v95bup6yEVABogZtUpOsW =YAoa -----END PGP SIGNATURE-----

On Fri, 2010-07-30 at 11:15 -0700, Jesse Keating wrote: > On 07/30/2010 09:14 AM, Josh Stone wrote: >> A git trick I'd like fedpkg to learn is to use separate url/pushurl, >> e.g. in .git/config: >> >> [remote "origin"] >> fetch = +refs/heads/*:refs/remotes/origin/* >> url = git://pkgs.fedoraproject.org/foo >> pushurl = ssh://user@pkgs.fedoraproject.org/foo >> >> This way it will fetch over the faster git protocol and still push over >> ssh. I expect that will lessen the load on Fedora infrastructure too. > Hrm, not sure this is something we should do by default, are there any > drawbacks? You need two network protocols to do work on the package. That might be difficult in some restricted networks.

For everyone with actual proper internet access, this looks like it could work quite well.

A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config: [remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.

On Tue, Aug 3, 2010 at 11:16 AM, Matt McCutchen <matt(a)mattmccutchen.net&gt; wrote: > don't want malware landing on my machine because someone did a MITM > attack on a Fedora maintainer's unencrypted "git fetch" and inserted > some extra patches to get pushed back to the real repository later. The git protocol makes it extremely hard to inject malware successfully. It would have to match sha1, _and_ match resulting filesize _and_ be meaningful code, all without the benefits of preimaging.

Matt McCutchen <matt(a)mattmccutchen.net&gt; wrote: > No. If the attacker MITMs the entire connection, they can lie about the > values of the remote refs too, so there is no need to find a hash > collision. And how would you then be allowed to push? The git server would see that your history doesn't match the history it has and will refuse the commits.

I suspect it might short-circuit the 'ahhh, but what about...' 'oooh, but then I can...' nature of the conversation if you just put together a proof-of-concept attack and document it somewhere. I suspect the git maintainers might be interested at that point as well. :)

On Wed, 04 Aug 2010 22:03:14 +0200 Till Maas <opensource(a)till.name&gt; wrote:

> The attack is quite trivial: > 1) clone the git pkg Fedora repos > 2) commit some nasty change > 3) publish the repo on some server > 4) if the victim wants to fetch from the Fedora pkg repo, use the MITM > attack to make him fetch from the server set up in step 3. Steps 1-3 > can obviously be done on-demand. > > If this is e.g. done on a conference / FUDCon / Fedora Action Day, the > attack can easily targeted to make the change in step 2 be expected to > be fast forward. E.g. if packages simply need to be bumped for a > rebuild, a upload of a bad tarball and modification of the sources > file might be unnoticed. Just to clarify, as this is a long thread: This only works if people are using git:// urls, not the default for fedora ssh: ones, right? (provided you have connected before to pkgs.fedoraproject.org and have the known_hosts entry?)

On Thu, 5 Aug 2010, Till Maas wrote:

> Yes ssh is secure if used properly. To get the proper known_hosts entry, > one has to download https://admin.fedoraproject.org/ssh_known_hosts btw. > We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)

On Fri, 6 Aug 2010, Till Maas wrote: > On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote: > > On Thu, 5 Aug 2010, Till Maas wrote: > > > > Yes ssh is secure if used properly. To get the proper known_hosts entry, > > > one has to download https://admin.fedoraproject.org/ssh_known_hosts btw. > > > > > > > We also use SSHFP records for those of you that want to enable > > VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts > > have it but many of our 'user' based external hosts do (pkgs, > > fedorapeople, fedorahosted, etc) > > Afaik the SSHFP records are not protected against tampering by an MITM > attacker. > They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.

On Fri, 6 Aug 2010, Till Maas wrote: > On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote: > > On Thu, 5 Aug 2010, Till Maas wrote: > > > > Yes ssh is secure if used properly. To get the proper known_hosts entry, > > > one has to download https://admin.fedoraproject.org/ssh_known_hosts btw. > > > > > > > We also use SSHFP records for those of you that want to enable > > VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts > > have it but many of our 'user' based external hosts do (pkgs, > > fedorapeople, fedorahosted, etc) > > Afaik the SSHFP records are not protected against tampering by an MITM > attacker. > They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.

On Fri, Aug 06, 2010 at 04:31:00AM -0500, Mike McGrath wrote: > On Fri, 6 Aug 2010, Till Maas wrote: > > On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote: > > > We also use SSHFP records for those of you that want to enable > > > VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts > > > have it but many of our 'user' based external hosts do (pkgs, > > > fedorapeople, fedorahosted, etc) > > > > Afaik the SSHFP records are not protected against tampering by an MITM > > attacker. > > They're better then ssh alone. They're only used for the first initation. > So you'd have to be MITM'ed on the first connection in which case you're > right, they wouldn't protect against that. Afaik using the SSHFP records make SSH not warn the user that the host key is not verified. If SSH would e.g. warn that the host key is unknown, but at least matches the SSHFP record, then it might be a little better. But actually it makes MITM attacks easier, because if one tampers the DNS response and the SSH connection, then the user does not even get a warning on the first attempt, making the situation even worse IMHO. And SSH is only vulnerable to MITM attacks on the first connection in general and I guess that SSHFP records are not used anymore after the first connection. What would they be good for when the host key is already known to SSH?

On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote: > Yes ssh is secure if used properly. To get the proper known_hosts entry, > one has to download https://admin.fedoraproject.org/ssh_known_hosts btw. I'm very glad to see that Fedora provides such a list.  I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons). Is it documented anywhere?  For full security, every packager should install it rather than allowing ssh to add host keys on first use.

-- Matt -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, 2010-08-10 at 09:07 -0600, Stephen John Smoogen wrote: > On Sun, Aug 8, 2010 at 14:04, Matt McCutchen <matt(a)mattmccutchen.net&gt; wrote: > > On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote: > >> Yes ssh is secure if used properly. To get the proper known_hosts entry, > >> one has to download https://admin.fedoraproject.org/ssh_known_hosts btw. > > > > I'm very glad to see that Fedora provides such a list. I just installed > > it on my computer (after filtering out hostnames not ending with > > fedoraproject.org, for obvious reasons). > > > > Is it documented anywhere? For full security, every packager should > > install it rather than allowing ssh to add host keys on first use. > > Well I am not sure that file would be all that useful as it contains > lots of hosts a packager would not get to AND could conflict with > other networks as it contains a lot of 10.X.X. and 192.X.X. ips. Then let's post an excerpt that would be useful to packagers. > It also gets updated from time to time as we rebuild hosts. That just speaks to the need for better tooling to maintain personal known-hosts files, or for Fedora to operate an ssh certificate authority. It appears that the ssh folks rejected X.509 out of disgust for the public CAs, found themselves left with no solution at all to authenticate hosts the first time, and are now reimplementing it incompatibly. The man page claims the ssh implementation is "much simpler" -- perhaps, but it won't integrate with X.509-based systems and will be playing catch-up on features for a while. CRLs or OCSP, anyone? A thread from 2002 with some frank discussion that is still valid now: http://marc.info/?t=101179752100001&r=1&w=2

On Tue, Aug 3, 2010 at 11:16 AM, Matt McCutchen <matt(a)mattmccutchen.net&gt; wrote: > don't want malware landing on my machine because someone did a MITM > attack on a Fedora maintainer's unencrypted "git fetch" and inserted > some extra patches to get pushed back to the real repository later. The git protocol makes it extremely hard to inject malware successfully. It would have to match sha1, _and_ match resulting filesize _and_ be meaningful code, all without the benefits of preimaging. Even for crypto hashes that have been "broken" for a while, doing the above is a huge challenge. If you do consider this a real risk, here's someone who wants to want to play with you, and build a bunker, 5 miles underground... http://marc.info/?l=git&m=111375923219555&w=2

Once again I want to thank everybody who helped out and for all the (continued) patience!  I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues.  Look for Oxf13 on freenode.  Happy gitting!

On Fri, Jul 30, 2010 at 1:35 PM, Konstantin Ryabitsev <icon(a)fedoraproject.org&gt; wrote: > Hmm... Doesn't seem to be working at all for me. > > $ fedpkg co verbiste > Traceback (most recent call last): > File "/usr/bin/fedpkg", line 959, in <module> > args.command(args) > File "/usr/bin/fedpkg", line 409, in clone > pyfedpkg.clone(args.module[0], args.user, args.path, args.branch) > File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line > 302, in clone > _run_command(cmd) > File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line > 115, in _run_command > stderr=sys.stderr, shell=shell) > File "/usr/lib64/python2.6/subprocess.py", line 483, in check_call > retcode = call(*popenargs, **kwargs) > File "/usr/lib64/python2.6/subprocess.py", line 470, in call > return Popen(*popenargs, **kwargs).wait() > File "/usr/lib64/python2.6/subprocess.py", line 621, in __init__ > errread, errwrite) > File "/usr/lib64/python2.6/subprocess.py", line 1126, in _execute_child > raise child_exception > OSError: [Errno 2] No such file or directory > $ rpm -q fedora-packager > fedora-packager-0.5.1.0-1.fc13.noarch Ah, as silly as it was, I didn't actually check whether git was installed -- I rather assumed it would be as a dependency. Everything works after actually installing git (though it's still a bug in fedora-packager deps). Regards,

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iEYEARECAAYFAkxTFvoACgkQ4v2HLvE71NVW9wCgpA0QFI/ta3ErcOIwVWP9AOAr efkAnjf0tGG4mAadskt59999wl0sTidW =HQ90 -----END PGP SIGNATURE-----

On 7/29/2010 10:55 PM, Jesse Keating wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > This evening we opened up dist-git for business. Dist-git is our git > based replacement for dist-cvs, the source control system we were using > for our package specs, patches, and source files. This has been a long > time coming and a massive effort. I want to take a little time here to > outline what we've done and where we are going. > > First a brief outline of how our CVS system worked. CVS is a daemon of > sorts, and all repos typically live within a single CVSROOT. Within > this CVSROOT we had an 'avail' system to make use of, that we could > populate with data from Fedora Account System and dump into this file. > Avail worked on path names, relative to the CVSROOT. Since we used > directories for each Fedora release as pseudo branches we could set > avail info on each release "branch". CVS also used some filesystem > symlink tricks to create a "common" subdir for every package module, and > this is where we stuffed common scripts and Makefile content. Pretty > clever on one hand, we can make updates to the make system without > touching every single package, but it is pretty hackish and we had > constant issues where somebody would attempt an action using old common > content and stuff would fall over. > > Now we look at git. Git is for the most part a daemonless system. Each > repository is completely stand alone and generally does not require any > other infrastructure to be useful. You can interact with a repository > directly on the filesystem using /usr/bin/git or you can interact with > it through say ssh, again using /usr/bin/git (your local /usr/bin/git > will call a remote /usr/bin/git). Generally there is no running daemon > to connect to and authenticate with. Basic authentication of who can > check out what and commit where with git happens at the filesystem > permissions level. One twist with this is that with git, we wanted to > use real branches within a package repo to reflect the different > Fedora/EPEL releases. In repo branches are not reflected with path > names that we could set filesystem ACLs upon, so this posed a problem > for our conversion. > > Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an > addon system to git that provides ACL functionality including different > rights for different branches within a given repository, and more! By > using gitolite as a replacement for /usr/bin/git when a user connects to > our git server we can again utilize the information we have within the > Fedora Package Database and properly allow / restrict changes on > specific branches for specific developers. The gitolite upstream ( > Sitaram Chamarty, sitaram(a)atc.tcs.com ) has been fantastically > responsive to our needs, which are admittedly a little unique. We have > a very large set of repositories (over 10.5K) and a largish number of > contributors (1050). The combination of the two leads to a very very > large and complex ACL structure that at first broke the system quite > badly. Upstream was very quick to create a "bigconfig" method of > compiling the ACLs without crashing the box. Our other unique needs > involve having individual accounts for each committer instead of a > shared account with a large list of allowed SSH keys. Add to that some > of our accounts need to be able to ssh shell into the git server for > administrative duties. Throughout our trials and testing with gitolite > every time we've ran into some issue that just didn't fit the mold, > Sitaram has been there with a smile and a fix. At this point our > production server is a whopping one line different from current gitolite > upstream. This is a fantastic win for us, for our sustainability, and > for the next large group that wants to make use of gitolite. > > Once we had a plan for ACLs and for branches, we had to decide if we > were going to replace the Make system and with what. I had never been a > fan of Make, it was entirely too difficult to modify and innovate with. > Since I was the one pushing this transition forward, I decided to write > a new tool in my favorite language, python. The fedpkg tool was born > and took off. fedpkg was born around January 4th, 2010 and has since > grown into 1,523 (via sloccount) lines of code. While far from > complete, it is a great start (if I do say so myself!) at replacing the > make system. Because it is written in python (or maybe just !Make) it > seems to be easier to contribute to, and I've already gotten a number of > contributions. More will come as it starts to be more widely used. The > biggest challenge with fedpkg is removing the need to update something > on the end user's system every single time we added a new Fedora release > and changed what happens when you build for rawhide. I'll spare you the > details but I'm fairly happy with what we have. The end result should > be far fewer misfires and end user confusion. > > The last major piece of the puzzle was how to actually convert the > existing CVS repositories, including the fun pseudo branches, into git > repositories. I tried a number of options over the years (I've been > working on this off and on for nearly 4 years!) ranging from the built > in git cvsimport to git-svn to parsecvs and a few others. In the end, > we took a page from the gnome project and used parsecvs ( > http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of > our repositories. There were a few that gave parsecvs fits and recent > versions of git cvsimport were able to handle them. The git system is > fantastic enough that we were able to merge our pseudo cvs branches into > actual git branches complete with a real shared history, but again I'll > spare you the details of the scripting to do this. All but the kernel > repo seems to have converted successfully which is a pretty good > success rate in my book. We may yet get the kernel converted, but in > the interest of time we opted to start fresh with dist-git for now. > > Without the help of many others, this project would never have gotten > done. Folks helped out with Koji modifications, with fedpkg > contributions, with repeated testing of attempted conversions, with > logic checking of my plans, of helping me understand more of git > internals and deciphering error output, and most of all with being > patient while we worked through the transition and very positive along > the way. Things will be bumpy over the next few weeks as we really > start putting distgit to the test. No amount of staging and testing can > really replace production use. There will be many more updates to > fedpkg as bugs are found and fixed and features are contributed. Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ). > > When I do fedpkg clone -B libfoo, to get the old-style layout, all the branches are identical to devel. The branches on the standard clone seem fine. I've not made any changes with the -B layout, but I thought this should be corrected or deprecated. fedora-packager-0.5.1.0-1.fc13.noarch -J

> Once again I want to thank everybody who helped out and for all the > (continued) patience! I'll be available via email and IRC as much as > possible the next few days to help anybody with dist-git issues. Look > for Oxf13 on freenode. Happy gitting! > > - -- > Jesse Keating > Fedora -- Freedom² is a feature! > identi.ca: http://identi.ca/jkeating > _______________________________________________ > devel-announce mailing list > devel-announce(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel-announce >

l return(int(fedoras[-1].strip('f')) + 1) IndexError: list index out of range Help is really appreciated. Thanks!

OK. I deleted the current repo and started again. I do have fedora-packager 0.5.1. Now fedora-packager crashes when uploading new sources. $ fedpkg clone zaz Cloning into zaz... [..] Resolving deltas: 100% (11/11), done. $ cd zaz/ $ cp -a ~/rpmbuild/SOURCES/zaz-0.8.0.tar.bz2 . $ cp -a ~/rpmbuild/SPECS/zaz.spec . $ fedpkg new-sources zaz-0.8.0.tar.bz2 Uploading: 6a63f986a80b4f4e1852ecf9871e9735  zaz-0.8.0.tar.bz2 Traceback (most recent call last):  File "/usr/bin/fedpkg", line 959, in <module>    args.command(args)  File "/usr/bin/fedpkg", line 540, in new_sources    mymodule.upload(args.files, replace=args.replace)  File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 1278, in upload    if lookaside.file_exists(self.module, file_basename, file_hash):  File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 580, in file_exists    curl.perform() pycurl.error: (35, 'SSL connect error') [andrea@panoramix zaz]$

I just yesterday had similar problems and then found out my ~/.fedora.cert was out of date since five days ago. Maybe it's the same for you, did you already had a look there? :)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This evening we opened up dist-git for business.  Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files.  This has been a long time coming and a massive effort.  I want to take a little time here to outline what we've done and where we are going. First a brief outline of how our CVS system worked.  CVS is a daemon of sorts, and all repos typically live within a single CVSROOT.  Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT.  Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch".  CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content.  Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over. Now we look at git.  Git is for the most part a daemonless system.  Each repository is completely stand alone and generally does not require any other infrastructure to be useful.  You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git).  Generally there is no running daemon to connect to and authenticate with.  Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level.  One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases.  In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion. Enter gitolite.  Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more!  By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers.  The gitolite upstream ( Sitaram Chamarty, sitaram(a)atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique.  We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050).  The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly.  Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box.  Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys.  Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties.  Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix.  At this point our production server is a whopping one line different from current gitolite upstream.  This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite. Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what.  I had never been a fan of Make, it was entirely too difficult to modify and innovate with.  Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python.  The fedpkg tool was born and took off.  fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code.  While far from complete, it is a great start (if I do say so myself!) at replacing the make system.  Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions.  More will come as it starts to be more widely used.  The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide.  I'll spare you the details but I'm fairly happy with what we have.  The end result should be far fewer misfires and end user confusion. The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories.  I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others.  In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories.  There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them.  The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this.  All but the kernel repo  seems to have converted successfully which is a pretty good success rate in my book.  We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now. Without the help of many others, this project would never have gotten done.  Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way.  Things will be bumpy over the next few weeks as we really start putting distgit to the test.  No amount of staging and testing can really replace production use.  There will be many more updates to fedpkg as bugs are found and fixed and features are contributed.  Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ). Once again I want to thank everybody who helped out and for all the (continued) patience!  I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues.  Look for Oxf13 on freenode.  Happy gitting! - -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating _______________________________________________ devel-announce mailing list devel-announce(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce