Copr post-mortem

Hi, today we experienced Copr failure and outage. First we got reports that builds are failing due failure to sign resulting packages. We had similar problem before Christmas, but investigation of logs shown that this was different issue. It took us a moment to find that we run out of inodes on storage where GPG keys are stored. The storage is only 1GB big, and default size of inodes was just 65k. Although we use only 40% of storage, we used all inodes. So we moved the data, formatted the partition with high -N option and moved data back. Right now the Copr and package signing is operational again. Thank to Clime, who helped me to restore the service. -- Miroslav Suchy, RHCA Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys