[fedora-arm] Status Update: ARM Koji Build System Outage

Hello Everyone! As you are aware that we had *some* issues with the build farm due to a power outage yesterday. We didn't have too many chances to send updates. So here it is: After the power outage, all the critical servers and the builders were up and running. However due to the power outage there was some other networking issues at Seneca. A short while later some of the systems rebooted and we started dealing with it as soon as we noticed. At that point some of the management servers were still down. There were issues with Apache, SE Linux or Networking in one server or another. After those issues were dealt with, there was the NFS Locking issue and it turned out to be due to some Firewall rules. All that got fixed last night. However, we are still not 100%. Some of the builders are still timing out as of this morning and that issue is still under investigation. Please consider the build farm to be out of commission till further notice. You can submit jobs and that may or may not go through. We are going to bring the farm back up as fast as we can. We apologize for the inconvenience and thank you for your patience. Today *will* be a better day and we will try to provide updates more frequently. If you have any question please reply to this email or express it in the channel. Sincerely Max Abed Fedora ARM Team Seneca CDOT