I was looking over issue #10267 after Leo [https://pagure.io/user/leo] offered to work it and noticed that it was being provisioned manually. Like you said, some of the instances on AWS are provisioned manually, but this seemed like a great candidate for being managed through fedora-infra/ansible playbooks. It made me want to know more. This might be a little different than what I was expecting, but yea. I can see how that would work. I am personally thumbs-up for this for a number of reasons and here is what I was thinking off the top of my head: 1) Deployments can be done either through a) AWS Cloudformation Templates or the new Cloud AP managed with Ansible or b) through the Ansible community.aws and amazon.aws collections and their respective modules (works in 2.9+) 2) The ec2_inventory plugin is very reliable. It's possible to create a filter to ensure that we are managing only instances that are: a) manageable by the deploying IAM user (resource groups can ensure proper authorization) b) tagged according to the requirements of the ec2 inventory plugin configuration (that can be specific to copr or maintainer instances, etc) 3) Instance availability isn't guaranteed. There are many reasons for a single instance to require replacement. It could be for an underlying hardware failure or reboot requirement, but occasionally, for whatever reasons, it happens and the requirement is best satisfied with a redeployment. Kicking that off from the Ansible script saves a lot of headache for, say, the newest on-call resource who doesn't want to learn a new skill before they can go back to sleep. 4) It fits with the expectations of the open practice library and pillars for Operational Excellence and Reliability.

On Wednesday, October 20, 2021 6:44:20 PM CET Kevin Fenzi wrote: Yeah, I was in particular lazy to teach the playbook to start the machines. We instead just documented the steps in for us for now: https://docs.pagure.org/copr.copr/how_to_upgrade_persistent_instances.html But we already worked with starting builder AWS machines via playbooks: https://pagure.io/fedora-infra/ansible/blob/0fb047d8ad3d9fa3a2d4d5c7c175c... ... points to: https://pagure.io/fedora-infra/ansible/blob/0fb047d8ad3d9fa3a2d4d5c7c175c... Something like that could be reused. We don't use this anymore, now we use scripts from https://github.com/praiskup/resalloc-aws/tree/main/bin . That's because we already start _many_ other types of VMs, and that "ansible ec2 module" approach was too different from others. Plus, from time to time, the playbook failed, and it caused headaches to us (we start thousands of VMs, and we need to have the boot up process robotized, and 100% under control). This wouldn't be a blocker for the infra machines (contrary to the builder playbooks, the infra playbooks execution is under a _human_ control). Definitely, it would simplify our life. Actually, very soon we'll have to migrate our servers from F33 to F35 so we would benefit from this change. Meh, yeah. Though except that we need to create a new user in AWS for batcave, I don't think it is too risky or hard to implement (can be configured in private git, in vars file). The VMs started using this "token" would be easily traceable I think. Pavel

On Thu, Nov 4, 2021 at 8:13 AM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote: There is a possibility to automate this within AWS but may require a change of url. We could delegate a subdomain to route53, something like copr-aws.fedoraproject.org then when the instances are created we can get the ip6/4 address as an output and update route53 with <serverx>.copr-aws.fedoraproject.org as part of the playbook. An example of which we use in arc projects: ec2_instance_create <https://pagure.io/fedora-infra/arc/blob/main/f/ansible/roles/ec2_instance... Mark