On 11/15/2016 10:35 AM, bruce wrote:
Yo Rick!!!
You are da mann!!! Owe you a six pac of OJ one of these days!
Heheheheh! OJ gives me heartburn. I'll take cranberry juice, though. ;-)
I completely forgot about the changing IP on the remote client, means that while the ssh key is valid, the request for confirmation will occur for the "known_host" (iirc). So yeah I'll have to mod either the local or remote sshd/ssh config file for that..
Subtle things like that drive me crazy. Well, perhaps not a drive but a short putt.
As to the initial IP of the cloned droplets, I populate predata with that via the API that's used when generating the set of remote instances.
I'll take a look at the tools you mentioned as well..
Thanks much for all, for being another set of "eyes"!!
Really do appreciate the comments.!!
That's what the list is for, my friend. Be warned, however. Sometimes the solutions offered become giraffes--which are horses designed by committees.
On Tue, Nov 15, 2016 at 12:36 PM, Rick Stevens ricks@alldigital.com wrote:
On 11/15/2016 07:39 AM, bruce wrote:
Hey!
--This is offtopic from fed --
Thanks to all who've commented on the SSH/screen issue! Think I got that part resolved.
Now, in order to use the ipAddresses that have been generated for the droplets, need to get opinions/thoughts on apps to use to handle connecting to the different numbers of droplets/instances for the project.
The following items are a kind of rough overview of what we're thinking needs to be done.
We're looking at clusterSSH (even though it appears to need to be built from source!). Other thoughts/opinions on apps that might suffice would be good.
ClusterSSH is available from the standard repos, no need to build it:
[root@prophead ~]# dnf list clusterssh* Last metadata expiration check: 2:18:20 ago on Tue Nov 15 06:48:02 2016. Installed Packages clusterssh.noarch 4.08-1.fc24 @updates
Any tool we use, will need to initially handle ~50 droplets/instances, and scale to ~500-1000
The tool should:
- be able to fire up group(s) of servers based on ipAddress (type of
droplet -fetch/parse/etc..) -be able to generate cmd to single/multiple -be able to display either single/group of terms based on ipAddress (single/group) -be able to group the term(ipAddress) -- single term/multiple term in the group(s) -be able to switch between the groups, which in turn display the terms for the group
cssh can do all of that, but once launched in interactive mode, you can't "switch between the groups". If you use the /etc/clusters mechanism, e.g. "cssh group-from-cluster-file", then the terminals for all machines in that cluster will be displayed on your workspace. You could have multiple workspaces and run a cssh for a different cluster on each one and switch between workspaces. We do that a lot.
The use case... -the crawler spins up a number of droplets/instances -the crawler generates the required ipAddresses and "groups" the ipAddresses, based on fetch/parse use
Uhm, if the droplets get different IPs each time, one of the first things I'd have them do is figure out what their IP is and report it to a database or something so it's known by your app.
-All the "instances' are generated/cloned to have the required apps on the server in order to be fetch or parse - there's no need to upload/scp files to the remote instances
This is outside the spec of a terminal manager.
Tool Requirements: -Nice if the termManagerApp is able to use config/xternal files to handle the ipAddresses to create groups as required -Nice if the termManagerApp is able to display terms in a given group -App has to handle external pub/priv key, all terms (cloned) have the same key
That's a function of the access method. If the droplets regenerate ssh keys and such each time they spin up, you'll have issues since the ssh client will see a new host key each time you spin up a droplet.
The termManagerApp needs to be able to display terms from the selected group
TermManagerApp needs to be able to send same command to all terms in the displayed group TermManagerApp needs to be able to select a given term, and insert cmds to that term only Terms being displayed, should display "realtime" window update
cssh can do that.
Nice if termManagerApp can display 20-50 terms simultaneously
With cssh, this is a function of your screen and font sizes. You'll need a big screen and relatively small fonts. Too small for these old eyes of mine.
Basically, the tool/app will be used to allow the project to manage the multiple instances/VMs that are being run for the crawl.
Project use functions requires: -Running commands on remote servers -Checking on the progress of the remote processes via the screen function -Starting up/Shutting Down remote servers as needed -TBD..
You may also want to look at pdsh/dshbak (available from the repos) to send commands to remote machines. pdsh doesn't open windows as cssh does, but sends commands and receives the responses back. The responses are fed to dshbak to sort them into host order so they're not all jumbled together. Other tools to look at are ansible and puppet (both available from the repos) and there are others I haven't mentioned. All of them have their uses and strengths and weaknesses and there really isn't any one, single "best" solution. If there were, there wouldn't be this plethora of tools. I think you'll find you use several, depending on exactly what you're trying to accomplish at that particular time.
Note that the issues with ssh keys and such will rear their heads regardless of what tool(s) you use. If you want password-less access, each droplet needs to use the same ssh hostkey each time (and ideally the same IP) or you'll need to set the /etc/ssh/ssh_config or ~/.ssh/config file's "StrictHostKeyChecking" to "no" or "ask" (not necessarily a good thing--it depends on your level of paranoia).
- Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com -
- AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 -
-
Heisenberg _may_ have slept here -
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org