cluster-commits May 2012

cluster-commits@lists.fedorahosted.org

6 participants
130 discussions

cluster: RHEL6 - rgmanager: Add man page for cpglockd; update rgmanager.8
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=32… Commit: 320e12394d7010c139e271f4345d97a70eec2529 Parent: 2a5c98a455a28890c2e3e6c02d31e9f542369008 Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Tue May 1 10:18:58 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:59 2012 -0400 rgmanager: Add man page for cpglockd; update rgmanager.8 Signed-off-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Ryan McCabe <rmccabe(a)redhat.com> --- rgmanager/man/cpglockd.8 | 21 +++++++++++++++++++++ rgmanager/man/rgmanager.8 | 7 ++++++- 2 files changed, 27 insertions(+), 1 deletions(-) diff --git a/rgmanager/man/cpglockd.8 b/rgmanager/man/cpglockd.8 new file mode 100644 index 0000000..8084d96 --- /dev/null +++ b/rgmanager/man/cpglockd.8 @@ -0,0 +1,21 @@ +.TH "cpglockd" "8" "May 2012" "" "Red Hat High Availability" +.SH "NAME" +cpglockd \- CPG lock server for rgmanager +.SH "DESCRIPTION" +.PP +.B cpglockd +utilizes the extended virtual synchrony features of the Corosync +Cluster Engine to implement a simplistic, distributed lock server +for rgmanager. +.SH "COMMAND LINE OPTIONS" +.IP \-F +Don't wait for the current node to join the fencing domain at startup +.IP \-Q +Don't wait for quorum formation at startup +.IP \-f +Operate in the foreground mode; do not daemonize +.IP \-h +Print command line usage. + +.SH "SEE ALSO" +rgmanager(8), corosync(8) diff --git a/rgmanager/man/rgmanager.8 b/rgmanager/man/rgmanager.8 index 2b018d2..bab52ef 100644 --- a/rgmanager/man/rgmanager.8 +++ b/rgmanager/man/rgmanager.8 @@ -380,8 +380,13 @@ Do not perform stop-before-start. Combined with the .I -Z flag to clusvcadm, this can be used to allow rgmanager to be upgraded without stopping a given user service or set of services. +.IP \-C [0|1] +Explicitly disable or enable CPG-based locking. The default is to +enable this when RRP is turned on (which requires a cluster outage). +This option MUST be the same on all hosts in the cluster and must +only be enabled or disabled with all instances of rgmanager turned off. .SH "SEE ALSO" http://sources.redhat.com/cluster/wiki/RGManager -clusvcadm(8), cluster.conf(5) +clusvcadm(8), cluster.conf(5), cpglockd(8)

1 0

cluster: RHEL6 - rgmanager: Retry connection to cpglockd
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=2a… Commit: 2a5c98a455a28890c2e3e6c02d31e9f542369008 Parent: 899e2d3863fcdcc5f70d175d149b3b6ad084054c Author: Lon Hohberger <lhh(a)redhat.com> AuthorDate: Fri Apr 27 18:33:34 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:52 2012 -0400 rgmanager: Retry connection to cpglockd Signed-off-by: Lon Hohberger <lhh(a)redhat.com> Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> --- rgmanager/src/clulib/cpg_lock.c | 27 ++++++++++++++++++--------- 1 files changed, 18 insertions(+), 9 deletions(-) diff --git a/rgmanager/src/clulib/cpg_lock.c b/rgmanager/src/clulib/cpg_lock.c index d257cf4..7c1c8c3 100644 --- a/rgmanager/src/clulib/cpg_lock.c +++ b/rgmanager/src/clulib/cpg_lock.c @@ -105,18 +105,27 @@ _cpg_lock_finished(const char *name) int cpg_lock_initialize(void) { - int ret, err; + int ret, err, retries = 0; - pthread_mutex_lock(&_default_lock); - if (_cpgh) { + while (retries < 10) { + pthread_mutex_lock(&_default_lock); + if (_cpgh) { + pthread_mutex_unlock(&_default_lock); + return 0; + } + + cpg_lock_init(&_cpgh); + ret = (_cpgh == NULL); + err = errno; pthread_mutex_unlock(&_default_lock); - return 0; - } - cpg_lock_init(&_cpgh); - ret = (_cpgh == NULL); - err = errno; - pthread_mutex_unlock(&_default_lock); + if (ret == 0) + break; + else { + sleep(1); + ++retries; + } + } /* Set up function pointers */ clu_lock = _cpg_lock;

1 0

cluster: RHEL6 - Increment the recovered locks count when a client is disconnected.
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=89… Commit: 899e2d3863fcdcc5f70d175d149b3b6ad084054c Parent: 7b51b3a6806b9e438e47a455dc30fa0b59c6ec0d Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Fri Apr 27 18:31:47 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:46 2012 -0400 Increment the recovered locks count when a client is disconnected. Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index 5d0fed9..1a5832c 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -643,6 +643,7 @@ del_client(int fd) l->l.state = LOCK_FREE; strncpy(m.resource, l->l.resource, sizeof(m.resource)); + ++recovered; if (grant_next(&m) == 0) send_unlock(&m); }

1 0

cluster: RHEL6 - Replace printf and fprintf calls with logt_print in the dlm locking module.
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=7b… Commit: 7b51b3a6806b9e438e47a455dc30fa0b59c6ec0d Parent: aae5ea434ad766063a085f86d0d67faab7fde87d Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Fri Apr 27 18:30:54 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:42 2012 -0400 Replace printf and fprintf calls with logt_print in the dlm locking module. Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/clulib/dlm_lock.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/rgmanager/src/clulib/dlm_lock.c b/rgmanager/src/clulib/dlm_lock.c index d3f4250..0752b41 100644 --- a/rgmanager/src/clulib/dlm_lock.c +++ b/rgmanager/src/clulib/dlm_lock.c @@ -12,6 +12,7 @@ #include <sys/types.h> #include <sys/select.h> #include <pthread.h> +#include <logging.h> /* Default lockspace stuff */ static dlm_lshandle_t _default_ls = NULL; @@ -50,9 +51,8 @@ clu_ls_lock(dlm_lshandle_t ls, int ret; if (!ls || !lksb || !resource || !strlen(resource)) { - printf("%p %p %p %d\n", ls, lksb, resource, - (int)strlen(resource)); - printf("INVAL...\n"); + logt_print(LOG_DEBUG, "Invalid lock request: %p %p %p %d\n", + ls, lksb, resource, resource ? strlen(resource) : 0); errno = EINVAL; return -1; } @@ -69,8 +69,8 @@ clu_ls_lock(dlm_lshandle_t ls, } if ((ret = (wait_for_dlm_event(ls) < 0))) { - fprintf(stderr, "wait_for_dlm_event: %d / %d\n", - ret, errno); + logt_print(LOG_DEBUG, "wait_for_dlm_event: %d / %d\n", + ret, errno); return -1; } @@ -108,7 +108,7 @@ clu_open_lockspace(const char *lsname) if (errno == ENOENT) continue; - fprintf(stderr, "failed acquiring lockspace: %s\n", + logt_print(LOG_DEBUG, "failed acquiring lockspace: %s\n", strerror(errno)); return NULL;

1 0

cluster: RHEL6 - Poll for fenced node updates less aggressively while waiting for fencing to complete.
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=aa… Commit: aae5ea434ad766063a085f86d0d67faab7fde87d Parent: 4c63dc81ecaaa45aa7365a669a9b26fe0e636cc7 Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Thu Apr 26 14:16:12 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:37 2012 -0400 Poll for fenced node updates less aggressively while waiting for fencing to complete. Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 17 +++++++---------- 1 files changed, 7 insertions(+), 10 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index f101084..5d0fed9 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -1675,17 +1675,14 @@ main(int argc, char **argv) ** don't need to wait for it. */ if (!victim && !x && !pf_node->force_wait) { - int retries = 0; /* Wait up to 1s for fenced to set victim */ - do { - usleep(250000); - if (fenced_node_info(pf_node->nodeid, &fn) < 0) { - logt_print(LOG_DEBUG, - "Unable to get fenced data for node %d\n", - pf_node->nodeid); - } else - victim = fn.victim; - } while (!victim && retries++ < 4); + sleep(1); + if (fenced_node_info(pf_node->nodeid, &fn) < 0) { + logt_print(LOG_DEBUG, + "Unable to get fenced data for node %d\n", + pf_node->nodeid); + } else + victim = fn.victim; if (!victim) { logt_print(LOG_DEBUG, "First entry in list and victim == 0, removing %d from pending fencing\n", pf_node->nodeid);

1 0

cluster: RHEL6 - Work around a fenced hang/delay that can result in us unpausing too soon while waiting for fencing to complete.
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=4c… Commit: 4c63dc81ecaaa45aa7365a669a9b26fe0e636cc7 Parent: 7d8f2f6ceabaf11725f73a290c0ad73c4cf3fb58 Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Thu Apr 26 14:12:37 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:31 2012 -0400 Work around a fenced hang/delay that can result in us unpausing too soon while waiting for fencing to complete. Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 53 +++++++++++++++++++++++++------------- 1 files changed, 35 insertions(+), 18 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index fa40e98..f101084 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -40,6 +40,7 @@ struct lock_node { struct pending_fence_node { list_head(); int nodeid; + int force_wait; uint64_t fail_time; }; @@ -96,6 +97,22 @@ flag_shutdown(int __attribute__ ((unused)) sig) shutdown_pending = 1; } + +static int +is_member(uint32_t nodeid) +{ + struct member_node *n; + int x; + + list_for(&group_members, n, x) { + if (n->nodeid == nodeid) + return 1; + } + + return 0; +} + + static int cman_nodes_lost(cman_node_t *old_nodes, size_t old_node_len, @@ -221,6 +238,14 @@ cman_callback(cman_handle_t ch, void *privdata, int reason, int arg) pf = do_alloc(sizeof(*pf)); pf->nodeid = cur_nodeid; pf->fail_time = cur_time; + /* + ** If the node is also a member of the cpglock group, wait + ** for positive confirmation from fenced that it was fenced. + ** It cannot have shut down cleanly if we did not process a + ** DELETE for it yet. + */ + if (is_member(cur_nodeid)) + pf->force_wait = 1; list_append(&pending_fencing, pf); } else { logt_print(LOG_DEBUG, "Lost node %d but fencing not configured\n", @@ -904,21 +929,6 @@ process_lock(struct cpg_lock_msg *m) static int -is_member(uint32_t nodeid) -{ - struct member_node *n; - int x; - - list_for(&group_members, n, x) { - if (n->nodeid == nodeid) - return 1; - } - - return 0; -} - - -static int process_grant(struct cpg_lock_msg *m, uint32_t nodeid) { struct lock_node *l; @@ -1645,7 +1655,8 @@ main(int argc, char **argv) } if (lft > pf_node->fail_time) { - logt_print(LOG_DEBUG, "Fencing for node %d finished at %ld (>%ld)\n", + logt_print(LOG_DEBUG, + "Fencing for node %d finished at %ld (>%ld)\n", pf_node->nodeid, lft, pf_node->fail_time); list_remove(&pending_fencing, pf_node); free(pf_node); @@ -1663,13 +1674,14 @@ main(int argc, char **argv) ** victim to 1 by now, we can deduce it has left cleanly, and we ** don't need to wait for it. */ - if (!victim && !x) { + if (!victim && !x && !pf_node->force_wait) { int retries = 0; /* Wait up to 1s for fenced to set victim */ do { usleep(250000); if (fenced_node_info(pf_node->nodeid, &fn) < 0) { - logt_print(LOG_DEBUG, "Unable to get fenced data for node %d\n", + logt_print(LOG_DEBUG, + "Unable to get fenced data for node %d\n", pf_node->nodeid); } else victim = fn.victim; @@ -1685,6 +1697,11 @@ main(int argc, char **argv) } goto fence_check; } + if (!victim && !x && pf_node->force_wait) { + logt_print(LOG_DEBUG, "Would have removed %d but now waiting\n", + pf_node->nodeid); + } + } if (shutdown_pending)

1 0

cluster: RHEL6 - rgmanager: fix conditional check for cpglockd startup
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=7d… Commit: 7d8f2f6ceabaf11725f73a290c0ad73c4cf3fb58 Parent: b87c913695c90af199b2335bdc5d117f658cd132 Author: Fabio M. Di Nitto <fdinitto(a)redhat.com> AuthorDate: Thu Apr 26 15:50:34 2012 +0200 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:27 2012 -0400 rgmanager: fix conditional check for cpglockd startup Signed-off-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Ryan McCabe <rmccabe(a)redhat.com> --- rgmanager/init.d/rgmanager.in | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/rgmanager/init.d/rgmanager.in b/rgmanager/init.d/rgmanager.in index e602ba0..8a8c6e3 100644 --- a/rgmanager/init.d/rgmanager.in +++ b/rgmanager/init.d/rgmanager.in @@ -78,8 +78,8 @@ stop_cluster() start_cpglockd() { rings="$(corosync-objctl 2>/dev/null |grep ringnumber | wc -l)" - [ "$rings" -gt "1" ] && service cpglockd start - while ! cpglockdump > /dev/null 2>&1; do sleep 1; done + [ "$rings" -gt "1" ] && service cpglockd start && \ + while ! cpglockdump > /dev/null 2>&1; do sleep 1; done } rtrn=0

1 0

cluster: RHEL6 - Copy lock messages before swapping bytes because we may use the lock message struct again after calling send_lock_msg()
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=b8… Commit: b87c913695c90af199b2335bdc5d117f658cd132 Parent: ef03894f10f1ffc51422dd89a310e887c043e543 Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Wed Apr 25 21:23:21 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:22 2012 -0400 Copy lock messages before swapping bytes because we may use the lock message struct again after calling send_lock_msg() Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index df18eb3..fa40e98 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -427,11 +427,13 @@ send_lock_msg(struct cpg_lock_msg *m) { struct iovec iov; int ret; + struct cpg_lock_msg out_msg; - swab_cpg_lock_msg(m); + memcpy(&out_msg, m, sizeof(out_msg)); + swab_cpg_lock_msg(&out_msg); - iov.iov_base = m; - iov.iov_len = sizeof (*m); + iov.iov_base = &out_msg; + iov.iov_len = sizeof(out_msg); do { ret = cpg_mcast_joined(cpg, CPG_TYPE_AGREED, &iov, 1);

1 0

cluster: RHEL6 - Set m.owner_nodeid to the our node id when NAKing requests when the cluster is inquorate.
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=ef… Commit: ef03894f10f1ffc51422dd89a310e887c043e543 Parent: 84afab18b0e17080dcc7868060231d3282d4ab51 Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Wed Apr 25 15:31:09 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:17 2012 -0400 Set m.owner_nodeid to the our node id when NAKing requests when the cluster is inquorate. Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index 1f69ef8..df18eb3 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -1736,11 +1736,13 @@ main(int argc, char **argv) /* XXX check for dup connection */ if (m.request == MSG_LOCK) { if (!cluster_quorate) { + m.request = MSG_NAK; + m.owner_nodeid = my_node_id; + logt_print(LOG_DEBUG, "Sending NAK for new lock request while not quorate (%s [%d:%d:%d])\n", m.resource, m.owner_nodeid, m.owner_pid, m.owner_tid); - m.request = MSG_NAK; if (write_retry(client->fd, &m, sizeof(m), NULL) < 0) { logt_print(LOG_DEBUG, "Error sending NAK for %s [%d:%d:%d]: %s\n",

1 0

cluster: RHEL6 - Add some additional debugging logging for try-locks
by Ryan McCabe 01 May '12

01 May '12

Gitweb: http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=84… Commit: 84afab18b0e17080dcc7868060231d3282d4ab51 Parent: 15993079738bf0e07e93c4f07e1816388654e283 Author: Ryan McCabe <rmccabe(a)redhat.com> AuthorDate: Wed Apr 25 13:07:54 2012 -0400 Committer: Ryan McCabe <rmccabe(a)redhat.com> CommitterDate: Tue May 1 11:00:13 2012 -0400 Add some additional debugging logging for try-locks Signed-off-by: Ryan McCabe <rmccabe(a)redhat.com> Reviewed-by: Lon Hohberger <lhh(a)redhat.com> Reviewed-by: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- rgmanager/src/daemons/cpglockd.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/rgmanager/src/daemons/cpglockd.c b/rgmanager/src/daemons/cpglockd.c index 689c494..1f69ef8 100644 --- a/rgmanager/src/daemons/cpglockd.c +++ b/rgmanager/src/daemons/cpglockd.c @@ -478,6 +478,8 @@ send_grant(struct request_node *n) static int send_nak(struct cpg_lock_msg *m) { + logt_print(LOG_DEBUG, "-> sending NAK %s to %d:%d:%d\n", + m->resource, m->owner_nodeid, m->owner_pid, m->owner_tid); m->request = MSG_NAK; return send_lock_msg(m); @@ -1013,6 +1015,9 @@ process_nak(struct cpg_lock_msg *m, uint32_t nodeid) if (!joined) return 0; + logt_print(LOG_DEBUG, "NAK %s for %d:%d:%d\n", m->resource, + m->owner_nodeid, m->owner_pid, m->owner_tid); + list_for(&requests, r, y) { if (strcmp(r->l.resource, m->resource)) continue;

1 0

← Newer
1
2
3
4
5
6
7
8
9
...
13
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

cluster-commits May 2012