Hunt Xu has uploaded a new change for review.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
vdsm_reg: make getRemoteFile IPv6-capable
This patch also tidies the getRemoteFile function's codes, making some of the helper functions reusable.
Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Signed-off-by: huntxu mhuntxu@gmail.com --- M vdsm_reg/deployUtil.py.in 1 file changed, 94 insertions(+), 57 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/42/11742/1
diff --git a/vdsm_reg/deployUtil.py.in b/vdsm_reg/deployUtil.py.in index f2bef9d..7a305d2 100644 --- a/vdsm_reg/deployUtil.py.in +++ b/vdsm_reg/deployUtil.py.in @@ -483,6 +483,97 @@ return strReturn
+def getRemoteSock(IP, port): + """ Return a tcp stream socket connect to ip:port """ + try: + af, socktype, proto, canonname, sockaddr = \ + socket.getaddrinfo(IP, port, 0, + socket.SOCK_STREAM, socket.SOL_TCP)[0] + sock = socket.socket(af, socket.SOCK_STREAM, socket.SOL_TCP) + sock.connect(sockaddr) + return sock + except IOError as e: + logging.error("getRemoteSock: Could not connect to %s port %s.(%s)" % + IP, port, e) + return None + + +def httpGETRequest(conn, uri): + response = None + + # N.B.: for python version < 2.7, http://bugs.python.org/issue5111 + httpHeaders = {} + + try: + if conn.sock.family == socket.AF_INET6: + httpHeaders = {'Host': '[%s]:%d' % conn.sock.getpeername()[:2]} + conn.request("GET", uri, None, httpHeaders) + response = conn.getresponse() + except: + pass + return response + + +def getHTTPResource(IP, port, uri, timeout=None, certPath=None): + data = None + response = None + conn = None + + fOK = True + + try: + nPort = 443 + if port is not None: + nPort = int(port) + + sock = getRemoteSock(IP, nPort) + sock.settimeout(timeout) + conn = httplib.HTTPSConnection(IP, nPort) + conn.sock = getSSLSocket(sock, certPath) + response = httpGETRequest(conn, uri) + except: + logging.debug("Getting %s from %s:%s failed in HTTPS. Retrying using" + " HTTP.", uri, IP, port, exc_info=True) + nPort = 80 + if port is not None: + nPort = int(port) + + try: + conn = httplib.HTTPConnection(IP, nPort) + conn.sock = getRemoteSock(IP, nPort) + response = httpGETRequest(conn, uri) + except: + logging.error("Failed to get %s from %s:%s using http.", uri, + IP, port, exc_info=True) + fOK = False + else: + logging.debug("Getting %s from %s:%s status: %s reason: %s", + uri, IP, port, str(response.status), response.reason) + + if response is None or response.status != 200: + status = "" + if response is not None: + status = str(response.status) + if conn is not None: + conn.close() + fOK = False + logging.error("Failed to GET %s from %s:%s status %s", + uri, IP, port, status) + + if fOK: + try: + try: + data = str(response.read()) + except: + logging.error("Failed to read %s from %s:%s", + uri, IP, port, exc_info=True) + finally: + if conn is not None: + conn.close() + + return data + + def preventDuplicate(bridgeName=None): """ This function checks if the needed bridge (@VDSMBRIDGE@) already exist. @@ -1567,63 +1658,9 @@ def getRemoteFile(IP, port, fileName, timeout=None, certPath=None): logging.debug("getRemoteFile start. IP = %s port = %s fileName = "%s"" % (IP, port, fileName)) - data = None - response = None - conn = None - sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - sock.settimeout(timeout) - fOK = True - - try: - nPort = 443 - if port is not None: - nPort = int(port) - - sock.connect((IP, nPort)) - conn = httplib.HTTPSConnection(IP, nPort) - conn.sock = getSSLSocket(sock, certPath) - conn.request("GET", fileName) - response = conn.getresponse() - except: - logging.debug("%s failed in HTTPS. Retrying using HTTP.", fileName, - exc_info=True) - strPort = ":" - if port is None: - strPort += "80" - else: - strPort += port - - try: - conn = httplib.HTTPConnection(IP + strPort) - conn.request("GET", fileName) - response = conn.getresponse() - except: - logging.error("Failed to fetch %s using http.", fileName, - exc_info=True) - fOK = False - else: - logging.debug("%s status: %s reason: %s", - fileName, str(response.status), response.reason) - - if response is None or response.status != 200: - status = "" - if response is not None: - status = str(response.status) - if conn is not None: - conn.close() - fOK = False - logging.error("Failed to fetch %s status %s", fileName, status) - - if fOK: - try: - try: - data = str(response.read()) - except: - logging.error("Failed to read %s", fileName, exc_info=True) - finally: - if conn is not None: - conn.close() - + data = getHTTPResource(IP, port, fileName, timeout, certPath) + if data is None: + logging.error("getRemoteFile: Failed to get %s." % fileName) logging.debug('getRemoteFile end.') return data
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/249/ (1/3)
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1138/ (2/3)
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1103/ (3/3)
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: Alon Bar-Lev alonbl@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1:
Build Successful
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/1103/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1138/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/249/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: Alon Bar-Lev alonbl@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Alon Bar-Lev has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1: (4 inline comments)
.................................................... File vdsm_reg/deployUtil.py.in Line 497: IP, port, e) Line 498: return None Line 499: Line 500: Line 501: def httpGETRequest(conn, uri): Can't this be inner function of gteHTTPResource? Line 502: response = None Line 503: Line 504: # N.B.: for python version < 2.7, http://bugs.python.org/issue5111 Line 505: httpHeaders = {}
Line 505: httpHeaders = {} Line 506: Line 507: try: Line 508: if conn.sock.family == socket.AF_INET6: Line 509: httpHeaders = {'Host': '[%s]:%d' % conn.sock.getpeername()[:2]} Isn't the httplib set the host header?
As you changed the location within source I cannot review the changes, I guess this is the only change, right? Line 510: conn.request("GET", uri, None, httpHeaders) Line 511: response = conn.getresponse() Line 512: except: Line 513: pass
Line 527: nPort = int(port) Line 528: Line 529: sock = getRemoteSock(IP, nPort) Line 530: sock.settimeout(timeout) Line 531: conn = httplib.HTTPSConnection(IP, nPort) Can we do better and connect using TLS or plain based on parameter?
The fallback from https to http was and is bad.
Well, not that it is important right now as it is the legacy implementation. Line 532: conn.sock = getSSLSocket(sock, certPath) Line 533: response = httpGETRequest(conn, uri) Line 534: except: Line 535: logging.debug("Getting %s from %s:%s failed in HTTPS. Retrying using"
Line 533: response = httpGETRequest(conn, uri) Line 534: except: Line 535: logging.debug("Getting %s from %s:%s failed in HTTPS. Retrying using" Line 536: " HTTP.", uri, IP, port, exc_info=True) Line 537: nPort = 80 We can definitely clean up the code. Line 538: if port is not None: Line 539: nPort = int(port) Line 540: Line 541: try:
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: Alon Bar-Lev alonbl@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Hunt Xu has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1: (4 inline comments)
.................................................... File vdsm_reg/deployUtil.py.in Line 497: IP, port, e) Line 498: return None Line 499: Line 500: Line 501: def httpGETRequest(conn, uri): Yes indeed. However this helper function is intended for python < v2.7 that we have to manually wrap a HTTP request. Once vdsm drops python<2.7(currently on Centos 6) dependency, we can simply remove the wrapping. Line 502: response = None Line 503: Line 504: # N.B.: for python version < 2.7, http://bugs.python.org/issue5111 Line 505: httpHeaders = {}
Line 505: httpHeaders = {} Line 506: Line 507: try: Line 508: if conn.sock.family == socket.AF_INET6: Line 509: httpHeaders = {'Host': '[%s]:%d' % conn.sock.getpeername()[:2]} For the first question, please refer the URL I gave in the comment.
Right, almost location changes and make it helper function for reusing in another patch. Line 510: conn.request("GET", uri, None, httpHeaders) Line 511: response = conn.getresponse() Line 512: except: Line 513: pass
Line 527: nPort = int(port) Line 528: Line 529: sock = getRemoteSock(IP, nPort) Line 530: sock.settimeout(timeout) Line 531: conn = httplib.HTTPSConnection(IP, nPort) Are you saying "is bad" because the try-except?
In fact I also have no idea about how to do it better so I just leave it as-is. Line 532: conn.sock = getSSLSocket(sock, certPath) Line 533: response = httpGETRequest(conn, uri) Line 534: except: Line 535: logging.debug("Getting %s from %s:%s failed in HTTPS. Retrying using"
Line 533: response = httpGETRequest(conn, uri) Line 534: except: Line 535: logging.debug("Getting %s from %s:%s failed in HTTPS. Retrying using" Line 536: " HTTP.", uri, IP, port, exc_info=True) Line 537: nPort = 80 Yes. I'll try to work out a better solution. Line 538: if port is not None: Line 539: nPort = int(port) Line 540: Line 541: try:
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: Alon Bar-Lev alonbl@redhat.com Gerrit-Reviewer: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: oVirt Jenkins CI Server
Alon Bar-Lev has posted comments on this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Patch Set 1: (1 inline comment)
.................................................... File vdsm_reg/deployUtil.py.in Line 497: IP, port, e) Line 498: return None Line 499: Line 500: Line 501: def httpGETRequest(conn, uri): Thanks!
I think it is better to use a workaround only if needed... ask for python version and install the workaround as monkey patch.
No that this is so important for this legacy code... and this code is not required for the ovirt-host-deploy.
Anyway better to move this as internal function and add '_' prefix to mark as private. Line 502: response = None Line 503: Line 504: # N.B.: for python version < 2.7, http://bugs.python.org/issue5111 Line 505: httpHeaders = {}
-- To view, visit http://gerrit.ovirt.org/11742 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I50cc10f3d5c5baef343fe2456c775001f712da6c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: Alon Bar-Lev alonbl@redhat.com Gerrit-Reviewer: Hunt Xu mhuntxu@gmail.com Gerrit-Reviewer: oVirt Jenkins CI Server
Itamar Heim has abandoned this change.
Change subject: vdsm_reg: make getRemoteFile IPv6-capable ......................................................................
Abandoned
abandoning stale patch - please re-open if relevant
vdsm-patches@lists.fedorahosted.org