Hi Jeff et al,
If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression.
I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds.
The changes I have made (against 3.9):
*File*: suds/transport/http.py: * * *Extra imports*:
import gzip import StringIO
*Class*: HttpTransport *Method*: send
*Added:*
After:
- headers = request.headers
added the following line
- headers.update([('Accept-Encoding', 'gzip')])
*Changed:*
- result = Reply(200, fp.headers.dict, fp.read())
to
- replydata = fp.read() - for header in fp.headers.dict.keys(): - if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': - replydatacompressedstream = StringIO.StringIO(replydata) - gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) - replydata = gzipper.read() - result = Reply(200, fp.headers.dict, replydata)
Best regards
Daniel
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff et al,
If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression.
I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds.
The changes I have made (against 3.9):
*File*: suds/transport/http.py:
*Extra imports*:
import gzip import StringIO
*Class*: HttpTransport *Method*: send
*Added:*
After:
- headers = request.headers
added the following line
- headers.update([('Accept-Encoding', 'gzip')])
*Changed:*
- result = Reply(200, fp.headers.dict, fp.read())
to
- replydata = fp.read()
- for header in fp.headers.dict.keys():
if header.lower() == 'content-encoding' and
fp.headers.dict[header] == 'gzip':
replydatacompressedstream = StringIO.StringIO(replydata)
gzipper = gzip.GzipFile(fileobj=replydatacompressedstream)
replydata = gzipper.read()
- result = Reply(200, fp.headers.dict, replydata)
Best regards
Daniel
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez danjrod@gmail.com wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff et al,
If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression.
I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds.
The changes I have made (against 3.9):
*File*: suds/transport/http.py:
*Extra imports*:
import gzip import StringIO
*Class*: HttpTransport *Method*: send
*Added:*
After:
- headers = request.headers
added the following line
- headers.update([('Accept-Encoding', 'gzip')])
*Changed:*
- result = Reply(200, fp.headers.dict, fp.read())
to
- replydata = fp.read()
- for header in fp.headers.dict.keys():
if header.lower() == 'content-encoding' and
fp.headers.dict[header] == 'gzip':
replydatacompressedstream = StringIO.StringIO(replydata)
gzipper = gzip.GzipFile(fileobj=replydatacompressedstream)
replydata = gzipper.read()
- result = Reply(200, fp.headers.dict, replydata)
Best regards
Daniel
Hi Daniel,
FWIW, you could use httplib2 for the transport - that would give you compression, redirects and keep-alive support for free.
--Dave
On Thu, Apr 29, 2010 at 10:53 PM, Daniel Rodriguez danjrod@gmail.com wrote:
Hello, I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py Requesting compression can be turned off and on with the "gzip" option (a bool) The gzip decompression code works always. The reasoning behind this is:
Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards Daniel On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez danjrod@gmail.com wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers. Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): File: suds/transport/http.py: Extra imports: import gzip import StringIO Class: HttpTransport Method: send Added: After:
headers = request.headers
added the following line
headers.update([('Accept-Encoding', 'gzip')])
Changed:
result = Reply(200, fp.headers.dict, fp.read())
to
replydata = fp.read() for header in fp.headers.dict.keys(): if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': replydatacompressedstream = StringIO.StringIO(replydata) gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) replydata = gzipper.read() result = Reply(200, fp.headers.dict, replydata)
Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Hi,
Well, I initially wanted to add the gzip support for the standard transport created by Jeff. The reason being that it is pretty well integrated with the overall architecture and most people (including myself) wouldn't probably care about using any other transport for quick hacks and short scripts.
In theory with the asynchronous usage patch I have provided, using the library with any other transport should be trivial, given that now suds may act as a request generator and response processor, with no intervening network support.
I have started playing with urllib3 (mentioned on the list) to create a transport based around that library and that has uncovered a bug and in my opinion a possible need to do some rework of the http and network error handling.
urlllib3 has also needed some extra work to add proxy support and I am still doing some work on the error/exception handling and preliminary support for authentication (server/proxy and basic/digest)
What I was thinking is to ask Jeff to open a contrib directory in the sources, so that if people contribute things like my urllib3 transport, any user of the library may choose to import it and use it (but Jeff is probably busy with many other things as we all are)
Anyhow, thank you becaue I will have a look at Httplib2
Best regards
Daniel
On Wed, May 5, 2010 at 03:27, David Robinson zxvdr.au@gmail.com wrote:
Hi Daniel,
FWIW, you could use httplib2 for the transport - that would give you compression, redirects and keep-alive support for free.
--Dave
On Thu, Apr 29, 2010 at 10:53 PM, Daniel Rodriguez danjrod@gmail.com wrote:
Hello, I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py Requesting compression can be turned off and on with the "gzip" option (a bool) The gzip decompression code works always. The reasoning behind this is:
Some network providers may compress the data in HTTP responses in order
to
decrease network traffic, expecting all clients to be ready to decompress data
Best regards Daniel On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez danjrod@gmail.com
wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers. Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez danjrod@gmail.com
wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is
not
using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be
possibly
added as an option to suds. The changes I have made (against 3.9): File: suds/transport/http.py: Extra imports: import gzip import StringIO Class: HttpTransport Method: send Added: After:
headers = request.headers
added the following line
headers.update([('Accept-Encoding', 'gzip')])
Changed:
result = Reply(200, fp.headers.dict, fp.read())
to
replydata = fp.read() for header in fp.headers.dict.keys(): if header.lower() == 'content-encoding' and
fp.headers.dict[header]
== 'gzip': replydatacompressedstream = StringIO.StringIO(replydata) gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) replydata = gzipper.read() result = Reply(200, fp.headers.dict, replydata)
Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Fair enough, gzip support is a nice addition :-)
Your async patches look interesting. I started using httplib2 for its keep-alive and compression support but have been looking for something that can do pipelining too - your async patches should help solve that problem (not sure whether urllib3 does, but I'll look into that too).
On Wed, May 5, 2010 at 1:53 PM, Daniel Rodriguez danjrod@gmail.com wrote:
Hi, Well, I initially wanted to add the gzip support for the standard transport created by Jeff. The reason being that it is pretty well integrated with the overall architecture and most people (including myself) wouldn't probably care about using any other transport for quick hacks and short scripts. In theory with the asynchronous usage patch I have provided, using the library with any other transport should be trivial, given that now suds may act as a request generator and response processor, with no intervening network support. I have started playing with urllib3 (mentioned on the list) to create a transport based around that library and that has uncovered a bug and in my opinion a possible need to do some rework of the http and network error handling. urlllib3 has also needed some extra work to add proxy support and I am still doing some work on the error/exception handling and preliminary support for authentication (server/proxy and basic/digest) What I was thinking is to ask Jeff to open a contrib directory in the sources, so that if people contribute things like my urllib3 transport, any user of the library may choose to import it and use it (but Jeff is probably busy with many other things as we all are) Anyhow, thank you becaue I will have a look at Httplib2 Best regards Daniel On Wed, May 5, 2010 at 03:27, David Robinson zxvdr.au@gmail.com wrote:
Hi Daniel,
FWIW, you could use httplib2 for the transport - that would give you compression, redirects and keep-alive support for free.
--Dave
On Thu, Apr 29, 2010 at 10:53 PM, Daniel Rodriguez danjrod@gmail.com wrote:
Hello, I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py Requesting compression can be turned off and on with the "gzip" option (a bool) The gzip decompression code works always. The reasoning behind this is:
Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards Daniel On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez danjrod@gmail.com wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers. Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): File: suds/transport/http.py: Extra imports: import gzip import StringIO Class: HttpTransport Method: send Added: After:
headers = request.headers
added the following line
headers.update([('Accept-Encoding', 'gzip')])
Changed:
result = Reply(200, fp.headers.dict, fp.read())
to
replydata = fp.read() for header in fp.headers.dict.keys(): if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': replydatacompressedstream = StringIO.StringIO(replydata) gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) replydata = gzipper.read() result = Reply(200, fp.headers.dict, replydata)
Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
* Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers. Best regards On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com <mailto:danjrod@gmail.com>> wrote: Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Hi Jeff,
The following will let you test it.
import suds l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Hi Jeff,
As stated I have worked in the transport module as I said, as I believed there was room for improvement. Please find it attached for your consideration and for anyone else that may be willing to test it (the options.py and http.py can directly replace the existing files in suds/transport and be used straight away)
I have run a series of tests with compression enabled and disabled, tested sites with gzip and deflate (found none with bzip2) and wsdl in files or in urls. So far, everything worked and since they have many more changes than just the compression, I think they deserve their own ticket, for you to choose which one to accept (if any)
In options.py
- Variable: "compression" added.
It may be assigned the following values:
- yes: request compression and decompress incoming compressed content - no: do not request compression and do not decompress content even if present in the reply - auto: do not request compression but decompress it if present in the reply
Default: yes
- Variable: compmethods
A list with the compression methods to be supported.
So far: gzip, deflate, bzip2
The changes in http.py are:
- open
The function is now urllib2 agnostic. It will try to detect if the passed url is a file, a url or data and act accordingly.
This allows passing a variable with the WSDL content inside, besides the current functionality (a url)
Still returning a "file-like" object
- send
The function is now urllib2 agnostic. It simply calls the appropriate function
I think that both could be moved upward to __init__.py to the Transport class, given its library independence
- invoke (possibly a better name can be found)
New function that is called by open and send to perform http. This should be in the base class Transport and be redefined by new transports like this
This function is the one working with urllib2 directly or through the functions you had already provided
It may return a suds.transport.Reply or a file-like object for open
It does call two new functions that in my opinion could make it to the standard interface of Transport
- prerequest - postreply
This should be called before sending the request to the network and before returning the reply to the caller.
I have used them to "request" compression and decompress if needed. Another example would be "HttpAuthenticated", where you redefine open and send to add the credentials. This could go into "prerequest"
- addcookies and getcookies
I think the could be removed since they are one-liners
- u2opener
As stated in my comments in the code, I think there was a "bug" in the original code since self.urlopener was never being used and a new urlopener was being created for each and every call
Now a urlopener is created if needed or if the proxy options change during execution.
This could be improved by just checking if the proxy for a given scheme (http, https) has changed, but would require passing the url to this function
- u2handlers
Now a copy of self.options.proxy is made (if needed) to detect if they change during execution
- prerequest and postreply as aforementioned
Let me know what you think
Best regards
Daniel
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Dear sudsers,
Someone created ticket 320 indicating that he could not use of either the patches included in ticket 311 or the full files attached to ticket 318 (and to the previous e-mail in this chain). I added a comment to ticket 320 but no response has been added, so.
Small summary of how to use the transport in 320:
- Download suds (I would suggest "svn export http://svn.fedorahosted.org/svn/suds/trunk" better than checking out as indicated, to avoid pulling the svn control directories) - Download http.py and options.py from ticket 318 - https://fedorahosted.org/suds/attachment/ticket/318/http.py - https://fedorahosted.org/suds/attachment/ticket/318/options.py - Replace suds/transport/http.py and suds/transport/options.py with the files just downloaded - Use suds as normal.
Anyhow and to make the rewritten transport usable without having to tweak the suds installation, I am attaching a version that can be used from outside. The usage is as follows:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport myclient = suds.client.Client(wsdlcontent, transport = mytransport)
The transport works with the starndard transport options available in suds and because it is working outside of the suds structure, it does manually add the options that were mentioned in the previous e-mail:
- Option: "compression" added.
A string that may be assigned the following values:
- yes: request compression and decompress incoming compressed content - no: do not request compression and do not decompress content even if present in the reply - auto: do not request compression but decompress if compression is present in the reply
Default: 'yes'
- Option: compmethods
A list of strings with the compression methods to be supported.
The supported methods: gzip, deflate, bzip2
Default: ['gzip', 'deflate', 'bzip']
An example disabling compression would be:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport # Disable the transport compression for debugging purposes myclient = suds.client.Client(wsdlcontent, transport = mytransport, compression = 'no')
Or just testing the presence of bzip2 for example:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport # Enable only bzip2 compression to test that myclient = suds.client.Client(wsdlcontent, transport = mytransport, compmethods = ['bzip2'])
Please let me know if there is any problem.
Best regards
Daniel
On Thu, May 6, 2010 at 23:51, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
As stated I have worked in the transport module as I said, as I believed there was room for improvement. Please find it attached for your consideration and for anyone else that may be willing to test it (the options.py and http.py can directly replace the existing files in suds/transport and be used straight away)
I have run a series of tests with compression enabled and disabled, tested sites with gzip and deflate (found none with bzip2) and wsdl in files or in urls. So far, everything worked and since they have many more changes than just the compression, I think they deserve their own ticket, for you to choose which one to accept (if any)
In options.py
- Variable: "compression" added.
It may be assigned the following values:
- yes: request compression and decompress incoming compressed content
- no: do not request compression and do not decompress content even if
present in the reply
- auto: do not request compression but decompress it if present in the
reply
Default: yes
- Variable: compmethods
A list with the compression methods to be supported.
So far: gzip, deflate, bzip2
The changes in http.py are:
- open
The function is now urllib2 agnostic. It will try to detect if the passed url is a file, a url or data and act accordingly.
This allows passing a variable with the WSDL content inside, besides the current functionality (a url)
Still returning a "file-like" object
- send
The function is now urllib2 agnostic. It simply calls the appropriate function
I think that both could be moved upward to __init__.py to the Transport class, given its library independence
- invoke (possibly a better name can be found)
New function that is called by open and send to perform http. This should be in the base class Transport and be redefined by new transports like this
This function is the one working with urllib2 directly or through the functions you had already provided
It may return a suds.transport.Reply or a file-like object for open
It does call two new functions that in my opinion could make it to the standard interface of Transport
- prerequest
- postreply
This should be called before sending the request to the network and before returning the reply to the caller.
I have used them to "request" compression and decompress if needed. Another example would be "HttpAuthenticated", where you redefine open and send to add the credentials. This could go into "prerequest"
- addcookies and getcookies
I think the could be removed since they are one-liners
- u2opener
As stated in my comments in the code, I think there was a "bug" in the original code since self.urlopener was never being used and a new urlopener was being created for each and every call
Now a urlopener is created if needed or if the proxy options change during execution.
This could be improved by just checking if the proxy for a given scheme (http, https) has changed, but would require passing the url to this function
- u2handlers
Now a copy of self.options.proxy is made (if needed) to detect if they change during execution
- prerequest and postreply as aforementioned
Let me know what you think
Best regards
Daniel
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL ' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL ' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
Dear all,
Following some e-mail exchange with Paul, it seems that the logging code was triggering an exception in his case. I have slightly altered the logging code in tickets 318 to avoid collision with the code in suds/transport/__init__.py
(I have closed ticket 311, since these were patches)
Additionally and to ticket 320 I have added the transport (DRoHttpTransport) with the same modifications as above, that can be used without patching the original suds installation. Please follow the instructions found below in the previous e-mails.
I guess Paul will be testing it (I would suggest the 320 approach as it allows you to have a clean suds installation by patching the suds.options on the fly rather than by patching the options file)
As always, I'd be glad to also hear from anyone else using this, in case any other error happened to appear.
Best regards
Daniel
On Sun, May 16, 2010 at 22:28, Daniel Rodriguez danjrod@gmail.com wrote:
Dear sudsers,
Someone created ticket 320 indicating that he could not use of either the patches included in ticket 311 or the full files attached to ticket 318 (and to the previous e-mail in this chain). I added a comment to ticket 320 but no response has been added, so.
Small summary of how to use the transport in 320:
- Download suds (I would suggest "svn export
http://svn.fedorahosted.org/svn/suds/trunk" better than checking out as indicated, to avoid pulling the svn control directories)
- Download http.py and options.py from ticket 318
- Replace suds/transport/http.py and suds/transport/options.py with the
files just downloaded
- Use suds as normal.
Anyhow and to make the rewritten transport usable without having to tweak the suds installation, I am attaching a version that can be used from outside. The usage is as follows:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport myclient = suds.client.Client(wsdlcontent, transport = mytransport)
The transport works with the starndard transport options available in suds and because it is working outside of the suds structure, it does manually add the options that were mentioned in the previous e-mail:
- Option: "compression" added.
A string that may be assigned the following values:
- yes: request compression and decompress incoming compressed content
- no: do not request compression and do not decompress content even if
present in the reply
- auto: do not request compression but decompress if compression is
present in the reply
Default: 'yes'
- Option: compmethods
A list of strings with the compression methods to be supported.
The supported methods: gzip, deflate, bzip2
Default: ['gzip', 'deflate', 'bzip']
An example disabling compression would be:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport # Disable the transport compression for debugging purposes myclient = suds.client.Client(wsdlcontent, transport = mytransport, compression = 'no')
Or just testing the presence of bzip2 for example:
import DRoHttp
mytransport = DRoHttp.DRoHttpTransport # Enable only bzip2 compression to test that myclient = suds.client.Client(wsdlcontent, transport = mytransport, compmethods = ['bzip2'])
Please let me know if there is any problem.
Best regards
Daniel
On Thu, May 6, 2010 at 23:51, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
As stated I have worked in the transport module as I said, as I believed there was room for improvement. Please find it attached for your consideration and for anyone else that may be willing to test it (the options.py and http.py can directly replace the existing files in suds/transport and be used straight away)
I have run a series of tests with compression enabled and disabled, tested sites with gzip and deflate (found none with bzip2) and wsdl in files or in urls. So far, everything worked and since they have many more changes than just the compression, I think they deserve their own ticket, for you to choose which one to accept (if any)
In options.py
- Variable: "compression" added.
It may be assigned the following values:
- yes: request compression and decompress incoming compressed content
- no: do not request compression and do not decompress content even if
present in the reply
- auto: do not request compression but decompress it if present in the
reply
Default: yes
- Variable: compmethods
A list with the compression methods to be supported.
So far: gzip, deflate, bzip2
The changes in http.py are:
- open
The function is now urllib2 agnostic. It will try to detect if the passed url is a file, a url or data and act accordingly.
This allows passing a variable with the WSDL content inside, besides the current functionality (a url)
Still returning a "file-like" object
- send
The function is now urllib2 agnostic. It simply calls the appropriate function
I think that both could be moved upward to __init__.py to the Transport class, given its library independence
- invoke (possibly a better name can be found)
New function that is called by open and send to perform http. This should be in the base class Transport and be redefined by new transports like this
This function is the one working with urllib2 directly or through the functions you had already provided
It may return a suds.transport.Reply or a file-like object for open
It does call two new functions that in my opinion could make it to the standard interface of Transport
- prerequest
- postreply
This should be called before sending the request to the network and before returning the reply to the caller.
I have used them to "request" compression and decompress if needed. Another example would be "HttpAuthenticated", where you redefine open and send to add the credentials. This could go into "prerequest"
- addcookies and getcookies
I think the could be removed since they are one-liners
- u2opener
As stated in my comments in the code, I think there was a "bug" in the original code since self.urlopener was never being used and a new urlopener was being created for each and every call
Now a urlopener is created if needed or if the proxy options change during execution.
This could be improved by just checking if the proxy for a given scheme (http, https) has changed, but would require passing the url to this function
- u2handlers
Now a copy of self.options.proxy is made (if needed) to detect if they change during execution
- prerequest and postreply as aforementioned
Let me know what you think
Best regards
Daniel
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = ' http://webservices.daehosting.com/services/isbnservice.wso?WSDL' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez danjrod@gmail.com wrote:
Hi Jeff,
The following will let you test it.
import suds l_wsdl = ' http://webservices.daehosting.com/services/isbnservice.wso?WSDL' l_client = suds.client.Client(l_wsdl, gzip = False)
l_result = l_client.service.IsValidISBN10('0123456789') print l_result
# default behaviour is gzip = True as below l_client2 = suds.client.Client(l_wsdl, gzip = True)
l_result = l_client2.service.IsValidISBN10('0123456789') print l_result
Anyhow, I think that the changes can be implemented also for Transport.open, so that if you have to download large WSDL files, the download will also happen compressed (if supported by the server)
I even think that the compression request and processing could be part of the Transport class, to let people implement transport with libraries that may not have native gzip support.
Since I have some free time in the next minutes, I will possibly add a new patch to the ticket, so you may decide which one you like more
Best regards
Daniel
On Wed, May 5, 2010 at 20:52, Jeff Ortel jortel@redhat.com wrote:
Hey Daniel,
Thanks for the great suggestion, patch and for creating a ticket for this. I'll review for inclusion in 0.4. Can you recommend a public test site/service for this?
Regards,
Jeff
On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
Hello,
I have added ticket 311 requesting this enhancement with attached patches for transport/http.py and transport/options.py
Requesting compression can be turned off and on with the "gzip" option (a bool)
The gzip decompression code works always. The reasoning behind this is:
- Some network providers may compress the data in HTTP responses in order to decrease network traffic, expecting all clients to be ready to decompress data
Best regards
Daniel
On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
A "break" at the end of the "if" statement will not hurt and avoid checking unnecessary headers.
Best regards
On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod@gmail.com mailto:danjrod@gmail.com> wrote:
Hi Jeff et al, If I am not wrong (I may have missed a configuration option), suds is not using or requesting gzip compression. I have added a few lines and tested at least requesting and accepting gzip compression. My changes have been trivial and this could be possibly added as an option to suds. The changes I have made (against 3.9): _File_: suds/transport/http.py: _ _ _Extra imports_: import gzip import StringIO _Class_: HttpTransport _Method_: send _Added:_ After: * headers = request.headers added the following line * headers.update([('Accept-Encoding', 'gzip')]) _Changed:_ * result = Reply(200, fp.headers.dict, fp.read()) to * replydata = fp.read() * for header in fp.headers.dict.keys(): * if header.lower() == 'content-encoding' and fp.headers.dict[header] == 'gzip': * replydatacompressedstream = StringIO.StringIO(replydata) * gzipper = gzip.GzipFile(fileobj=replydatacompressedstream) * replydata = gzipper.read() * result = Reply(200, fp.headers.dict, replydata) Best regards Daniel
suds mailing list suds@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/suds