[Twisted-web] Noob question about POST
Donald Steiny
steiny at steiny.com
Sat Dec 31 14:22:32 EST 2011
Hi,
I have just been learning twisted and been Googling for an example of
something that is simple to do with pycurl or even straight python but I
can't figure out for the life of me how to do with twisted. I want to
be able to do a POST and debug a login to a system. Ultimately, what I
am trying to do (and twisted seems to be the best choice for this) is to
make the login process automatic for the user so after he or she is
logged in he or she will interact with the remote system as if he or
she had logged in his or herself. The proxy class seems perfect for the
second part and getPage should allow me to do the first, but I don't
seem to understand something. What I have done is to set up a simple
login/password page on my own server with login fred and password 1234.
I am using Python 2.6 on Linux 2.6. I will describe what I expect to
happen, then what I did, then what happened that I did not expect and if
anyone could set me straight I would be eternally grateful AND because
of the good Karma you will find parking spaces easily all during 2012.
If I telnet to localhost port 80 and copy and paste the following:
=== BEGINNING OF WORKING INPUT TO SERVER ===
POST /locked/checklogin.php HTTP/1.1
Host: www.mysite.com
User-Agent: Mozilla/4.0
Content-Length: 31
Content-Type: application/x-www-form-urlencoded
myusername=john&mypassword=1234
===== END OF WORKING INPUT TO SERVER ====
I get just what I would expect. On successful login it goes to the page
"login_success.php"and I print out the _POST array to make sure the
variables are getting passed:
==== BEGINNING OF CORRECT OUTPUT FROM SERVER ===
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
POST /locked/checklogin.php HTTP/1.1
Host: www.mysite.com
User-Agent: Mozilla/4.0
Content-Length: 31
Content-Type: application/x-www-form-urlencoded
myusername=john&mypassword=1234
HTTP/1.1 302 Found
Date: Sat, 31 Dec 2011 18:42:28 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.3-7+squeeze3
Set-Cookie: PHPSESSID=tueqdd71kha040d18d1qabg424; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Location: login_success.php
Vary: Accept-Encoding
Content-Length: 57
Content-Type: text/html
IN CHECKLOGIN<BR>
myusername: john, mypassword: 1234<BR>
Connection closed by foreign host.
=== END OF CORRECT OUTPUT FROM SERVER ===
I took a getPage example that is in the documentation:
=== START CODE ===
from pprint import pformat
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
class BeginningPrinter(Protocol):
def __init__(self, finished):
self.finished = finished
self.remaining = 1024 * 10
def dataReceived(self, bytes):
if self.remaining:
display = bytes[:self.remaining]
print 'Some data received:'
print display
self.remaining -= len(display)
def connectionLost(self, reason):
print 'Finished receiving body:', reason.getErrorMessage()
self.finished.callback(None)
agent = Agent(reactor)
d = agent.request(
'GET',
'http://localhost/locked/checklogin.php',
Headers({'User-Agent': ['Twisted Web Client Example']}),
None)
def cbRequest(response):
print 'Response version:', response.version
print 'Response code:', response.code
print 'Response phrase:', response.phrase
print 'Response headers:'
print pformat(list(response.headers.getAllRawHeaders()))
finished = Deferred()
response.deliverBody(BeginningPrinter(finished))
return finished
d.addCallback(cbRequest)
def cbShutdown(ignored):
reactor.stop()
d.addBoth(cbShutdown)
reactor.run()
==== END CODE ===
The output was just what I expected:
=== OUTPUT FROM RUN OF CODE ==
Response version: ('HTTP', 1, 1)
Response code: 200
Response phrase: OK
Response headers:
[('Date', ['Sat, 31 Dec 2011 18:55:17 GMT']),
('Content-Type', ['text/html']),
('X-Powered-By', ['PHP/5.3.3-7+squeeze3']),
('Vary', ['Accept-Encoding']),
('Server', ['Apache/2.2.16 (Debian)'])]
Some data received:
IN CHECKLOGIN<BR>
myusername: , mypassword: <BR>
Wrong Username or Password: ,
Finished receiving body: Response body fully received
shutdown called
===== END OF OUTPUT FROM RUN OF CODE --
I modified the example by changing the GET to a POST and including
StringProducer and changing the None to a body and the header to
multipart/form-data. Here:
==== START OF CODE MODIFIED WITH POST ===
from pprint import pformat
from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
from twisted.web.iweb import IBodyProducer
from zope.interface import implements, Interface
from stringprod import StringProducer
class BeginningPrinter(Protocol):
def __init__(self, finished):
self.finished = finished
self.remaining = 1024 * 10
def dataReceived(self, bytes):
if self.remaining:
display = bytes[:self.remaining]
print 'Some data received:'
print display
self.remaining -= len(display)
def connectionLost(self, reason):
print 'Finished receiving body:', reason.getErrorMessage()
self.finished.callback(None)
creds = "myusername=john&mypassword=1234\r\n"
body = StringProducer(creds)
agent = Agent(reactor)
d = agent.request(
'POST',
'http://localhost/locked/checklogin.php',
Headers({'User-Agent': ['Mozilla 5.0'],
'Content-Type': ['multipart/form-data; charset=utf-8']}),
body)
def cbRequest(response):
print 'Response version:', response.version
print 'Response code:', response.code
print 'Response phrase:', response.phrase
print 'Response headers:'
print pformat(list(response.headers.getAllRawHeaders()))
finished = Deferred()
response.deliverBody(BeginningPrinter(finished))
return finished
d.addCallback(cbRequest)
def cbShutdown(ignored):
print "shutdown called"
reactor.stop()
d.addBoth(cbShutdown)
reactor.run()
=== END CODE MODIFIED WITH POST ===
When I run this, I do not get the output I expect, for some reason, it
does not seem to be sending the body.
==== BEGINNING OF RUN OF MODIFIED CODE ====
Response version: ('HTTP', 1, 1)
Response code: 200
Response phrase: OK
Response headers:
[('Date', ['Sat, 31 Dec 2011 19:06:39 GMT']),
('Content-Type', ['text/html']),
('X-Powered-By', ['PHP/5.3.3-7+squeeze3']),
('Vary', ['Accept-Encoding']),
('Server', ['Apache/2.2.16 (Debian)'])]
Some data received:
IN CHECKLOGIN<BR>
myusername: , mypassword: <BR>
Wrong Username or Password: ,
Finished receiving body: Response body fully received
shutdown called
==== END OF RUN OF MODIFIED CODE ===
What am I missing? The body does not seem to be being send. I would
like to have a way to easily be able to debug logins to remote systems.
This is NOT for spamming. I am trying to develop a system for higher
security where our users would log locally (VPN) with their own
favorite passwords and that would map to awful username and password
that no one would ever use or remember and would be unguessable to log
them into external cloud based systems. In some cases there are
libraries and APIs to help me, but I want a general solution. Mechanic
seems to do part of this and pycurl, but the proxy and networking part
seem much better in twisted. HELP!!!
Thanks,
Donald
More information about the Twisted-web
mailing list