[Twisted-Python] Can transport.write() to hostname instead of IP address?

Fri Aug 17 10:24:48 MDT 2018

Starting to look over the docs and I realize I might have just been able to
incorporate the `optionsForClientTLS` into what I was already using.

I was using `reactor.connectSSL()` with a default
``ssl.ClientContextFactory()` so I think I can make my own
sslClientContextFactory with `optionsForClientTLS(hostname=host)` and pass
that in to connectSSL instead.

It would look something like this:

reactor.connectSSL(hostname, port, MyProtocolClientFactory(),
optionsForClientTLS(hostname=hostname))

I'll give it a try and play around with it.  If it doesn't work, I'll head
back over to the endpoint examples.

Thanks again!

On Thu, Aug 16, 2018 at 9:06 PM, Sean DiZazzo <sean.dizazzo at gmail.com>
wrote:

> To summarize, you are writing a client application using Twisted which
>> needs to talk with a service behind an Nginx reverse proxy. The reverse
>> proxy requires use of TLS (a.k.a. SSL) and SNI to identify the appropriate
>> backend service.
>
>
> Why was that so difficult for me?  lol.
>
> This is all great, Tom.  Thank you.  I guess I didn't find this because up
> until now I have only played around with endpoints.  The twisted servers I
> have running are all using the older interfaces for setting up and handling
> connections.  I balked at learning endpoints after struggling to figure out
> Twisted the first time.  I guess I'm a holdout.  I had barely even looked
> at python 3 until the last few months.  Now I love it.  ;P  Its time for
> endpoints to get their turn in the sun.
>
> It will take some rethinking and rewriting, but at least now I have a way
> forward.  Thank you!
>
> On Thu, Aug 16, 2018 at 8:18 PM, Tom Most <twm at freecog.net> wrote:
>
>> Hi Sean,
>>
>> To summarize, you are writing a client application using Twisted which
>> needs to talk with a service behind an Nginx reverse proxy. The reverse
>> proxy requires use of TLS (a.k.a. SSL) and SNI to identify the appropriate
>> backend service.
>>
>> If you were using HTTP, Twisted's twisted.web.client.Agent
>> <https://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html>
>> API already does the right thing here -- it takes the hostname from the URL
>> it is given and populates SNI.
>>
>> For your custom protocol on top of TLS, you should use a TLS endpoint to
>> connect to the server from your client. This is a little difficult to
>> discover because there isn't a TLS endpoint per se -- instead, there is a
>> function which wraps another endpoint. See the TLS section in the
>> endpoint documentation
>> <https://twistedmatrix.com/documents/current/core/howto/endpoints.html#endpoint-types-included-with-twisted>,
>> which includes this example:
>>
>> wrapped = HostnameEndpoint('example.com', 443)contextFactory = optionsForClientTLS(hostname=u'example.com')endpoint = wrapClientTLS(contextFactory, wrapped)conn = endpoint.connect(Factory.forProtocol(Protocol))
>>
>>
>> I'll break this down:
>>
>> 1. HostnameEndpoint will resolve the hostname to an IP address and
>> creates a TCP connection to port 443.
>> 2. The optionsForClientTLS
>> <https://twistedmatrix.com/documents/18.7.0/api/twisted.internet.ssl.optionsForClientTLS.html>
>> function generates an object which represents the TLS connection options.
>> Importantly, it enables SNI based on the hostname passed to it.
>> 3. wrapClientTLS
>> <https://twistedmatrix.com/documents/current/api/twisted.internet.endpoints.html#wrapClientTLS>
>> returns an endpoint which layers TLS on top of the plain TCP connection
>> generated by HostnameEndpoint. It also takes the TLS options as an argument.
>> 4. conn is a Deferred which will fire with a protocol instance generated
>> by the factory passed to connect().
>>
>> This is basically what Agent does internally, as I understand it.
>>
>> Hope this helps,
>> Tom
>>
>> On Thu, Aug 16, 2018, at 6:44 PM, Sean DiZazzo wrote:
>>
>> I guess thats still kind of confusing without making something more
>> clear...
>>
>> In my example, both myprotocol.example.com and test.example.com DNS
>> records would point to the same IP address.  One nginx instance then
>> listens on that IP and serves up several ssl apps. They go through a
>> "mapper" that uses the SNI and the ssl_preread directive to read the
>> destination hostname of the packet to determine which app to route the
>> traffic to.
>>
>> I just want transport.write() to not resolve the ip address of the host I
>> pass in.  Everything will work if it connects and sends packets to
>> myprotocol.example.com:443 instead of 23.23.23.23:443.
>>
>> Nginx reference:
>> http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
>>
>> On Thu, Aug 16, 2018 at 6:14 PM, Sean DiZazzo <sean.dizazzo at gmail.com>
>> wrote:
>>
>> Thanks for responding, Adi!
>>
>> I don't want each packet to go it's own way from Twisted.  They all go to
>> the same place from each instance of the server/protocol.  They go to my
>> custom protocol listening on another local port.
>>
>> It's just that I'm serving up several different ssl apps on the same
>> nginx server, and nginx uses the hostname to route the packets.  So in this
>> case, traffic coming in on http.example.com:443 might be routed to an
>> https app listening on a socket, and traffic coming in to
>> myprotocol.example.com:443 should be routed to my own protocol listening
>> on port 9999.  So if nginx doesn't get the hostname, it doesn't know to
>> route the packet to my custom protocol instead of the web server.  Does
>> that make sense?
>>
>> It seems that the transport is resolving the hostname to an ip address
>> and then sending the traffic to the generic ip which is not enough info for
>> nginx to route the packet correctly.
>>
>> On Thu, Aug 16, 2018 at 5:49 PM, Adi Roiban <adi at roiban.ro> wrote:
>>
>> On Fri, 17 Aug 2018 at 01:25, Sean DiZazzo <sean.dizazzo at gmail.com>
>> wrote:
>> >
>> > Hi all!
>> >
>> > After I start a reactor connecting to a specific hostname and port, I
>> do my thing and then call transport.write() to send the data to the peer.
>> >
>> > From what I can tell, though, the hostname is resolved, and the data is
>> written back to the ip address itself, instead of the hostname I started
>> the reactor with.
>> >
>> > This is a problem in my case because we are using nginx's ssl_preread
>> server_name directive to route several different streams all coming in on
>> the same ip address.
>> >
>> > So the write() method needs to explicitly use the hostname to route the
>> packet properly.
>> >
>> > So... Is there any way to have transport.write() use the hostname given
>> instead of it's resolved IP address? Or am I missing something?
>> >
>>
>> I assume you are using TCP here.
>>
>> I guess that you are missing something.
>>
>> If you want each write to go over its own way / route and have the
>> hostname re-resolved you should open + write + close a connection for
>> each write.
>>
>> But I think that there is something else there and this is now what you
>> want :)
>> Do you use HTTP or have a custom protocol?
>>
>> Cheers,
>>
>> Adi Roiban
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>> *_______________________________________________*
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20180817/539a322a/attachment-0002.html>