[Twisted-Python] Can transport.write() to hostname instead of IP address?

[Tom Most]

Hi Sean,

To summarize, you are writing a client application using Twisted which
needs to talk with a service behind an Nginx reverse proxy. The reverse
proxy requires use of TLS (a.k.a. SSL) and SNI to identify the
appropriate backend service.
If you were using HTTP, Twisted's twisted.web.client.Agent[1] API
already does the right thing here -- it takes the hostname from the URL
it is given and populates SNI.
For your custom protocol on top of TLS, you should use a TLS endpoint to
connect to the server from your client. This is a little difficult to
discover because there isn't a TLS endpoint per se -- instead, there is
a function which wraps another endpoint. See the TLS section in the
endpoint documentation[2], which includes this example:
wrapped = HostnameEndpoint('example.com', 443) contextFactory =
optionsForClientTLS(hostname=u'example.com') endpoint =
wrapClientTLS(contextFactory, wrapped) conn =
endpoint.connect(Factory.forProtocol(Protocol))
I'll break this down:

1. HostnameEndpoint will resolve the hostname to an IP address and
   creates a TCP connection to port 443.2. The optionsForClientTLS[3] function generates an object which
   represents the TLS connection options. Importantly, it enables SNI
   based on the hostname passed to it.3. wrapClientTLS[4] returns an endpoint which layers TLS on top of the
   plain TCP connection generated by HostnameEndpoint. It also takes the
   TLS options as an argument.4. conn is a Deferred which will fire with a protocol instance generated
   by the factory passed to connect().
This is basically what Agent does internally, as I understand it.

Hope this helps,
Tom

On Thu, Aug 16, 2018, at 6:44 PM, Sean DiZazzo wrote:
> I guess thats still kind of confusing without making something more
> clear...> 
> In my example, both myprotocol.example.com and test.example.com DNS
> records would point to the same IP address.  One nginx instance then
> listens on that IP and serves up several ssl apps. They go through a
> "mapper" that uses the SNI and the ssl_preread directive to read the
> destination hostname of the packet to determine which app to route the
> traffic to.> 
> I just want transport.write() to not resolve the ip address of the
> host I pass in.  Everything will work if it connects and sends packets
> to myprotocol.example.com:443[5] instead of 23.23.23.23:443.> 
> Nginx reference:
> http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
> 
> On Thu, Aug 16, 2018 at 6:14 PM, Sean DiZazzo
> <sean.dizazzo at gmail.com> wrote:>> Thanks for responding, Adi!
>> 
>> I don't want each packet to go it's own way from Twisted.  They all
>> go to the same place from each instance of the server/protocol.  They
>> go to my custom protocol listening on another local port.>> 
>> It's just that I'm serving up several different ssl apps on the same
>> nginx server, and nginx uses the hostname to route the packets.  So
>> in this case, traffic coming in on http.example.com:443 might be
>> routed to an https app listening on a socket, and traffic coming in
>> to myprotocol.example.com:443 should be routed to my own protocol
>> listening on port 9999.  So if nginx doesn't get the hostname, it
>> doesn't know to route the packet to my custom protocol instead of the
>> web server.  Does that make sense?>> 
>> It seems that the transport is resolving the hostname to an ip
>> address and then sending the traffic to the generic ip which is not
>> enough info for nginx to route the packet correctly.>> 
>> On Thu, Aug 16, 2018 at 5:49 PM, Adi Roiban <adi at roiban.ro> wrote:
>>> On Fri, 17 Aug 2018 at 01:25, Sean DiZazzo <sean.dizazzo at gmail.com>
>>> wrote:
>>>  >
>>>  > Hi all!
>>>  >
>>>  > After I start a reactor connecting to a specific hostname and
>>>  > port, I do my thing and then call transport.write() to send the
>>>  > data to the peer.
>>>  >
>>>  > From what I can tell, though, the hostname is resolved, and the
>>>  > data is written back to the ip address itself, instead of the
>>>  > hostname I started the reactor with.
>>>  >
>>>  > This is a problem in my case because we are using nginx's
>>>  > ssl_preread server_name directive to route several different
>>>  > streams all coming in on the same ip address.
>>>  >
>>>  > So the write() method needs to explicitly use the hostname to
>>>  > route the packet properly.
>>>  >
>>>  > So... Is there any way to have transport.write() use the hostname
>>>  > given instead of it's resolved IP address? Or am I missing
>>>  > something?
>>>  >
>>>
>>> I assume you are using TCP here.>>> 
>>>  I guess that you are missing something.
>>> 
>>>  If you want each write to go over its own way / route and have the>>>  hostname re-resolved you should open + write + close a
>>>  connection for>>>  each write.
>>> 
>>>  But I think that there is something else there and this is now what
>>>  you want :)>>>  Do you use HTTP or have a custom protocol?
>>> 
>>>  Cheers,
>>> 
>>>  Adi Roiban
>>> 
>>>  _______________________________________________
>>>  Twisted-Python mailing list
>>> Twisted-Python at twistedmatrix.com
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> _________________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


Links:

  1. https://twistedmatrix.com/documents/current/api/twisted.web.client.Agent.html
  2. https://twistedmatrix.com/documents/current/core/howto/endpoints.html#endpoint-types-included-with-twisted
  3. https://twistedmatrix.com/documents/18.7.0/api/twisted.internet.ssl.optionsForClientTLS.html
  4. https://twistedmatrix.com/documents/current/api/twisted.internet.endpoints.html#wrapClientTLS
  5. http://myprotocol.example.com:443/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20180816/7159fed4/attachment-0002.html>