[Twisted-Python] bad news about twistedmatrix.com's hosting

[Evilham]

On dt., ag. 03 2021, Adi Roiban wrote:

> On Sun, 1 Aug 2021 at 10:29, Evilham <contact at evilham.com> 
> wrote:
>
>> \o
>>
>> On dg., ag. 01 2021, Glyph wrote:
>>
>> > Hello Twistors,
>> >
>> > Our venerable hosting provider, tummy.com 
>> > <http://tummy.com/>,
>> > will be terminating their business operations on September 
>> > 30,
>> > 2021.
>> >
>> >>>> datetime.date(2021, 9, 30) - datetime.date.today()
>> > datetime.timedelta(days=60)
>> >
>> > In addition to being a bit personally sad - Tummy has been an
>> > avid supporter of the Twisted community since the very 
>> > beginning
>> > - this means we have just about 60 days to fully migrate our
>> > website, mailing list, and DNS's hosting from Dornkirk, the
>> > machine where it's been running continuously since 2013, to 
>> > ...
>> > something else.
>>
>>
>> it is sad indeed when friendly businesses terminate operations,
>> but Twisted will go on :-).
>>
>>
>> > There are 2 problems here:
>> >
>> > We need a sponsor to help us find a new hosting or cloud
>> > provider where things can be hosted.  We do pay tummy 
>> > currently,
>> > and we can probably ask the SFC to pay a different hosting
>> > provider a similar price for equivalent services, but ideally
>> > we'd find someone willing to donate something so as not to 
>> > put a
>> > drain on those resources.
>>
>>
>> Let's define this so I and others can ask around with various
>> infra providers:
>>
>> - What is the service currently? (e.g. VPS/Physical? Which
>>   resources?)
>>
>
> We only have a single server. It is  called "dornkirk"
> I don't know if it's a VM or bare-metal... but it shouldn't 
> matter.
>
> - Are the resources allocated adequate? Can/should they be
>>   increased/reduced?
>>
>
> I don't understand this question. Sorry.
> Can you please rephrase ? Thanks.


This was very poorly worded, my bad.
I meant how big of a machine (storage excluding ML archives, RAM, 
CPU, traffic) is currently used and whether or not it is 
over/under-provisioned.
Without knowing this concretely, it is hard to try to get an 
entity to commit to donating resources.

It is good to know that VM vs physical and location are not 
important factors.


>> - Is currently something on the "wishlist" for this service? 
>> (e.g.
>>   IPv6 support!)
>>
>
> Many things are on the wishlist.
> The priority is to have the services up and running, even after 
> dornkirk is
> dead.
>
>
>> - Even if the goal is and should be for that expense to 
>> disappear
>>   or go down significantly: roughly what is the compensation 
>>   being
>>   offered for this service? (it might be easier to convince
>>   someone to offer it at cost/partial compensation than it is 
>>   to
>>   offer it for free)
>>
>
> I don't know what is the current cost to cover the expenses for 
> dornkirk.
>
>
>
>> - Is location important? E.g. if I asked around and it were
>>   fruitful, it'd be EU or Switzerland-based and not US-based as 
>>   it
>>   is right now.
>>
>
> No. Location is not important.
>
>
>>
>> > I will need significant help with the operational aspects of 
>> > the
>> > migration.  I typically do a bunch of behind-the-scenes 
>> > system
>> > administration for twistedmatrix.com 
>> > <http://twistedmatrix.com/>
>> > to keep the whole thing clunking along, but after my brain 
>> > has
>> > been fried by a year and a half's worth of pandemic stress, I
>> > simply don't have the time, energy, or mental capacity to do
>> > this.  I know I've asked for help before and rarely received
>> > any, but if I have to do this entirely by myself, the most
>> > likely outcome is that I'll migrate DNS to some free cloud 
>> > thing
>> > so at least my email address doesn't break (I migrated SMTP, 
>> > and
>> > thereby everything associated with personal email addresses 
>> > to
>> > Mailgun some years ago, so Tummy is not in that loop for 
>> > that),
>> > and the website (and this mailing list) will simply stop 
>> > working
>> > sometime between now and October.  So if you'd like to keep
>> > www.twistedmatrix.com <http://www.twistedmatrix.com/>,
>> > speed.twistedmatrix.com <http://speed.twistedmatrix.com/>, 
>> > our
>> > IRC bot (which is offline anyway right now due to Freenode
>> > shutting down), https://twistedmatrix.com/highscores/
>> > <https://twistedmatrix.com/highscores/>, this mailing list, 
>> > or
>> > our dogfooding instance for Twisted Names DNS, please 
>> > volunteer
>> > so we can start to coordinate.
>>
>>
>> Thank you for both doing this so long and so far and for being
>> straightforward about not being able to do it as much now.
>>
>> I'd like to help and would be particularly (aka more 
>> intensively)
>> able to do so during August, even if that's just helping 
>> document
>> what needs to be done and how.
>> On a day-to-day basis I do some other sysadmin work and if
>> necessary would be able to accommodate taking care of
>> Twisted-related things into that.
>>
>
> To coordinate migration and also day-to-day Twisted project 
> things we use
> the IRC Libera channel #twisted-dev
>
> If you don't like IRC ... we are also experiencing with 
> Gitter/Matrix
> https://gitter.im/twisted/twisted
>
> I start this to-do list 
> https://github.com/twisted-infra/braid/issues/277


IM is particularly taxing these days.

The TODO list looks quite comprehensive, thank you.


>
>
>> This is the documentation that I am aware of:
>> https://twistedmatrix.com/trac/wiki/Infrastructure
>
>
> There is not much documentation... we need to rely on memories 
> :)
> So don't bother to search for documentation, and just ask :)
>
>
>>
>> Particularly braid: https://github.com/twisted-infra/braid
>>
>>
> Yes. braid is our deployment as code thing... is the best we 
> have in terms
> of documenting the infrastructure.
>
> "braid" is (was?) the Twisted customization on top of fabric for 
> doing
> primitive automated deployment.
>
>
>
>> Is there anything else to look at? Something known to not be
>> supported in braid / only partially supported?
>>
>
> The plan is to stop using braid and do all the deployment via 
> images and
> containers.
>
>
>> Would you rather schedule some online voice session where I 
>> poke
>> your brain at what needs to be done (IIRC something similar was
>> done about 5 years ago) and take notes that can be used (also 
>> by
>> someone else) while you let your mind flow happily?
>>
>
> We are split between Europe time zone (UTC -> UTC+2) and US ( 
> something
> like UTC-6.. but also UTC-7)
> For now, we have done only ad-hoc meetings.
>
>
>>
>> > If you've been wanting to just use Github Issues and don't 
>> > care
>> > about preserving any of the data in Trac, congratulations,
>> > you're about to win that argument by default ;-).
>> >
>> > In the worst case scenario, I will download a backup image
>> > before things get turned off in case someone wants to deal 
>> > with
>> > this
>>
>>
>> I personally hope this doesn't happen; particularly the mailing
>> list is a great low-bandwidth way of staying mostly up-to-date
>> with the project.
>>
>
> I think that the plan is to migrate the mailing list to the 
> Python Org
> mailing list hosting.
>
> We will need some help with setting  up archive redirections.
>
> ---------------
>
> I think the current problem is finding or deciding for the next 
> hosting /
> VM / container provider.
>
> We have 150 USD per month from MS to spend on Azure, but this is 
> renewed
> any year without any guarantee that next year we will receive 
> the free
> credit.
> The subscription was requested by Python Software Foundation and 
> was
> granted by MS, but it was associated to the Azure AD of my 
> organization.
>
> I have no idea how to create an Azure AD for Twisted 
> organization, so that
> I can transfer the subscription.
>
> So...if anyone has any experience with Azure AD, it would help 
> to get this
> thing sorted.
>
> For now, using Azure is plan B... so if we don't manage to 
> migrate the
> services somewhere else, we (I) can just create an Ubuntu VM in 
> Azure and
> migrate the existing dornkirk services there.
>
> ------------
>
> The DNS was already migrated by Glyph. Thanks Glyph :)
>
> The mailing lists will be migrated soon to Python Org... we only 
> have to
> see how to do it in a seamless way.



Mailing list was certainly the most concerning point, it's great 
that it's already being solved \o/ handling mailing lists is 
really not fun.

Using the registrar's DNS makes sense short-term, mid-term as 
Glyph mentioned twisted.names should be used by the project so 
there is starker incentive to keep it working and somewhat up to 
date.
Having this functionality allows for interesting use-cases: 
https://github.com/evilham/weneeda.name
IIRC there was a way to setup Gandi to mirror a zone periodically, 
so that'd be good redundancy.
Anyway, that's for somewhen else.


>
> --------
>
> So Evilham, if you want to help, try to see if you can hang out 
> over IRC or
> Gitter and we can see how you can help.
> Thanks!


After catching up on some other things, I'll take a more serious 
look at what I can help with from the TODO.