[Twisted-Python] GitHub Actions parallelism limit increase

[Adi Roiban]

On Thu, 1 Apr 2021 at 13:46, Kyle Altendorf <sda at fstab.net> wrote:

> On 2021-03-31 05:45, Adi Roiban wrote:
>
> > On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, <sda at fstab.net> wrote:
> >
> >> On 2021-03-30 15:24, Glyph wrote:
> >>>> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf <sda at fstab.net> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> Has anyone contacted GitHub to see if they would be willing to
> >>>> increase the parallelism limit in Actions?  My understanding is that
> >>>> we maintain two CI systems (GitHub Actions and Azure Pipelines) for
> >>>> the sake of more parallelism.  While perhaps worthwhile, this
> >>>> doesn't
> >>>> seem fun.  Maybe GitHub would be willing to help out.
> >>>
> >>> Not as far as I know. Do you want to give it a shot?
> >>
> >> That was the plan.  Submitted.  I'll let you all know.
> >
> > Thanks.
> >
> > On a related note I am working to reduce the general duration of
> > Twisted CI jobs.
> >
> > This should result in keeping the jobs slot busy for less time and
> > allowing more total jobs per day.
> >
> > The first step was to enable parallel trial tests. Use both CPUs
> > available to the CI VMs.
> > This was already done for Linux and MacOS.
> >
> > Distributed trial is not yet supported on Windows, but I am working on
> > it.
> > I already have a working distributed trial for Windows on my system.
> > It needs various fixed and I have submitted smaller PR for review.
> >
> > Just getting distrial to work on Windows is not all...as we need to
> > update and clean the  Twsited test to reduce the side effects.
> > Linux and macOS are more relaxed in terms of file access and for
> > example you can delete a file, even if its opened by another process or
> > by the same process.
> > Windows is not happy about that.
> >
> > And then there are the 2 pypy jobs taking 30 minutes each - working to
> > fix it here https://github.com/twisted/twisted/pull/1543/
> >
> > And then there are also general CI jobs improvements.
> >
> > For example we can save about 40 seconds  if we decide to stop sending
> > coverage to coveralls and only use codecov.io
> > Another example is save another 1 minute by stop using `tox --notests`
> > and replace it with something like
> > https://github.com/twisted/python-info-action to show the dependencies.
> > With `tox --notest` we now create the wheels and install the
> > dependencies twice for each job.
>
> Would `tox --skip-pkg-install` cut it for the second invocation?
>
>
> https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145
>
>
I don't know. I still don't understand the tox install and test process...
and tox is not my friend as every week tox hits me with various errors that
I don't understand.
Most probably, tox is an advanced tool and I am a poor developer who
doesn't know much and sticks to the manual labour of calling venv + pip +
trial  :)

--------

For example, I don't understand why you need to create an sdist and from
sdist to create a wheel and then install that wheel inside tox for each tox
test environment.

Why not create the wheel once and install the same wheel in all
environments.


> > There is an extra 1 minute required for each job to generate the
> > coverage XML file. We can download all the coverage artifacts and
> > generate the XML only once.
>
> I did coverage collection from all jobs for pymodbus.  Looks like I
> still had the XML report generating in each job, but that could
> presumably be shifted.  Anyways, I'm sure several of us could do this
> off the top of our heads but in case it is of any interest here's what I
> did.  Or maybe you just point out something silly I did there and should
> learn from.  :]
>
> https://github.com/riptideio/pymodbus/pull/592/files
>


That is one big matrix :) ... I don't understand why you need test and
check and coverage, each with a separate matrix.

I am thinking of something like this:

* A matrix to run jobs with coverage on each environment (os + py
combination + extra stuff noipv6, nodeps, etc)
* Each job from the matrix will run an initial combine to combine
sub-process coverage
* Each job will upload once python coverage raw file.
* A single job will  download those coverage file, combine them once again
to generate an XML file to be pushed to codecov.io

The maximum jobs time will continue to be the same...as you still need to
wait for the slowest env, but the total run time can be reduced by more
than 20 minutes
as each job will no longer have to spend 2 minutes creating XML and
reporting JSON to coveralls.

-----------

Note that coveralls API is different. Instead of receiving an XML file,
with coverall there are separate API calls for reporting the coverage in
JSON format.
This is why coveralls reporting takes so long
Also, for coveralls, the python uploader is a 3rd party tool, and not an
official tool from the Coveralls team.
So maybe the python coveralls uploader tool can be improved to send the
reports faster.

Cheers

-- 
Adi Roiban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210401/1f1f2584/attachment-0001.htm>