Re: [pgadmin-hackers] Feature test issues

On Mon, Feb 27, 2017 at 7:51 PM, Atira Odhner <aodhner@pivotal.io> wrote:
> The team here at Pivotal is doing TDD already. Running so many
> configurations sounds like it belongs in a CI not in a TDD cycle.

It is in a CI - that doesn't mean I don't want us to aim towards the
point where we're doing TDD as well.

> I want to make sure we are making the most effective technical investment
> that we can. My thought process is generally:
> Are we feeling pain from this right now? (is it slowing down development,
> leading to confusion, bugs, etc?)

Well the randomly failing tests are, yes. The speed isn't - but it
likely will if not addressed.

> If not, are we painting ourselves into a corner? (Will it be an order of
> magnitude harder to make this investment later?)

Possibly, if tests need to be rewritten for single-server-instance or
parallel execution.

> How does this pain compare to other sources of technical pain?

For me, aside from reviewing of patches which will always take time
and effort, getting the tests to run fully and reliably is my biggest
source of pain.

> Once we've decided to invest in easing a pain, I find it valuable to step
> back and try to see if there are other options that might also address it.
>
> In the case of running multiple configurations,  I suspect you might get a 2
> or 3x speedup of the entire suite by running tests in parallel.
> The app startup time is pretty fast. It takes about two to three seconds. At
> this point, that would shave about 6 seconds per suite run or 30 seconds in
> your 5-server run vs 3-4 minutes that I expect would be saved through
> parallelization.

On my top-of-the-range-last-year Mac, startup takes about 14 seconds
per test - that's from the start of the test, until the point at which
the "Loading pgAdmin..." spinner vanishes.

Running the tests in parallel on each DB server would certainly help I
agree, but given the current architecture of the feature tests when we
get to 100 tests in the suite we'll be wasting 23 minutes per run just
on startup time - and given that changing the tests to run in a single
browser session with a single pgAdmin server would almost certainly
require changes to the tests themselves, I think it's prudent to do
that now rather than when we actually have 100 of them.

> Regarding the python versions, have we considered packaging the app bundle
> with the latest version of python that we support? Then we could run tests
> with that version and the latest version of python that we are working
> towards supporting. That would save us the headache of trying to straddle
> the 2.x/3.x language gap.

The appbundle is just one package - and is one that only needs Python
2.7 as that's all macOS has supported for years. On Windows, we do
ship Python as part of the package as there is no Python interpreter
there by default.

However, we obviously cannot ship Python as part of the Python wheel,
nor is it practical to ship a private build of Python in RPM or DEB
packages (and would almost certainly be a breach of the distro
packaging rules if we did, thus preventing pgAdmin 4 from ever
shipping in EPEL or Canonical's package repos).

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company