Re: [pgadmin-hackers] Feature test issues

Cool, we already have a task about proper teardown and a couple other things in our backlog. we'll probably get to it in the next day or so. I'll take a look at the other stuff.

Also, regarding speed, even without the app startup time, end to end tests are always going to be relatively slow. We definitely want to make sure that the time it takes to run the tests does not grow to where it is a deterrent to running them.
There are a variety of things we can do to help address that as our suite grows. For instance,  we could consider parallelizing the tests, making setup and teardown more efficient,  combining related tests, or even breaking the tests into suites and running only some of them locally by default.
Since we only have a couple feature tests so far the speed hasn't really felt like an issue for me yet, but I understand it may be different if you are trying to run in a variety of configurations.

Out of curiosity, what is the goal in supporting multiple python versions?

Are we working on moving to 3.x and just haven't gotten fully there yet?

Tira

On Sun, Feb 26, 2017, 3:39 AM Dave Page <dpage@pgadmin.org> wrote:

Hi Tira, George,

I've just been updating our internal automated test system to run the
feature tests, and ran into a couple of additional issues that need to
be addressed. Can you look into the following please?

- When starting pgAdmin, it's using the default configuration database
(CONFIG['SQLITE_PATH']), however for testing we should be using
CONFIG['TEST_SQLITE_PATH']). This means that it's polluting the user's
default configuration (and if they don't have one, causing an
additional initialisation step).

- With Python 2.6, the following failure is seen when the first
feature test is run:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/web/regression/runtests.py",
line 286, in <module>
    verbosity=2).run(suite)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/runner.py",
line 172, in run
    test(result)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/suite.py",
line 87, in __call__
    return self.run(*args, **kwds)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/suite.py",
line 126, in run
    test(result)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
line 673, in __call__
    return self.run(*args, **kwds)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
line 633, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "/var/lib/jenkins/workspace/pgadmin4-master-python26/pgadmin-venv/lib/python2.6/site-packages/unittest2/case.py",
line 563, in _feedErrorsToResult
    if issubclass(exc_info[0], self.failureException):
TypeError: issubclass() arg 2 must be a class or tuple of classes

For completeness, other issues outstanding that we've previously discussed:

- pgAdmin processes may remain running after test failures.

- The test suite may hang following a feature test failure, at the end
of the run.

- The screenshot functionality should be fixed (ideally) or removed.

- The tests really need to run with a single instantiation of pgAdmin.
It's clearly going to be far too slow to start/stop pgAdmin for every
test once we start adding more (and moving forward, I really want
feature tests to become the default to ensure we're end-to-end testing
everything). For reference, each test run (currently one version of
Python, against 5 different database servers) is now taking ~5 minutes
vs. 1m47s without the feature tests.

On the plus side, test runs are now green across the board with
feature tests enabled, except for Python 2.6 :-)

Thanks!

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company