Testing websites with Selenium is fun. Py.Test framework is really cool.
Joining those two and running tests inside of PyCharm IDE makes you wanna cry with joy. That is if you previously worked with iMacros :)
But sometimes testing takes hours, is run inside of crontab and you only need to check the results. The TimeoutException or NoSuchElementException tell you nothing if you haven't looked what the browser state was. The screenshot would really be helpful.
Acutally screenshot made after each test, regardless of the result would be helpful too - for example to detect layout errors or errors not covered by asserts.
I found a way to do this in pretty simple way. We will need a fixture that
- is used automaticaly with every test step
- calls the selenium driver to make a screenshot
I found no way yet to learn test outcome and make use of it when doing screenshot (like naming file with ``FAIL`` prefix or so on).
The key feature here is that our screenshot-making fixture uses the selenium driver fixture. And it is the same fixture that is provided to test functions.
Say we have selenium driver fixture that is module-wise parametrized with list of countries. That is a new browser session is delivered to module functions in a loop of countries.
Eeach test module gets a browser and is run in a loop of countries given in params=:
__author__ = 'sigviper'
import os
import pytest
import datetime
@pytest.fixture(scope="module", params=["pl", "de"])
def driver(request):
"""Selenium driver that creates a loop by countries"""
class FirefoxImprovedBySigviper(object):
def screenshot(self, request_for_test_function, session_test_timestamp):
country = request.param # module-wise request (each param creates new)
out_dir = "/tmp/shutter-{timestamp}/{country}/".format(timestamp=session_test_timestamp, country=country)
try:
os.makedirs(out_dir)
except OSError:
pass
fname = "{out_dir}/{country}_{module}_{function}_{timestamp}.log".format(
country=country,
out_dir=out_dir,
module=request_for_test_function.module.__name__,
function=request_for_test_function.function.__name__,
timestamp=get_timestamp()
)
f = open(fname, "ab")
f.write(repr(self) + "\n")
f.close()
return FirefoxImprovedBySigviper()
@pytest.fixture(scope="function", autouse=True)
def shutter(request, driver, session_test_timestamp):
"""Screenshot-making fixture"""
def fin():
driver.screenshot(request, session_test_timestamp)
request.addfinalizer(fin)
def get_timestamp():
"""Provide formatted current timestamp"""
return datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
@pytest.fixture(scope="session", autouse=True)
def session_test_timestamp(request):
"""Provide timestamp of test start time - fixed for a session"""
return get_timestamp()
Now the driver must implement screenshot logic. Please take a carefull look on usage of request and request_for_test_function variables - they are not the same.
The request is a attribute of driver fixture that is injected into FirefoxImprovedBySigviper class. The request_for_test_function parameter is given to screenshot function by shutter fixture that is ran before each test function call. The scope in test session of those variables is different. The module-wise request knows nothing about current function. On the other hand, request_for_test_function function-wise knows nothing about current param for the loop in driver. Consequently we need to use both.
Order of execution:
create session-wide timestamp and provide to all modules and functions
for each module create a Selenium driver:
for each parameter in driver run all test functions in a module:
save screenshot after each function call
Let's say we have two test modules: TestCatching and TestCopycatch each with three similarly named functions: *_ok, *_failed and *_raises
The test session in PyCharm looks like this:
The disk output with screenshot (here .log files, because I needed simple and quick sandbox for testing) looks like this:
viper@OptiPlex780:/tmp$ ls -lrctR shutter-2014-10-12_16\:16\:07/
shutter-2014-10-12_16:16:07/:
razem 8
drwxrwxr-x 2 viper viper 4096 paź 12 16:16 pl
drwxrwxr-x 2 viper viper 4096 paź 12 16:16 de
shutter-2014-10-12_16:16:07/pl:
razem 24
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_catching_test_1ok_2014-10-12_16:16:08.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_catching_test_2assert_failed_2014-10-12_16:16:09.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_catching_test_3raises_2014-10-12_16:16:10.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_copycatch_test_1copy_ok_2014-10-12_16:16:16.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_copycatch_test_2copy_assert_failed_2014-10-12_16:16:18.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 pl_test_copycatch_test_3copy_raises_2014-10-12_16:16:19.log
shutter-2014-10-12_16:16:07/de:
razem 24
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_catching_test_1ok_2014-10-12_16:16:11.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_catching_test_2assert_failed_2014-10-12_16:16:12.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_catching_test_3raises_2014-10-12_16:16:13.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_copycatch_test_1copy_ok_2014-10-12_16:16:22.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_copycatch_test_2copy_assert_failed_2014-10-12_16:16:24.log
-rw-rw-r-- 1 viper viper 57 paź 12 16:16 de_test_copycatch_test_3copy_raises_2014-10-12_16:16:25.log
All screenshots saved. Easy to find and sort. Names match country and test name. In case of error, timestamp is saved (sometimes helps developers narrow down periods in logs).
I like it. I'll just copy&paste it into my working code monday morning! :)
