End-to-end tests with Playwright

The problem

Take a look at the image below. You’ve made sure that your components work as expected by writing unit tests. So both windows can be opened on their own. But do they work together as expected? You can’t be sure without end-to-end tests, so a person that comes and tries to interact with them.

The Solution

While unit tests check individual components, end-to-end tests check the application as a whole, ensuring all components work together as expected. In case of dashboards, it means for example that if you fill some field and click a button, the expected output is displayed. You can think of it as a user interacting with the application. But user usually means you, the developer. And you don’t want to click through the app every time you make a change.

Tip

End-to-end dashboard tests are like a real user interacting with the application.

What’s Playwright?

Tests with playwright are like an automated user that clicks through the app and checks if everything works as expected. Pytest-playwright is a plugin that integrates Playwright with Pytest, making it easy to write and run end-to-end tests that are then run with a single pytest command (so you run unit and e2e tests with the same command).

In tapyr, we provide a pre-configured setup for Playwright with Shiny for Python that starts a shiny app in the background and allows playwright interacting with it. This means that if your app can start it will start. What’s maybe even more important, if it can’t start it won’t start and the test will fail, you will know that something is wrong.

How to write tests?

Shiny for Python testing API

Since Shiny for Python version 1.0, there’s a testing API that allows you interact with Shiny apps in an elegant and convenient way.

Let’s say that you’ve created the slider that squares element and you want to test if it contains some text. You can write a test like this:

from playwright.sync_api import Page
from shiny.playwright import controller
from shiny.run import ShinyAppProc
def test_square_slider(page: Page, app: ShinyAppProc):
    page.goto(app.url)
    given, expected = 2, 4
    slider = controller.InputSlider("slider")
    slider.set(n)
    result = controller.OutputText("result")
    result.expect_text(f"Square of {given} is {expected}")

In the test above, we set the slider to 2 and expect the output text field to contain the text Square of 2 is 4. The controller object allows you to interact with the app in a very convenient way. You can set the value of the slider, click buttons, check if the text is displayed, and many more. For more information on the Shiny for Python testing API check out the official documentation.

Raw Playwright

Using the testing API is very convenient, but it make assumptions about the structure of the app. There are cases in which you want to interact with the app in a more raw way, just like the user would.

In such cases, you can test the Shiny app, the same way you would test a web app.

Let’s consider the following ui element:

ui.a(
    "Start with the docs!",
    href="https://appsilon.github.io/tapyr-docs/",
    class_="docs-link",
    data_testid="docs-link",
),

We’ve added a data_testid attribute to the element, so we can easily select it in the test. The test could look like this:

from conftest import APP_URL
from playwright.sync_api import Page, expect
def test_footer(page: Page):
    page.goto(APP_URL)
    expect(page.get_by_test_id("docs-link")).to_contain_text("Start with the docs!")

conftest.py contents

conftest.py contains the setup for the test. It starts the server in the background and waits for it to start.

import threading

import pytest
import requests
from tenacity import retry, stop_after_delay, wait_fixed
from uvicorn import Config, Server

APP_HOST = "127.0.0.1"
APP_PORT = 8989
APP_URL = f"http://{APP_HOST}:{APP_PORT}"


@retry(wait=wait_fixed(0.5), stop=stop_after_delay(10))
def wait_for_server_to_start(url):
    response = requests.get(url)  # noqa: S113
    response.raise_for_status()  # Will raise an exception if the request is unsuccessful, i.e. server is not ready


def run_server():
    config = Config(app="app:app", host=APP_HOST, port=APP_PORT, log_level="info")
    server = Server(config)
    server.run()


@pytest.fixture(scope="session", autouse=True)
def _shiny_server():
    # Start the server in a background thread
    thread = threading.Thread(target=run_server, daemon=True)
    thread.start()

    wait_for_server_to_start(APP_URL)

    return

The test check if the docs-link element contains the text Start with the docs!. This is a very simple test, but it shows how you can interact with the app in a more raw way.

For more information on testing Shiny for Python apps check out our Introduction to Quality Assurance for Shiny for Python Dashboards with Playwright blog post. For more information on Playwright check out the official documentation.

Tip

Long gone are the days of clicking through the app to check if everything works as expected. Also, long gone are the days when writing end-to-end tests was a pain!

How to run tests?

Run pytest. That’s it!

Note

In tapyr, the setup is already done for you. You just need to write the tests.