Testing

Testing for the frontend is implemented using Playwright to run the tests and Testomat to report them properly.

Both, Playwright itself and the integration with Testomat are only installed on the development environment. Tests are run against the development environment for early development stages but mostly against the Staging environment for System-Integration-Testing (SIT) and Release-to-Production (RTP) tests.

WARNING

Running test against the development server website more often than not fails for the SIA. This seems to be related to a combination of Vite's Hot Module Replacement and the fact that Vue.js does check type assertions in development mode. The SIA fails especially when reloading with a rather strange error Right-hand side of 'instanceof' is not an object. Spent many hours trying to fix this - but could not succeed. As this never happened in Staging or Production environment this was left alone. However, tests for the SIA should never be run against the development environment!

Playwright provides reporting capabilities through a web report which is however rather limited in its analytic capabilities and has close to none historic reporting capabilites. Testomat is hence used for reporting and archiving.

In order to utilize Playwright's codegen (hardly used) and debug capabilities a graphical user interface is required on the server where tests are run. As all servers in the existing infrastructure do not provide this generically, TigerVNC server (for installation see here) on the server itself and a RealVNC client under Windows are used.

For installation details see below.

Install Playwright

Pre-requisites: node.js must be installed with minimum version 20+

Playwright has to be installed with sudo permission to be able add the required linux OS depedencies, especially the browser binaries Playwright is supposed to test with.

When node.js or the package manager (yarn) are installed in user space, using sudo will however not work out-of-the-box. The path to yarn must be added to the sudo default path first with:

which yarn
sudo visudo

add yarn binary path to secure_path

Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/ubuntu/.nvm/versions/node/v20.18.1/bin

Only then we can install Playwright correctly with:

sudo yarn create playwright

answer yes, when asked to install dependencies.

After installation has finished, running:

npx playwright test

should run through the example tests without error.

Testomat

To synchronize the local tests with Testomat the npm package check-tests must be installed (this is done automatically in the background when utilizing the Testomat sync command line, which can be found under Tests -> ... -> Import automated tests on the Testomat website). But, it can also be installed manually with:

yarn add -D check-tests

IMPORTANT

For all synchronization activities Testomat needs a sync key set as an environment variable (either directly in the command line or for convenience loaded in the development servers .bashrc).

export TESTOMATIO=tstmt_xxx

To actually synchronize the tests with Testomat run

TESTOMATIO=tstmt_xxxxxxx npx check-tests@latest playwright "**/*{.,_}{test,spec,cy}.js"

To forward the test results directly to Testomat we need to set up the reporter accordingly in the playwright.config.js. If results from locally run tests shall be uploaded to Testomat, the respective npm package must be installed locally:

yarn add @testomatio/reporter

The reporter entry in playwright.config.js must be changed to:

  reporter: [
    ["list"],
    ['@testomatio/reporter/lib/adapter/playwright.js', {
    apiKey: process.env.TESTOMATIO,
  }]
  ],

Now, when running

npx playwright test

Tests are executed and results are automatically uploaded to Testomat

For convenience a number of scripts were defined in package.json to run tests with different scopes:

    "test:local": "npx playwright test --project=chromium-dev --reporter=line",
    "test:staging": "TEST_ENV=staging npx playwright test --project=chromium-staging --reporter=line",
    "test:report": "TEST_ENV=staging npx playwright test --project=chromium-staging",
    "test:sync": "npx check-tests@latest playwright -d tests '**/*{.,_}{test,spec,cy}.js' --update-ids",
    "test:gen": "~/scripts/frontend/codegen.sh"

where ~/scripts/frontend/codegen.sh is implemented as:

#!/bin/bash

# Start VNC server if it's not running
if ! pgrep Xtightvnc > /dev/null; then
  echo "Starting VNC server..."
  vncserver :1 -geometry 1920x1080
else
  echo "VNC server already running."
fi

# Run Playwright codegen
export DISPLAY=:1
npx playwright codegen https://klharriettes.club

Running Tests

INFO

GitHub actions is a nice experiment 🧪 - but a bit difficult to handle. Although this is implemented and should work seamlessly, it runs very long and consumes a fair bit of resources. Moreover, there is little control over the test run. Running the tests locally on the developement server and reporting it using the @testomat/reporter seems for now to be the better option. For details, see the script below.

GitHub Action

Testomat cannot launch tests/test suites generically and needs a GitHub Action to be implemented in .github/workflows:

name: Testomatio Tests

on:
  workflow_dispatch:
    inputs:
      grep:
        description: 'tests to grep '
        required: false
        default: ''
      run:
        required: false
      testomatio:
        required: false

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Install Node
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: yarn install --frozen-lockfile

      - name: Install WebKit dependencies
        run: npx playwright install-deps
      
      - name: Install Playwright browsers
        run: npx playwright install
      
      - name: Install check-Tests
        run: yarn add check-tests
      
      - name: Install @testomatio/reporter
        run: yarn add @testomatio/reporter

      - name: Ensure all tests are available in testomat.io
        run: npx check-tests@latest playwright "**/*{.,_}{test,spec,cy}.js" 
        env: 
          TESTOMATIO: ${{ secrets.TESTOMATIO }}

      - name: Run Playwright tests
        run: |
          echo "Testomatio Run: $TESTOMATIO_RUN"
          echo "Testomatio Key: $TESTOMATIO"
          npx playwright test --grep "${{ github.event.inputs.grep }}"
        env:
          TESTOMATIO: "${{ github.event.inputs.testomatio }}"
          TESTOMATIO_RUN: "${{ github.event.inputs.run }}"
          VITE_BACKEND_PATH: "https://admin.klharriettes.club/api"
          VITE_API_TOKEN: "${{ secrets.API_TOKEN }}"
          VITE_PIWIGO_PATH: "https://gallery.klharriettes.org/ws.php?format=json&method="
          VITE_PIWIGO_USERNAME: "${{ secrets.PIWIGO_USERNAME }}"
          VITE_PIWIGO_PASSWORD: "${{ secrets.PIWIGO_PASSWORD }}"
          VITE_COMMITTEE_PWD: "${{ secrets.COMMITTEE_PWD }}"
          VITE_ONCASH_PWD: "${{ secrets.ONCASH_PWD }}"
          VITE_ENV: "test"
          TEST_ENV: "staging"

The GitHub action needs to be linked by some .env variables to Testomat and must be configured once in Testomat itself. The setup process is well explained here

Subsequently, test can be run with the "Run automated Tests" button directly from Testomat.

WARNING

The GitHub Action runs the full test scope, i.e., tests against all browsers (Chrome, Firefox, and Safari) and additionally a test on a mobile device. Use this considerably.

The local test executed on the development server are only running against Chrome on a Desktop device

Running tests locally using the run_tests script

In cooperation with Claude a script to flexibly run the tests directly on the development server. The script run_tests.sh is located in the main directory and can be reviewed there. For orientation only the help message will be repeated here:

Multi-Project Playwright Test Runner (Local Testing)

USAGE:
    ./run_tests.sh [OPTIONS]

OPTIONS:
    -e, --env "environment"           Test environment: dev or staging (default: staging)
    -p, --projects "project1,project2"  Comma-separated list of projects (required)
    -s, --suffix "Custom String"        Custom suffix for run names (default: "Test Run")
    -g, --grep "pattern"               Grep pattern for test filtering
    -t, --testfiles "file1,file2"      Comma-separated list of test files
    -c, --config "path"                Path to Playwright config file
    --group "Group Name"               Group name for Testomat runs
    --labels "label1,label2"           Comma-separated labels for Testomat runs
    --reporter-key "key"               Testomat reporter key (overrides TESTOMATIO env var)
    --quiet                            Only log to file, minimal console output
    --dry-run                          Show commands without executing
    -h, --help                         Show this help message

ENVIRONMENT VARIABLES:
    TESTOMATIO                         Testomat reporter key (can be overridden by --reporter-key)

EXAMPLES:
    # Basic usage - runs against staging by default
    ./run_tests.sh --projects "chrome,firefox" --suffix "Smoke Tests"

    # Run full staging regression
    ./run_tests.sh --env "staging" --projects "chrome,firefox,safari,mobile" --suffix "Full Regression"

    # Run against local dev server
    ./run_tests.sh --env "dev" --projects "dev-chrome,dev-mobile" --suffix "Dev Tests"

    # Run specific test files with pattern matching
    ./run_tests.sh -e "staging" -p "chrome,mobile" -t "login.spec.ts,checkout.spec.ts" -g "should login"

    # Quick smoke test on staging
    ./run_tests.sh --projects "chrome" --suffix "Quick Smoke" -g "@smoke"

    # With group and labels for organization (project auto-labeled)
    ./run_tests.sh --projects "chrome,firefox" --group "Daily Tests" --labels "smoke,critical" --suffix "Daily Run"

    # Environment-specific with labels (project auto-labeled)
    ./run_tests.sh --env "dev" --projects "dev-chrome" --labels "local,development" --suffix "Dev Testing"

    # Group runs and use custom config
    ./run_tests.sh --projects "dev,staging" --group "Daily Regression" --config "./custom.config.ts"

    # Dry run to see commands
    ./run_tests.sh --projects "web,mobile" --dry-run

NOTES:
    - Each project run will be named: "YYYY-MM-DD Custom String" (project in labels)
    - Project name is automatically added as a label in Testomat.io
    - All output is logged to: test-results/runs/YYYY-MM-DD_HHMM_suffix_env.out
    - Default environment is 'staging' (https://klharriettes.club)
    - Use 'dev' environment for local development (http://localhost:3000)
    - Test environment files: .env.test.dev and .env.test.staging
    - Available staging projects: chrome, firefox, safari, mobile
    - Available dev projects: dev-chrome, dev-mobile
    - Groups and labels help organize runs in Testomat.io
    - Test results will be saved locally in test-results/ and playwright-report/
    - Works with nohup for background execution: nohup ./run_tests.sh [options] &
    - Set TESTOMATIO environment variable for Testomat.io integration (optional)

Unfortunately, Testomat does not support labels in the free plan. In order to structure tests properly only tags (the --group parameter of the script) can hence be used. The script logs the entire commmand line output into a log file test-runs/2025-08-21_1758_System_integration_test_staging.out where the filename itself is concatenated from the test start time, the --suffix and the --env parameter.

An example run command would be:

nohup ./run_tests.sh --env "staging" --suffix "System Integration Test" --projects "chrome" --group "staging:development,sit,frontend:v1.2.6,backend:1.1.7" --quiet &

VNC

Running Playwright's codegen (which was hardly used when developing the tests) or the debugger with --debug (which is an important helper during test development) both require a graphical user interface to run.

For the usually cli driven Ubuntu servers used for development and staging we need to install a window manager and connect to it using a remote client like, e.g., RealVNC

In the following the installation process for XFCE is described. Using TigerVNC is a valid alternative.

1. 🔧 Install XFCE (lightweight desktop) + VNC server

sudo apt update
sudo apt install -y xfce4 xfce4-goodies tightvncserver

XFCE is light and fast — ideal for VM environments.

2. 🔐 Set up VNC server Run this once to create a VNC password and config:

vncserver :1

In the dialog:

• Set a password (8 characters max).
• Say No to a view-only password.

INFO

running vncserverwith the :1 arguument means it listens on port 5901 (i.e. 5900 + 1)

Stop vncserver for now so we can configure it properly:

vncserver -kill :1

3. 🛠️ Configure VNC to launch XFCE Edit the startup script:

vim ~/.vnc/xstartup

Replace everything with:

#!/bin/bash
export XKL_XMODMAP_DISABLE=1
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

xrdb $HOME/.Xresources
startxfce4 &

Then make it executable:

chmod +x ~/.vnc/xstartup

WARNING

This procedure does not setup vncserver to start automatically upon boot. It needs always to be started when required with vncserver :1 like described below

4. 🚀 Start the VNC server

vncserver :1 -geometry 1920x1080

5. 🖥️ Connect from your Windows laptop Download RealVNC

Connect to: server-ip:1

Use the password set earlier.

✅ You should see a desktop session!

6. 🧪 Run Playwright codegen in a terminal or a npx playwright test --debug to start Playwright in debugging mode, where you can execute every step one-by-one. In order to run a test until a certain point add await page.pause() where you want Playwright to stop and then click the play button in the dev tools window.