UI Testing Automation PoC

Proof of concept for automated UI testing of a Rust GUI application using Python and platform-native accessibility APIs.

• Linux: Uses AT-SPI (Assistive Technology Service Provider Interface)
• Windows: Uses Windows UI Automation API

Project Structure

├── app/                    # Rust GUI application (eframe/egui)
│   ├── src/
│   └── Cargo.toml
├── tests-linux/            # Linux UI automation tests (AT-SPI)
│   ├── test_ui_automation.py
│   └── requirements.txt
└── tests-windows/          # Windows UI automation tests (UI Automation)
    ├── test_ui_automation.py
    └── requirements.txt

Prerequisites

For Running the Application

Install Rust (all platforms):

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Linux GUI dependencies (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install -y \
    libgtk-3-dev \
    libxcb-render0-dev \
    libxcb-shape0-dev \
    libxcb-xfixes0-dev \
    libxkbcommon-dev \
    libssl-dev

Windows: No additional dependencies needed for the app.

For Running the Tests

Linux (Ubuntu/Debian):

sudo apt-get install -y \
    at-spi2-core \
    python3-pytest \
    python3-pyatspi \
    python3-dbus

Windows:

cd tests-windows
pip install -r requirements.txt

Running the Application

cd app
cargo run --release

The application window will show:

• A heading: "Automation Demo"
• A text input field
• A button showing click count

Running the Tests

Important: Tests must be run in a graphical session (not over SSH).

Linux

cd tests-linux
pytest -v -s

Run specific test:

cd tests-linux
pytest -v -s test_ui_automation.py::test_click_button

Note: If you get display errors, set the DISPLAY variable:

DISPLAY=:0 pytest -v -s

This is only needed if running from SSH or a terminal without $DISPLAY set.

Windows

cd tests-windows
pytest -v -s

Run specific test:

cd tests-windows
pytest -v -s test_ui_automation.py::test_click_button

How It Works

Application

Rust GUI app built with eframe/egui and AccessKit for cross-platform accessibility support.

Linux Tests

• Uses pyatspi library to interact with AT-SPI
• AT-SPI is the Linux accessibility protocol that exposes UI elements
• AccessKit bridges the Rust app to AT-SPI

Windows Tests

• Uses pywinauto library to interact with Windows UI Automation API
• Windows UI Automation is the native Windows accessibility framework
• AccessKit bridges the Rust app to Windows UI Automation

Debugging

Linux

Use Accerciser to inspect the accessibility tree:

sudo apt-get install accerciser
accerciser

Windows

Use Inspect.exe (included with Windows SDK) to inspect the UI Automation tree:

• Press Win + R, type inspect.exe, press Enter
• Or download from Windows SDK

Alternative: Use the Windows Accessibility Insights tool

Platform Differences

Feature	Linux	Windows
Library	pyatspi	pywinauto
API	AT-SPI	UI Automation
Find App	Desktop tree search	Window title match
Find Button	Role-based	Control type
Click	queryAction().doAction(0)	.click()

License

MIT