Testing playbook

Run every check on a build installed from a local repository (submitting an app), never on the host, and never with FLATPAK_USER_DIR. Before launching from an IDE terminal, strip ELECTRON_RUN_AS_NODE and ELECTRON_NO_ATTACH_CONSOLE (env -u … flatpak run). Check ~/.local/share/flatpak/overrides/global first: a global override can make a permission look granted when the package doesn’t grant it. When done, flatpak kill <app-id> and clean up as in cleaning up after a run.

In the PR, say honestly what you tested and what you couldn’t; a line per check is enough. Problems a reviewer finds come up in review.

It unpacks as a system install

./scripts/check-apply-extra.sh <app-id>

Downloads the real pinned files, checks sha256 and size, and runs apply_extra.sh as root without capabilities, offline, with only the runtime and /app/extra visible: what a system-wide install does and a --user install never shows. CI runs it on every PR that changes a pin and before every automatic update. Then install with publish.sh --verify.

It starts and draws

Launch on a private display so nothing lands on the user’s desktop, and look at the window:

Xwayland :99 -hidpi -geometry 1400x900 &
DISPLAY=:99 flatpak run <app-id> &
sleep 8; ffmpeg -loglevel error -f x11grab -video_size 1400x900 -i :99 -frames:v 1 /tmp/shot.png
flatpak kill <app-id>; kill %1

• Set DISPLAY=:99 in the host environment. --env=DISPLAY=:99 still binds the real display’s socket and the window opens on the user’s screen.
• Electron: add --nosocket=wayland --socket=x11 --env=XDG_SESSION_TYPE=x11. Tauri: --env=GDK_BACKEND=x11. GTK apps whose manifest has only fallback-x11 need --socket=x11 too.
• A rootful Xwayland :99 accepts XTEST input: click about 0.3 s after moving the pointer, type slowly (about 0.12 s a key), and expect the first click on a new dialog only to focus it. A host input method can grab XTEST keys.
• Look at the screenshot: a blank, black or see-through window is a failure even if the process runs. For WebKit apps, check the web process is alive (cat /proc/<pid>/task/*/children).
• Close the display as soon as the test ends; an idle rootful Xwayland shows as a black window on the user’s desktop.

Data survives a restart

1. Start the app and change something it keeps: a setting, a note, a login.
2. Quit, flatpak kill <app-id>, and start it again. The change must be there.
3. Repeat from a fresh first run with no ~/.var/app/<app-id> and no granted dotdir on the host: :create and --persist behave differently when the folder doesn’t exist yet.

Then look where it wrote: ~/.var/app/<app-id>/ and any dotdir the package grants. An app that writes into $HOME/.something without a grant writes into a folder that vanishes on exit (permissions).

Nothing new in the home folder

ls -A ~ > /tmp/home-before; ls -A ~/.config/autostart ~/.local/share/applications >> /tmp/home-before
# … run the app, use it, quit …
ls -A ~ > /tmp/home-after;  ls -A ~/.config/autostart ~/.local/share/applications >> /tmp/home-after
diff /tmp/home-before /tmp/home-after

Anything new must be the app’s own data in a granted folder. Folders for tools the user doesn’t have, desktop entries or autostart files with Exec=/app/extra/…, and shortcuts on the desktop are failures.

Window class

Measure the class of the app’s main window and compare it with StartupWMClass in the desktop file:

• X11 (on :99): walk every window with XQueryTree and read XGetClassHint. Don’t filter on XFetchName; Electron main windows set only _NET_WM_NAME.
• Wayland compositors: the client list (hyprctl clients -j, the compositor’s equivalent).

They must be equal.

Bus names: tray, media, second launch

DISPLAY=:99 flatpak run --log-session-bus <app-id> 2>&1 | grep -E 'Filtering|RequestName|ownership'
busctl --user list | grep -E 'StatusNotifierItem|org.mpris|<app-id>'

• Any Filtering … line for a name the app tries to own is a failure.
• Tray: if the app has one, a StatusNotifierItem name or registration appears.
• Media: if the app plays audio or video, play something and check an org.mpris.MediaPlayer2.* name appears. Chromium registers it only during playback: start the app with --remote-debugging-port=9333 and play a generated tone over CDP with new Audio(<blob URL of a WAV>). On a machine with no audio device, load a null sink first (pactl load-module module-null-sink).
• Second launch: for a single-instance app, start it again; the second must exit at once and leave one owner on the bus.

The main feature

Do what the app is for, as stated in the PR: play a track, open and save a document, connect to a server, take a screenshot, sync. Use the paths a user would use:

• File dialogs: apps that call the file chooser portal open the dialog on the real desktop, out of reach of :99. A fake portal on a private session bus can answer it, but a private bus can start a second document portal and, when closed, unmount /run/user/$UID/doc; check the mount afterwards and systemctl --user restart xdg-document-portal if it’s gone.
• Saved passwords: check an item was created over Secret Service, and delete test items afterwards.
• Helpers the app shells out to (ffmpeg, gs, git): run the feature that uses them. An app that only checks a command exists reports it available even when it can’t run.
• Can’t test it: say exactly what and why in the PR. Hold the listing until a tester or the developer confirms; don’t merge on the hope it works. For now a tester builds the PR’s branch locally (submitting an app); there is no test remote yet.

What CI does and doesn’t

pr-checks covers the mechanical part: it validates descriptors, audits packaging declarations, checks links, builds the changed apps, and runs the unpack check on changed pins. A check that inspects files or a build without running the app can be added there.

Running the app is not CI’s job. Launching, drawing, data across restarts, the home-folder diff, bus names and the main feature are tested as above, by the packager, and recorded in the PR.