Testing VSCode extensions

Dan Čermák <dcermak@suse.com>

FOSDEM 2021, 6.2.2020

This work is licensed under CC BY 4.0

whoami

Dan Čermák

Agenda

Visual Studio Code

OBS Connector

SUSE/open-build-service-connector

Roadblocks

  1. need to connect to OBS
  2. have to store passwords
  3. unit tests are brittle

Solutions:

  1. use upstream's development environment
  2. use a dirty LD_PRELOAD trick
  3. test the workflows of the extension

Testing Extensions

Why is it hard?

  • UIs cary a lot of state
  • testing workflows is uncommon
  • what you get is not what you see

Unit testing

  • test one functionality of one unit/function
  • anything touching the UI needs extensive setup & teardown
  • external services need to be mocked

Getting started

  • documentation has an example setup
  • code coverage setup more involved

Extension Settings

  • can be read & modified in tests

clean up after yourself!

Events

quickPick.onDidChangeValue(async (val: string) => {
  if (verifyInput(val)) {
    await launchBackgroundTask();
  }
});

→ use fake events when possible

Disposables

  • "destructors" in VSCode
  • use after() or afterEach()

UI Elements

  • only check the interesting parts
  • preferably keep UI part as small as possible

Manual testing

Do it yourself

Make a test plan

Integration testing

automated execution of your extensions' workflows

vscode-extension-tester

redhat-developer/vscode-extension-tester leverages selenium webdriver
const editor = new TextEditor();
const pkgJsonEditor = await new EditorView().openEditor('package.json');

await pkgJsonEditor.setText('{"foo": [1, 2, 3], "bar": "baz"}');
await pkgJsonEditor.formatDocument();

What to test?

  • check your main workflow(s)
  • don't test corner cases & minor regressions

How to test?

  • upstream using mocha
  • use root hooks for setup
  • run steps as individual it()

Catches

  • integration tests tend to be very slow and resource demanding
  • avoid explicit sleeps
  • certain elements invisible by default
  • no test coverage

Legal

Questions?

Thank you for your time!