How to write maintainable, not brittle, unit tests for a GUI?

Question

I tried writing UI unit tests for my GUI apps and I face the problem that, while they work well when I initially write them, they turn out to be brittle and they break whenever the design changes (that is, quite often). I'm struggling to find a set of guidelines that would lead me to having maintainable unit tests for the GUI.

For now, one thing that I discovered is that tests saying "this component should show its input data somewhere" are good (and that's very easy with HTML). Tests checking for a specific state of a specific part of the component are usually brittle. Tests going like click-click-click-expect, that try to follow the user behaviour and the underlying business logic (which is the most important part) usually turn out to be brittle. How do I write good tests?

---

To be more precise, I'd like to know some patterns about what could I test in my UI, not exactly how to test it. Naming conventions and fixed identifiers are good, but don't solve the core problem, which is, that GUIs change a lot. I'd like to test the behaviours that are most unlikely to change. How to find the right thing to test?

Answer 1

This is a common problem. I would pay attention to:

• How you name elements

  Use css id or class to identify elements. Favor using the CSS ID when the object is unique. Consider the framework you are using, for example with Ruby on Rails the name attribute is assigned automatically and can (non-intuitively) be better than using the css id or class

• How you identify elements.

  Avoid positional identifiers like table/tr/td/td in favor of forms such as td[id="main_vehicle" or td[class='alternates']. Consider using data attributes when appropriate. Even better try to avoid layout tags such as <td> altogether so for the above you could either add a span and use that, e.g. <span id="main_vehicle"> or a wildcard selector such as *[id="main_vehicle"] where the * could now be a div, span, td, etc.

• Using test specific data attributes that are only used for qa and testing.

• Avoid unnecessary qualification for elements. You might find yourself using the following:

  body.main div#vehicles > form#vehicle input#primary_vehicle_name

  However this requires that input field to remain in a form with an exact id of vehicle and on a page with a body that has a class of main and a div with an id of vehicles that has an immediate child of a form with an id of vehicle. Any change to any of that structure and the test breaks. In this case you might find that

  input#primary_vehicle_name

  is enough to uniquely identify the element.

• Avoid tests that refer to visible text. The text on the page that is shown to the user usually changes over time as the site is maintained and updated, so use identifiers such as css id and css class or data attributes. Elements such as form, input and select used in forms are also good parts of identifying elements, usually in combination with id or class, e.g. li.vehicle or input#first-vehicle You can also add your own identifiers, e.g. <div data-vehicle='dodge'>. This way you can avoid using the element IDs or classes, which are likely to be changed by developers and designers. I've actually found over time that its better to just work with developers and designers and come to agreement over names and scopes. It is hard.

• How fixed data is maintained.

  Similar to identifying actual elements, try to avoid in-line hard coded selector identifying values in favor of page objects - small pieces of text that are held in variables or methods and thus can be reused and also maintained centrally. Examples of javascript variables following this pattern for hard-coded values:

  storedVars["eqv_auto_year"] = "2015";
  storedVars["eqv_auto_make_1"] = "ALFA ROMEO";
  storedVars["eqv_auto_make_2"] = "HONDA";`  
  

  More on page objects at selenium wiki and selenium docs

• Communication with developers.

  Regardless of technical approach in terms of 'developers make changes and break the QA automation' that is a workflow issue. You need to make sure that: everyone is one the same team; developer run the same integrated tests; standards are agreed on and followed by both groups; the definition of done includes running and possibly updating the UI tests; developers and tester pair on test plans and both attend ticket grooming (if doing Agile) and talk about UI testing as part of the grooming. You should make sure that whatever approach and strategy you use for naming is coordinated with application developers. If you don't get on the same page you'll like clash over object naming. Some examples of page object methods that I recently created for a ruby project:

  def css_email_opt_in_true
    'auto_policy[email_opt_in][value=1]'
  end 
  
  def css_phone_opt_in
    '*[name="auto_policy[phone_opt_in]"]'
  end 
  
  def css_phone_opt_in_true
    'input[name=phone_opt_in][value=true]'
  end 
  
  def css_credit_rating
    'auto_policy[credit_rating]'
  end
  

  Here's the same page objects as javascript variables:

  storedVars["css_email_opt_in"] = "css=*[name='auto_policy[email_opt_in]']";
  storedVars["css_phone_opt_in"]="css=*[name='auto_policy[phone_opt_in]']";
  storedVars["css_phone_opt_in_true"]="css=input[name='phone_opt_in'][value=true]";
  storedVars["css_credit_rating"]="css=select[name='auto_policy[credit_rating]']";
  

Answer 2

The reason why people developed things like MVC, MVP and presenter first, and similar design patterns, was to separate the business logic from the user interface.

Obviously, the view part can only be tested by starting the program, and checking what it shows - in other words, it can only be tested in acceptance tests.

On the other hand, testing the business logic can be done in unit tests. And that is the answer to your question. Test everything in the model, and if you can and want, you can also test the controller's code.

---

GUIs change a lot

That can only happen when you got changing requirements. When a requirement changes, there is no way around it, except to modify the code. If you manage to create good design and architecture, the change will not propagate in many places.

Answer 3

A common problem with GUI tests... The main reason these tests are considered brittle is because they cannot survive a change in the GUI that is not a change in requirements. You should strive to structure your test code in such a way that a change in the GUI is isolated to a single place in the tests.

As an example, consider a test that is worded as:

When the user enters '999' into the phonenumber field and clicks the save button, the error message popup should say 'invalid phone number'.

Plenty of room here for this test to break when you rework the interface, even if the requirement for the validation stays.

Now, lets put this into a little alternative wording:

When the user enters '999' as a phonenumber and saves the profile page, it should show an error that says 'invalid phone number'

The test is the same, the requirements are the same, but this kind of test will survive a makeover for your UI. You will need to change code, obviously, but the code will be isolated. Even if you have ten or twenty such tests for your profile page and you move your errormessage-displaying validation logic from javascript-alerts to jquery-popups for example, you only need to change the single test part that checks error messages.

Answer 4

GUI interaction tests should be no more or less brittle than any other kinds of tests. That is; if your application is changing in some way, the tests need to be updated to reflect that.

As a comparison:

Unit Test

Original: validateEmail() should throw an InvalidData exception. Which is correctly covered in your unit test.

Change: validateEmail() should throw an InvalidEmail exception. Now your test is incorrect, you update it, and everything is green again.

GUI Test

Original: Entering an invalid email will result in a popup error box containing "Invalid data entered". Correctly detected by your tests.

Change: Entering an invalid email will result in an inline error containing "Invalid email entered". Now your test is incorrect, you update it, and everything is green again.

---

Remember that you're testing for inputs and outputs - some well defined behaviour. Regardless of if it's a GUI test or a Unit test or an Integration test, etc.