3

I am currently working on a library that provides functionality for interactive terminal applications.

For now, I implemented functionality for reading an arbitrary keypress from the User to the application as well as setting the currently active color.

I want my library to support a broad spectrum of terminals on different OSs.

Currently, I had to test every newly added feature by hand in a multitude of terminals manually which took a lot of time.

To simplify the task of testing and allow for regression testing, I want to write a test script which automatically opens a Terminal, reads some automated input and writes back colored text. I want my test to automatically check whether the correct Key Sequence was read and whether the color is displayed.

What would be the best approach of doing this?

The solution that I came up with is based on creating a virtual x server using Xvfb, starting a terminal, send some input & output, screenshot what I have so far and analyze the screenshot (e.g. by comparing it to a predefined image).

Thanks for any help.

EDIT: doc-brown suggested that I should give the screenshot method a test. I hacked something together that creates a new xvfb instance, launches a test script in a new xterm instance and takes a screenshot:

screenshot of eight color blocks inside a terminal window

(note that they are almost square because I'm printing double spaces)

Now for the analysis part. Currently, I export the image as both an xwd and png file. Now I'm going to read those files and check for the colors in each cell.

For that I need to get the default dimensions of the terminal (how much border, padding is there, how big is a single cell).

writzlpfrimpft
  • 139
  • 3
  • 1
    While your question is valid, what you're describing here isn't a unit test: unit tests test one piece of code in isolation from everything else; what you're describing is more like an integration test. – Philip Kendall Feb 27 '21 at 10:45
  • 1
    I guess your current screenshot solution works similar to the simplified example [in Wikipedia](https://en.wikipedia.org/wiki/Xvfb)? And the automation of analyzing the screenshot is most probably the issue which gives you headaches? If I am right, maybe you could post two example screenshots to give us an impression what you are facing. – Doc Brown Feb 27 '21 at 11:15
  • @PhilipKendall I've removed the offending tag and invented a new one. [Feedback?](https://softwareengineering.meta.stackexchange.com/q/9231/131624) – candied_orange Feb 27 '21 at 12:50
  • I was actually interested in how much those two images you usually need to compare differ from each other, to see if there is a chance for making that kind of comparison automatic. – Doc Brown Feb 27 '21 at 20:57
  • @DocBrown I guess the test should fail if some of these colors are swapped / not drawn – writzlpfrimpft Feb 28 '21 at 13:37
  • @writzlpfrimpft: sure, but how much can the images differ? Do the colors need to have the same RGB values? Do you expect the images have identical pixels? If not, where do you draw the line between acceptable and not acceptable differences? – Doc Brown Feb 28 '21 at 21:06

1 Answers1

3

Sampling

Run the test, take the screenshot compare it to all previous screenshots taken in the last 12 months (or 100 runs, or some other arbitrary limit). If it doesn't match 100% add it to a list of to be manually confirmed.

Get a human to look see and either say yes passed, or no failed. Add the image to the catalog of images for that test. The next time the test runs it'll compare against this picture and if it matches 100% will pass or fail as tagged.

Eventually you could try training an AI to estimate how similar the two pictures are. Or you could write your own similarity algorithm tailored to each kind of test and have some acceptable threshold (like purple is allowed to be one unit of red off, or the similarity of 95% is okay).

Mock Terminal

Literally write your own virtual terminal for the tests. It doesn't draw to the screen but interprets the key commands and constructs a representation of the screen that can be easily interrogated. Is this character here purple?

Kain0_0
  • 15,888
  • 16
  • 37