How to Unit-Test a parser of a file?

Question

I'm implementing a metadata parser of image files from all formats. I want to write tests for it. One trivial way to do so is to have test image files of all formats as a resources for the tests, and actually to read them as input. This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O. Is it a good way to do so or are they any alternatives?

Let the parser work on a block of memory that contains the image data. — πάντα ῥεῖ, Mar 18 '18 at 08:29
@πάνταῥεῖ Yes, this is the second option. The problem with it is that some formats (e.g. RAW ) may contain up to 50MB of data. Is it feasible to generate such a block for test? — Sanich, Mar 18 '18 at 08:33
Wtite the test. Write it in the easiat way possible. I think I would go and just read from files, as this is probably what you will need to do in integration and acceptance testing. — Robert Baron, Mar 18 '18 at 10:39
Ps: I know it is not the pure tdd solution, bit I really don't like to do the same thing twice. — Robert Baron, Mar 18 '18 at 10:41
Probably, TDD's worshipers will curse me after say this but, not every single line of code worth a UT. And much less a dogmatic approach of TDD. If your solution is light and simple, the complexity of the tests should not exceed that complexity. Otherwise, the costs of the QA exceeds the benefits. In other words, if programming and running the tests takes longer than coding and evolving the solution, then you are wasting time and money. That said, I would weight which of the "moving parts" worth to be tested unitarily and make everything else e2e test. Just make them deterministic. — Laiv, Mar 18 '18 at 14:53
@Sanich Does the `parse` method you want to test need to be fed 50 MB of data to work? — Stop harming Monica, Mar 18 '18 at 16:05
@Goyo It should work on a valid file data that in some cases can be 50MB. — Sanich, Mar 19 '18 at 08:02
If you need to verify that it works with data that size then you should have at least one test that uses data that size. It's perfectly feasible, 50MB isn't exactly a huge amount of data. Yeah, the test will be a bit slower than most of the others, but it will probably only be one or two tests that need that size, most of them can work with a much smaller set of data. — Sean Burton, Mar 19 '18 at 14:16
You can split the tests into small format parsing tests and larger load testing tests. You don't need a 50MB file to ensure that the _header_ has the information you expect. However, in your integration tests, you can use 50MB+ files in your tests where I/O is allowed--just to make sure things still work. — Berin Loritsch, Mar 19 '18 at 16:23

score 12 · Answer 1 · edited Jun 16 '20 at 10:01

I want to write tests for it.

What you are intending to test?

I want to use TDD. I'm refactoring a parser and want to test the 'parse()' method.

So the aim is to clean things up.

I would argue that refactoring legacy code isn't 100% compliant w/ TDD.

Bad code restricts testing.

More importantly - intention to clean it up (the drive - reason for changing code) differs from original intention for code to do whatever business domain stuff.

step 1

I would start with a sloppy integration test/s that covers most of the functionality.

Feed tests crude input - e.g. those 50mb resource files.
Ask only polished output and ignore internal stuff.

It's actually important - higher test abstractedness is what loosens implementation restrictions.

That will give you a safety net so you can open up code for refactoring w/o fear.

step 2

Once you have that - you are ready to actually go in & refactor.

Read the code. Start small. (good book)

Things like code formatting, removal of excess white-space, removal of too verbose variable prefixes.

Then move forward to structural changes - extract methods, interfaces, classes where needed.
And don't just divide & conquer - try combining stuff where it "makes sense" ™.

Only with decent code structure you will be able to write unit tests for isolated units of functionality.

If the integration test you started with performs well enough - I wouldn't even bother trying to build unit test network.

Either way - proper code structure will lead you to natural & easy to stub I/O seam.

Once the network of unit tests is strong enough - remove integration test/s.
Or stub the input the same way as in unit tests (sort of devalues integration test).

score 8 · Answer 2 · answered Mar 19 '18 at 17:52

This approach may work but as far as I understand from unit test methodology, unit-tests shouldn't perform I/O

I think you are overlooking the elephant in the room here: your parser should not perform I/O.

Your parsers job is parsing data, not disk I/O. Whatever language you are using, it probably has the concept of a stream. That is where your parser gets data from. And whether that happens to be a FileStream or MemoryStream or even std::cin is not your parsers business.

So back to your tests: It does not matter. Your tests will have the data to test your parser. And whether that is written on disk in separate files or resource manifest streams or hardcoded byte-arrays... does not really matter. You CI automation should be able to work with it and that's all you need to know. In the end, the data is on disk. Where else would it be? Even your executable code is loaded from disk.

So to sum it up: make your parser independent from disk access. Separation of concerns. Then have your tests load data in the way that is most elegant for your CI solution so your tests don't fail when someone else runs them on their machine (someone else might as well be your CI agent). How you achieve that is a detail that nobody really cares about as long as it works.

Exactly, I may have my test pull something from a file when it is convenient (big chunk of data, a large list, etc.), but not be testing I/O functionality. — JeffO, Mar 20 '18 at 16:07

Doc Brown · Answer 3 · 2018-03-18T13:30:13.317

4

as far as I understand from unit test methodology, tests shouldn't perform I/O

You have to distinguish which kind of tests you are talking about:

if you want to use TDD with small "write test" - "write code" - "refactor" cycles, then you need very quick unit tests, ideally with small data sets and no I/O.
when you want to do acceptance or integration testing, especially by using several external images, then using files & I/O is not just perfectly fine, but probably required.

So decide on the basis which kind of tests you are writing.

edited Mar 18 '18 at 13:30

answered Mar 18 '18 at 09:31

Doc Brown

199,015
33
367
565

I want to use TDD. I'm refactoring a parser and want to test the 'parse()' method. This is a unit-test. (By no means an acceptance or integration). So the quick way to write a test is to read the input from a test file. But as might be implied from your answer to my other question, the better but much longer way is to use a cache-free solution. – Sanich Mar 18 '18 at 09:44
4

"Parse a complicated file format" doesn't sound like a unit test to me - you'd have separate independently testable components for the header, the body and whatever else. You unit test those components, mock them out when unit testing the whole parser, and then integration test the whole parser. – Philip Kendall Mar 18 '18 at 10:46
"the better but much longer way is to use a cache-free solution" - no, you got me wrong on this: I was assuming if a cache-free solution is less (or at least not more) effort than a solution with a cache. Let me reword my other answer to make this more clear. – Doc Brown Mar 18 '18 at 13:41

score 2 · Answer 4 · answered Mar 19 '18 at 16:32

When testing parsers, there's at least two modes of testing that need to happen:

Testing in the micro: the parser knows how to parse a phrase (a set of bytes or characters)
Testing in the macro: the parser needs to parse a whole set of phrases

If your parser is modal, then you'll have tests to ensure the mode transitions are handled properly.

The bottom line is that they are two different sets of tests. Usually I end up splitting the work this way:

Unit Tests are for testing individual phrases
Integration Tests are for testing whole files, and problem cases

Usually I can set up the bytes in memory for the micro tests, and use the filesystem for the integration tests. This has served me well. That said, creating test phrases for textual parsers (the bulk of what I've done) is much easier than doing the same for binary parsers. I have done some binary parsing, so my test case will likely have what looks like magic arrays of bytes, but it works.

score 1 · Answer 5 · edited Mar 19 '18 at 17:54

If I understood you correctly, you have two goals

add new features/functionality/refactoring to the existing code
While extending/refractoring the existing code, make sure that the current functionality does not break (aka "regression test" or saftey net)

If I were you, I would start with the "make sure that the current functionality does not break" part as a file based integrationtest with

a folder of example images
for each image there is a textual representation of the exprected result.

This (probably slow) regression test would iterate over all image files and compares parsing with the expected file.

Once this is working you can start thinking about TDD with new features, which is something completly different from the proposed regression test.

For the TDD part of the question we cannot help you without seeing code. (question is too broad)

score 0 · Answer 6 · answered Jan 01 '20 at 21:53

A parser is a syntax-directed translator that converts a set of input words conforming to a base grammar to output "words", the words normally taking the form of a sequence of actions.

Reverse the parser. Instead of it recognizing a base language and producing outputs for the input words it recognizes; have it generate the words of the base language and feed that into the parser. That means that at each state of the original parser, instead of accepting an input word and testing its validity and carrying appropriate actions for the input accepted, the reverse-parser generates the input word (both valid and invalid) using a probability metric. So it becomes a stochastic automaton feeding off of a random number generator that generates the base language + errors (for stress testing).

Thus, the tester is just the parser itself - run in reverse - to generate words in the base language. In effect, it becomes a translator that converts the random stream of a random number generator into the output language that, itself, is (apart from intentionally generated errors) the input language of the parser.

The test suite then consists of a set of automatically generated random number seeds. Each update of the parser is regression tested against its predecessor on the same set of automatically generated test cases.

The test cases are generated with the reversal of the original parser. But after sufficient development, you may then reverse the upgraded parser and use that as the test generator instead. Then, the test-generator has to be regression-tested by running it against the previous test-generator to ensure it produces the same test cases off the random number generator. In the process, you may design the upgraded parser so that it can be more easily reversed.

Thus, both tester and parser are bumped up side by side.

I think your answer is interesting, but I fear it might be a little too abstract (not providing a response for an image file scenarios). It's also not clear how a parser is 'reversed'. It might have been easier to understand to say 'find a way to generate all possible permutations of file image headers' — Martin K, Jan 05 '20 at 20:03
This approach is all very well, but it seems like an automated regression testing approach, to find unintentional deviations of parsing behaviour in new versions. It doesn't deal with the question of how you test the original version of the parser! — Robin Green, Dec 31 '21 at 22:39

How to Unit-Test a parser of a file?

6 Answers6

step 1

step 2