11

I have a number of (large) test files that need to be maintained. This means access to their history is a requirement.

Benefits

  • Any new developers get the entire test suite with just a git pull.
  • The history of the files is backed up.
  • The files themselves are backed up.

Drawbacks

  • Huge increase in size of repository.
  • Huge increase in download size of new developers copying the repository.

What are the best practices for maintaining test files?

Do I store these files in source control? Are there any alternatives?

sdasdadas
  • 511
  • 1
  • 3
  • 12
  • 8
    Have you considered storing them in a different repository, one that will be pulled only when it's absolutely necessary? A git submodule perhaps? – yannis Sep 26 '13 at 18:14
  • If you decide not to version the actual files, version a list of hashes of the files so at least you can confirm you have the right ones when trying to repeat tests. –  Sep 26 '13 at 19:00
  • @YannisRizos I really like the idea of a submodule, thanks. – sdasdadas Sep 26 '13 at 20:06
  • 2
    If you have some really huge files that don't change often in the mix, [git annex](http://git-annex.branchable.com/) might be interesting too. But that's talking about files that have on order of hundreds of megabytes or more. Smaller files are fine just stored in git even if there is many of them. – Jan Hudec Sep 27 '13 at 07:11

3 Answers3

13

Store them in source control. The benefits you listed are all very good ones. When you say that doing this will result in a "huge" size, just how huge are you talking? 100's of gigabytes? Terabytes?

If storage is really that much of a problem, could you zip the files, store the zip file in source control and then have a script that unzips them when the test cases are run? You'd lose detailed history of each file that way (unless you can find a tool that will unzip and show history in memory), but new developers would still have easy access to test files.

Depending on the nature of your test data, you could have a script to generate test files. This might work if you are testing very large images that can be generated procedurally (SQL inserts to populate a database can also be easily generated by a program or script), but this technique does not apply to all kinds of test data...

FrustratedWithFormsDesigner
  • 46,105
  • 7
  • 126
  • 176
  • 7
    Storing the zip file in source control will result in more storage used than just the text, since it will have to keep binary blobs instead of just diffs. – Daenyth Sep 26 '13 at 18:15
  • @Daenyth: I guess it might depend on how much the files can be compressed - if they can be compressed, and they don't change too often (so there aren't too many diffs) it might be worth it. Impossible to say for sure without details about these files. – FrustratedWithFormsDesigner Sep 26 '13 at 18:39
  • 1
    Huge (in this instance) means 10s or 100s of GB. – sdasdadas Sep 26 '13 at 20:05
  • 1
    @sdasdadas: Hmm 100's of GB could get big over time... is compression an option? Or generating test data with small scripts? – FrustratedWithFormsDesigner Sep 27 '13 at 13:50
  • @FrustratedWithFormsDesigner I guess I can always compress them if it gets out of hand. A generating test script sort of defies the point since this is supposed to be "real world" tests - scripts tend to make contrived data. – sdasdadas Sep 27 '13 at 20:36
5

I'd store them in source control. Is there ever a case where you want the test suite without the code? Doesn't make sense. Is there ever a case where you want to edit the code without updating the test suite? There shouldn't be.

Drawback 1 shouldn't be a problem given the storage prices these days, and modern SCM systems should make the network activity not a concern outside of initial setup either.

Your drawback 2 doesn't seem like it should be much. How frequently do you have new developers coming on board this project that initial clone time becomes an issue?

Daenyth
  • 8,077
  • 3
  • 31
  • 46
2

I would store them in a different repo . The drawbacks you mentioned are inevitable if you have a bulky test suite.

vpillai
  • 21
  • 3