0

I have just released an iOS app for beta-testing with TestFlight.

I'm wondering what is good practice during beta-testing?

  • Should I halt development of the app during beta-testing?

  • Should I continue developing but make a snapshot image of what the code looked like when it was released for beta-testing?

  • Issues that could rise that I can think of is that if I make any changes to the code and then a bug is detected in the code that I have changed this would not be ideal.

There are probably other issues to deal with but these are the ones that I could think of at the moment.

I'm the only developer of the app at the moment.

gnat
  • 21,442
  • 29
  • 112
  • 288
Peter Warbo
  • 297
  • 1
  • 2
  • 10

1 Answers1

7

When you ask if you should make a "snapshot", it sounds to me like you're not using source control - or if you are, you're not taking full advantage of it. I would VERY strongly recommend getting hold of a client for Git or Mercurial (Git has better integration with iOS dev tools, if I remember correctly) and setting up a repository on Github or Bitbucket. You can set it to private, so you wouldn't be giving the world access to your source code, and it keeps your code safe from any mishaps, from accidental deletion to loss or damage to the computer you're working on. It also lets you roll back to previous versions in case of problems, compare to earlier versions to find changes that might have caused some behaviour you're trying to track down, and - crucially for your question - it preserves revision history and allows branching.

A source control branch basically lets you split off a version, work on it, and then decide whether to merge it back into the original version. So, you could create a branch from where you are now, and that branch would preserve the state of your code as it was when you gave it to the testers. Any features you add from now on would be kept separate in one branch, and if the testers find bugs, you can fix them in the other branch. Each time you fix a bug, you check to see if the "features" branch needs the fix as well - if it does, you can just merge the fix into both branches. When the test is over, you can pretty much forget about the "testing" branch because you know that any fixes you need to have in the main version were merged in already.

There are various patterns and techniques to determine when and why to branch, all aimed at solving exactly the problem you're describing. A popular one is the "Git-Flow" pattern, although (in my opinion) that has the flaw that it looks rather more complex than it needs to be (I've used it a fair bit, and it's not as complex as it looks, but it's not as simple as it could be, either).

My advice would be to:

  • get Git installed
  • get a Bitbucket or Github account
  • shove your current version of the code into a repository using the above
  • make a branch called "version-1"
  • from "version-1", make a branch called "version-2"
  • keep new features in "version-2"
  • fix bugs that the testers find in "version-1" and rebase "version-2" each time you do
  • when the testers stop finding bugs, forget about "version-1", make a new branch from "version-2" called "version-3", give "version-2" to the testers and repeat

You can keep doing that until you run out of new features to add, and then you just get the current version tested until they stop finding bugs, and then when you've fixed the bugs, release the app.

Edit:

The difference between "merge" and "rebase":

Merge does pretty much what it says - it tries to combine the two versions. If one version has changed an area and the other hasn't, that's easy - it takes the changes. If both versions have made changes to the same area, it'll ask you to pick one or the other. There are a variety of diffmerge tools designed to help you work out what's been changed in each version, compare the two side by side, spot where the conflicts are and work out which you need to take.

Rebase is a little different. When you rebase a branch, it "unwinds" the changes you've made to that branch - basically doing them backwards and remembering them. When the branch is back to how it was when it was branched, it gets the current version of the parent branch. Then, it attempts to replay your changes back on top of the new version. Basically, it's as if you'd made the branch now, not a few days ago or whenever. This can be a really powerful tool, and it's a great way of including changes to parent branches in your code. Imagine you have a version being tested in one branch, and you're working on new features in a second branch. Using the method I described above, the "test" branch would be the parent of the "new features" branch. When your testers find a bug, you jump into the "test" branch and fix it. Now, if you'd waited until after the bugfix before starting to develop new features, then you could have built those features with this fix already in place. Rebase basically tries to act like a time machine so that you can do just that. It undoes what you've done, gets the current version, and then makes your changes to it. Obviously, in some cases there will be changes in your parent branch that affect the changes in the branch you're rebasing, so it won't always work, but in my experience it's generally pretty good. When it doesn't work, you can always just use a regular merge, but when it does work, it's really useful - and like I said, in my experience it tends to work quite well. Sometimes you'll have to go back and tweak after rebasing, but often it Just Works.

Git-Flow (ish)

This isn't quite Git-Flow, but it takes the principles. One of the main criticisms of Git-Flow is that it can get complex and leave you with loads of branches; this tries to address that a little.

  • Master branch. You only put working, tested code in here, and you never actually push to it directly - you just merge it in. Each new commit to this branch is a new release to the public.
  • Development branch. This is branched from master. This represents the state of the version you're working on. You only put stuff in here when you're pretty sure it's working, although sometimes bugs will turn up.
  • Feature branches. When you want to add a feature, you make a branch off development, build the feature in there, and merge back in. These have names starting with "feature/" to help you organise. Name them something that explains what they're for, like "feature/SearchFunction" or whatever. When the branch is merged in, you can close it. You can leave the branch open if you want, maybe to improve the feature or as somewhere to collect fixes for the feature if it turns out you need any, but remember to rebase the branch before you start work on that. Either way, the next time you merge development into master you should close any feature branches that had already been merged in. If you have a feature that you want to save for a future release, or that isn't finished yet, then you wouldn't merge this into development until it's ready, so it can stay open from one release to the next.
  • Bugfix branches. If you find a bug in development and there isn't an open feature branch to fix it in, create a branch whose name starts with "bugfix/" and fix the bug in there. When you've checked that the fix works, merge the branch back into development and close it. If the bug is in a feature you've merged in but whose branch is still open, you can rebase that branch, fix the bug in there, and then merge it in again.
  • Hotfix branches. If you find a bug in master, then that means it's in the version your users have. This can't wait for the next version. Make a branch from master whose name starts with "hotfix/" and fix the bug in here. Don't merge it back in until you're sure it works. When you merge it back in, you'll need to merge it into development as well, or else you'll go back to the old buggy code when you next merge development into master. You then release the contents of master as a patch release.
  • Release branches. At some point, you have to say "ok, no more new features - we need to get this version bug-free and released". When this happens, you make a branch from development. This branch is given a version number; whichever comes after the last version you released. You can then give this branch to your beta testers. Only bugfixes are allowed in this branch. When it's done and you're ready to release, you merge it into master, and into development to make sure the fixes don't get overwritten. Then you can release the contents of master to your users.

This can lead to a lot of branches, but generally speaking, only a few of them exist at any given point. Master is always there, and you can almost always ignore it. Just remember: when code goes into master, build that code and release it as your next version. Development is also always there, but again, you can kind of ignore it; nobody will be working directly in this branch. If you have any in-house testers (not beta testers), they'll be testing what's in here; it's kind of the "alpha-test" code.

You said you're the only developer; that means that I wouldn't expect you to have more than four branches open for most of the time. Master and develop make two, then the feature you're working on makes three, and then if you get interrupted by a bug you decide you should fix then you'll have a bugfix branch. If you get a report of a bug in master and decide to fix that before finishing any of the rest then it'll go up to five. If you get partway into implementing a feature and get stuck or decide to put in on hold for a bit, then that branch will hang around and add to your total. So, worst-case is six branches open at once, but only three of those are being actively worked on, so it's not too difficult to organise.

anaximander
  • 2,285
  • 3
  • 19
  • 21
  • Thanks for the thorough explanation. I will use your approach. By the way what is the difference between git merge and rebase? – Peter Warbo Mar 25 '13 at 21:54
  • 1
    The comment length limit caught my answer; check the edit to my post. – anaximander Mar 26 '13 at 09:21
  • Thanks once again for your answer. What about storing third party source in Git? I am using some thirdy party source in my app. And in some of this source I have even made my own modifications. – Peter Warbo Mar 26 '13 at 12:45
  • Provided your modifications are within the scope of what you're allowed to do with the third-party code in the first place, and you've included whatever notifications and license documents you're supposed to include, then using Git to keep that source safe should be fine. If you're uncertain then maybe read the license that came with the third-party stuff. If you're not allowed to distribute any of it, then as long as your repository is set to "private" and only developers on the project have access, you should be ok. – anaximander Mar 26 '13 at 13:23
  • Alright, I was more thinking of the workflow when I need to update my third party source code (should I perhaps use gitsubmodules for that?) and then I have updated the code push it to Bitbucket? Also when I created the git repository it creates a branch called "master", should I rename this to "version-1" or just leave it as it is? What do you think? – Peter Warbo Mar 26 '13 at 13:37
  • I tend to leave the master branch, and branch from it. My usual setup is something like the Git-Flow pattern, which you can look up if you fancy. It's a little more complex than what I outlined above, but it works nicely. At some point I should write a blog post or something; I find myself explaining it frequently. [This article](http://nvie.com/posts/a-successful-git-branching-model/) is one of the better explanations out there, but it could still be simpler and that big diagram up top makes it look much more complex than it needs to be. – anaximander Mar 26 '13 at 13:45
  • Alright, I think I'm starting to get the hang of it. You write that "when testers stop finding bugs forget about version-1..." do you mean that I should delete the "version-1" branch? Also I've read that you should merge your code to the "master" branch but you don't mention anything about that... sorry for all the questions :P – Peter Warbo Mar 27 '13 at 18:25
  • 1
    Yeah; I gave you a simplified version. What I meant by "forget it" is that all the fixes that went into it also went into your version-2 branch, so you don't need any extra merging. As for merging to master, the branch names are arbitrary; you don't *have* to do anything specific with them. It sounds like you're reading a Git-flow description - I'll post a run-down of it in a moment. – anaximander Mar 27 '13 at 21:33
  • Did you ever finish writing that blog post? :) – Peter Warbo Mar 28 '13 at 13:40
  • I'm curious, if you are working on a "big" feature in a branch then you decide it's time to release a new version but leave out this "big" feature from the release. You wrote that it's important to close any open feature branches when it's time to merge development into master. So what do you do with this "big" feature that is not ready then? – Peter Warbo Apr 18 '13 at 09:40
  • 1
    Ah, unclear explanation on my part - sorry. When I merge from `dev` to `master` and release it, I close any branches that have been merged into `dev`, but haven't themselves been closed yet. Sometimes I'll leave a feature branch open after merging it so that I can add enhancements or fixes to it later, to keep things tidy (remembering to rebase or merge out from `dev` first, of course). If I want to leave a feature out of a release, then yes, I'll hold off on merging that one into `dev`, leave it open, and merge it in when I want to include it. I'll edit that into my post, for clarity. – anaximander Apr 18 '13 at 14:08
  • Another question arose, when I have created a release branch, let's say (1.0.1) and I have given this release branch for my users to test. Then my users find a bug and I fix this bug in the release branch. You don't mention anything about giving my users the updated 1.0.1 (with the bug fixed) to test? Should I just be confident that when I have fixed the bug in the release branch I don't need to test this fix? – Peter Warbo Apr 24 '13 at 09:36
  • If your beta test phase is pretty short, then you don't really need to fix bugs in the beta version; as a test version it's kind of expected that there will be bugs. If your beta test is longer, then you can just merge the fix into the beta branch and back into `development`. Generally, a fix should be tested and verified as working *before* you merge it into `development`, and DEFINITELY verified as working before it goes anywhere near `master`. You rarely need to give a fix to your testers to verify; if you can't check it yourself then you need to ask for more detail in your bug reports. – anaximander Apr 24 '13 at 10:08
  • Don't think I follow you... you wrote "Only bugfixes are allowed in this branch. When it's done and you're ready to release, you merge it into master, and into development to make sure the fixes don't get overwritten. Then you can release the contents of master to your users." My understanding of this is that when I have sent out my `release-1.0.1` branch/build to my users to test whenever they find a bug I will fix this bug in the `release-1.0.1` branch right? But you are saying that when I have made this fix, it is not necessary for my users to test that the fix I made is working for them? – Peter Warbo Apr 24 '13 at 11:19
  • Ah, I think I see the confusion. You make the `release-1.0.1` branch *before* your users get it, and if your testers find a bug in `release-1.0.1` then you fix branch from there, fix it in the branch and merge the bugfix back into `release-1.0.1`. Your testers should always have the latest version of the branch they're testing. When you're happy with it, you merge `release-1.0.1` into `master` (and `development`) and *then* give what's in `master` to the users. Any bugs found after that are fixed in branches off `develop` and released in the next version, or a patch if they're serious enough. – anaximander Apr 24 '13 at 12:08
  • When you say "fix it in the branch and merge the bugfix back into `release-1.0.1`" are you saying that I should create a bugfix branch from `release-1.0.1` then? Up until now I have done all the bugfixes directly inside the `release-1.0.1` branch. Anyhow, when all bugs have been fixed in `release-1.0.1` do I make a new build of it and give it to my testers to do the final testing? – Peter Warbo Apr 24 '13 at 13:39
  • Yes - bugs should be branched, fixed in the branch, and merged. In general, you don't make a change in code that you've given to anyone outside the development team; instead, you do it in a branch, merge it in, and give them the new version. That way, you know that at any given point you have a record of exactly what they have. If a bug report comes in while you're working on a previous one, you can commit or stash what you're doing, switch back to the branch that reflects what they have, and run it to see the bug for yourself. After all, that was the original point of all this... – anaximander Apr 24 '13 at 14:01
  • Hmm okay, so let's say I create a bugfix branch from `release-1.0.1` let's call it `bug-username` and I fix the bug inside the branch, then I merge `bug-username` into `release-1.0.1`? and then I make a new build from `release-1.0.1` to give to my testers? By versioning standards, wouldn't this make `release-1.0.1` be `release-1.0.1.1` since it has had a bugfix? ( I suppose these are minor concerns but It's interesting to know ) – Peter Warbo Apr 24 '13 at 14:08
  • It depends on your approach to version naming. It's my understanding that branch `release-1.0.1` isn't called that because it *is* v1.0.1; it's called that because it *will be* v1.0.1 *when you release it*. Based on that, you can get away without changing the version number. What I do is say that what's sent to the testers is "v1.0.1_beta1", and then after `bug-username` is merged in, that would be "v1.0.1_beta2", etc. When you're done with the beta test, you drop "_betaX" from the end and release it as v1.0.1. – anaximander Apr 24 '13 at 15:24
  • Alright, so if I'm starting to understand this approach. For every bug that is detected in the release I need to create a new branch for it and fix the bug and send my users the build with the fixed bug. When it's confirmed to be fixed by my users I then merge the bug branch into my release branch? Finally when no more bugs are detected I can merge my release branch into `master` and `development`? – Peter Warbo Apr 24 '13 at 15:39
  • Yes. Of course, then you'll need to rebase any feature branches you may have left open to wait for a future release. – anaximander Apr 24 '13 at 15:43
  • Still curious about this "rebasing". Not really sure when it's necessary to "rebase". If I have understood correctly it is necessary to rebase if I have started working on a feature which I don't have time to finish for a release. When the feature is finished and I decide to include it in a release, I **then** have to rebase? Do I choose the "git-flow" option in SourceTree and choose "finish feature" and then "rebase on development branch"? There is also another option in SourceTree, if I right click on my feature branch there is an option saying "Rebase current changes onto feature-name" – Peter Warbo May 08 '13 at 21:32
  • I've never used SourceTree; I do it manually with git bash. Generally, a rebase is used to bring changes from a parent branch into the child branch; you can avoid using rebase by merging (and some prefer to do so). – anaximander May 09 '13 at 07:55
  • What options do you have if you want to "undo" a feature branch? Let's say you add a new feature `supercool-feature` which you merge into development and then it goes into a release. But then your users really dislike this `supercool-feature`. How can I undo this feature which has been already merged into development and a release? – Peter Warbo May 24 '13 at 07:43