8

I would like to know what strategies do you use to deal with A/B tests of your app and gitflow.

Overview:

We are a team of 6 programmers who develop and maintain a large App. So far we have worked on gitflow with a few add-ons on the process added by us which worked perfectly well for a couple years. In a simplified way, we use:

  • master branch (only code of published versions)
  • release branch which merges in master after final redundancy tests
  • hotfix which only interacts with master branch in extreme cases
  • develop which accumulates the developed modules as they are finished and tested, eventually merging into release.
  • /feature which is a group of features that branch from develop and once they are finished and pass the different stages of testing merge back into develop adding functionality
  • /fix_develop which is a group of features that contain fixes to bugs encountered from earlier versions that are not too urgent to start a hotfix.

Now, as the app evolves, with the UX team and other stakeholder teams we are adopting a stronger A/B testing strategy where new releases will have 2 versions, and based on how our users like one version or the other will become the final master version for as long as they make sense for our users.

Having explained that, the question is: What strategies have you used or recommend to manage code of A/B testing versions in gitflow?

The options I've considered are somehow inconsistent, for example branching A and B branches from master and then joining the release branch to one or the other, where I don't know how to deal with separating the code contained from release branch to feature branches. Another option is creating release A,and B branches and develop A and B branches which sounds like too much branches and confusion for my team mates.

I hear your opinions, thanks!

Update: The App we develop is an Android App and we are implementing the A/B testing using PlayStore platform for A/B testing, which requires to create two APKs and upload one of them with a rollout %. Also to keep things simpler, and since changes may sometimes be greater than just a button position, we decided not to add our own switch for A and B testing in one single APK.

alexm
  • 188
  • 6
  • Aren't these simply feature branches? – Robert Harvey Nov 30 '17 at 16:07
  • 1
    thing is that these branches will go to the store, so would have to break all the testing protocols if I just upload code from feature branches. what do you think? – alexm Nov 30 '17 at 16:09
  • You're going to go through the entire vetting process for a branch that may never become an actual product? Doesn't that essentially double your work? – Robert Harvey Nov 30 '17 at 16:10
  • anyways this is just the way I see it now, my idea with the post is to learn how other people deal with this matter and apply what I think would make sense for us – alexm Nov 30 '17 at 16:11
  • it does double the work, but there are essential things we need to test and prove if they work for our users that are quite complex, this is all done to enlarge the bottom of our active user's fanel, so it is a lot of work, but makes sense to test – alexm Nov 30 '17 at 16:26

3 Answers3

11

The way I've always seen it done is to have a single code base capable of serving both pages/views/forms.

ie. Its Feature flagged and deployed with two or more configs, or the 'does the user get A or B' method itself is part of the app.

In this case you only have the one version of the code, so your source control doesn't come into play.

This is far more preferable than the alternative of keeping multiple versions of the codebase with different features. These tend to diverge very quickly and you end up having to implement the same features multiple times.

In general I would be careful and restrict the scope of differences A and B can have both so that they can be plugged into a generic switch method (view A or view B for example) and so that you can understand the reasons for any different stats you get out of your testing. eg was the increase in sales linked to the button colour or the pricing formulae?

Edit. for App clarification

You can still have features flags in a play store app and package the codebase into more than one apk.

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • 3
    Yes, this is what I was going to suggest: feature flagging. The only problem with it is it increases the complexity of your software, unless you intend to remove the (unused) feature(s) in the follow-on release. – Robert Harvey Nov 30 '17 at 16:12
  • I understand what you mean and it makes sence, the thing is that we are using the PlayStore to accomplish the A/B deliver to the users and make use of their analytics which cover every install, on top of our own tracking. So I need to upload 2 APKs and rollout one of them on 50%. Will update the question with this – alexm Nov 30 '17 at 16:15
  • 1
    yes, I'm not in favour of feature flags usually, but this is the way I've done it and seen it done. I'm sure we've all seen the horrors you can get into having multiple live versions of the same product in branches. So I would naturally shy away from that approach myself. – Ewan Nov 30 '17 at 16:16
  • 2
    you can still have config settings and get two apks from the same code base. Sure your app will be bigger, but.... maintaining multiple codebases is challenging to say the least – Ewan Nov 30 '17 at 16:22
  • Ewan you are right having multiple versions in store may be a horror, we have actually made A/B testing with switch for small changes like buttons and stuff before, but this time it requires a lot of changes in structure, so basically, what we want is to keep old version and the new in store, then decide for one of them and discontinue the other, while keeping the versions clean. – alexm Nov 30 '17 at 16:23
  • anyways will consider your point of view, so I'm upvoting you – alexm Nov 30 '17 at 16:24
  • Yeah i see where you are coming from. I wonder though if that is really A/B testing anymore? If you have a whole new app I'd be surprised if people didn't find excuses for why you should go live with it even though ppl prefer the old one. – Ewan Nov 30 '17 at 16:31
  • not actually a whole new app, but for instance we change the way we ask for registration, what we show first. We've made an analysis of where do people leave the app in the intro part, so now we have two options, 1) make it quicker show how it works fast and don't ask too much info, 2) go straight to the main part and teach them on the way what they can do. The app is the same app, but there are structure and flow changes. – alexm Nov 30 '17 at 16:40
  • 2
    surely that kind of flow can be abstracted into a view – Ewan Nov 30 '17 at 17:28
  • I like that last comment – alexm Nov 30 '17 at 17:40
  • 1
    Yes, for an enterprise development team, I would suggest so. For me, in an intensively committing enterprise development environment, trunk base development flow is much easier. Check out here: https://trunkbaseddevelopment.com/. Sam Newman had a nice talk about feature toggle in youtube: https://www.youtube.com/watch?v=lqRQYEHAtpk. – ivenxu Nov 30 '17 at 22:27
  • @ivenxu I think you just gave the answer to my problem. Investigating about trunk and it seems to be the best option so far. Thanks! – alexm Dec 05 '17 at 17:52
1

I agree with @Ewan that feature flags are the way to go. That way you can add something in the build process that will deploy APK for either A test or B test. But if want an idea that does not use feature flags, here is one that I have not tried.

You could pull the common code out into a separate library in a separate repository. Then, each version of the test has its own repository with its own gitflow and master branch that can be deployed.

This will take more effort in the beginning in that all the common code will have to be pulled out into library(s) then referenced in the repository. But after the initial setup, I believe this can streamline the development of new features into the A/B test.

**Again, this is only an idea and not something that I have done personally.

DFord
  • 1,240
  • 2
  • 15
  • 30
  • Yes that makes sence, we have a library with the main functionality (C++) and then consumer Apps (Android and iOS) that only serve as interface and service deployers, what we are changing is just the way the app is presented to the user, so yes, that would work for A/B tests in Android. However, I'm looking to have the code centralized in one repository, so I'm looking for a gitflow strategy in the middle of: "copying all branches for A and B" and "having just one A and B feature or normal branch which would mean a dirty repo". – alexm Nov 30 '17 at 17:38
0

Since your app is already running and has some users, I suggest one of the following options:

  • I strongly suggest that you do a design thinking before implementing a new big feature;
  • If you still need to do the A/B testing, I suggest you keep with normal gitflow and have an A branch and a B branch;

Source control

Considering A/B testing, what I want to make it clear is that:

  • A branch: contains code regarding A version of the app. EG: A_HomeActivity, A_SettingsActivity, etc.
  • B branch: contains code regarding B version of the app. EG: B_HomeActivity, B_SettingsActivity, etc.

From this point, you could use 2 strategies:

  • Create two totally different versions: A.apk and B.apk; or
  • Make the B branch to also contain code from A branch, and provide an app capable of using the two flows. The user downloads only one app and has the option to turn the new flow on for evaluation purposes. I've seen something like this done by Bitbucket, on which a totally new navigation was available for few users for evaluation; after some period this new navigation became the default one.

Either way, after the experiment you simply remove code for the obsolete feature/flow.

Emerson Cardoso
  • 2,050
  • 7
  • 14