Betas and testing

An interesting post by Steven Frank of Panic Software has recently started a discussion about difficulties of doing efficient software testing for Mac shareware programs. I found many similarities with our own testing workflow experience as we’re also “old school”, e.g. we’re doing our beta testing in a closed private group and almost never release something called “beta” to the public. Steven writes:

Having a couple hundred beta testers sign up, and only getting any feedback at all from maybe 10% of them is still a bit depressing and sometimes worrisome. It’s hard to know whether to interpret it as “people aren’t using the beta” or “there’s nothing wrong with the beta” or “I just signed up in the hope that I’d get a free copy” or “I had some weird stuff happen but didn’t think it was worth writing about”.

This is true to some extent concerning our own beta testing experience. However I discovered that a very important factor in receiving more feedback is to foster a sense of community among your beta testers. There are tons of posts written on “how to get more comments on your blog” or “how to make a successful forum”, and I think that this kind of tips apply to beta testing as well, since you basically get a small community of people united for the same task. So it all comes to the way you’re moderating this community. While Joel on Software already has some excellent tips on the topic, let me mention a few strategies that worked for us to improve beta testing feedback:

• Mailing list - my experience on our support forum shows that people like to communicate with each other about software issues. When one person reports something, the other one will add some details, etc. On our beta list we sometimes get very heated discussions about usability issues or feedback about UI changes, which are indeed very valuable and can’t happen if people don’t interact with each other. I think a mailing list is essential in building a passionate beta tester community.

• Listen and respond - be as present as possible. I try to respond to each filled bug and tell what happens next with this report, so that people know that we’re actively working on it. If not beta testers don’t know wether they should interpret the silence as “does it mean that the devs agree” , “does it mean they already knew about it”, etc. We don’t open our bug tracker to the testers, so our responses are the only mean for them to know about our progress.

• Fix what they tell ya - sounds obvious, but if people reported tons of stuff and the next build doesn’t contain those fixes, don’t be surprised by the silence. If for some reason it was impossible to fix, consider including it in the “known issues” seed note section.

• Regular builds - I think beta builds are still mini-releases, so same rules apply on a smaller scale - reasonably frequent builds show that we’re working on it and thus generate interest. Too much seeds make people tired and they stop downloading them. A way to know if seeds schedule is correct is to have some kind of a statistic following the number of beta downloads. This way you will be able to know if everyone downloaded a particular build or if you make builds too fast, and people started to skip them, etc. I think that a build per week is a reasonable timeframe, but your milage may vary.

• Comprehensive seed note - I think it should at least include following sections: a detailed changelog mentioning which bug of which tester got fixed in this build, a “known issues” section mentioning reported bugs that didn’t get fixed, focus areas - containing things that you think might be broken and you would like your testers to try out.

• Ask questions - People like being asked about their opinion. Also very few beta testers will write to the list just to say that this or that part of the software works. Asking them to confirm something will often “wake up” those who generally don’t write much. Asking a question is also a good way to initiate a discussion, which can uncover usability issues and opinions that people always hesitated to tell you about.

And if everything else fails, make a call for new people and refresh the list.

Of course the intensity of the feedback greatly depends on the way people use the software. Some programs are made to be used every day - and we fall in this category. Of course a program that is supposed to be used not often and only for specific tasks will probably have more trouble to get tested throughly in a short period of time.

I use the opportunity to thank all our beta testers for their continuos support. Our last beta cycles were very intense and beta feedback helped us a lot to have a stable and smooth release.

Did you participate in any beta testing? What makes you want to write feedback to the developer? Tell us about your experience.