Bynomial Code

This post gives a quick introduction to beta testing for iOS developers.  Specifically, I’ll walk through the major steps needed to legitimately send out a working copy of your app to testers without going through the app store – a process known as ad hoc distribution.

Why beta test?

All programmers become attached to their code over time.  It’s just being human.  You think to yourself, “why do I need to spend time sharing this with people who don’t know anything about my app?”  You can just test it yourself, faster and cheaper!

Right?

The problem is, it’s really hard for you to treat your app like you haven’t built it yourself.  There’s a tendency to only test what you just wrote, and to play with things that you know will work.  I can’t tell you how many times I’ve done this – gotten so used to using the app the way I know it works, that I forget about all the ways it can break.

So there is a huge value to a fresh pair of eyes.  You may like to pretend that all customers will perceive the app exactly the way you do, but that isn’t reality.  What a new user thinks and does with your app is not obvious, and this is why we test.

Getting testers’ UDIDs

A UDID is a unique identifier string per device (iPhone, iPad, or iPod touch).  Your Apple developer program account will usually give you 100 device slots that you can register.  Basically, by requiring you to register devices, Apple can avoid people selling their apps directly through this system.  You can recycle your slots, but only very slowly, as there is a 1-year waiting period to recycle any given slot (see the last two paragraphs of this page).

A user can find their UDID a couple ways:

1. They can search for “udid” in the app store, and download one of those apps.  These generally give the user the option to email their UDID directly to you.  Although I think option 2 is faster, some users might feel like this is easier, since there seems to be some psychological “work” associated with plugging in an iPad/iPhone.
2. They can plug in their iPads, open iTunes, and click on “Joe’s iPad” (whatever the name is) under Devices in the left-hand panel.  Next click on the serial number of your device, and it switches to display the UDID.

Here’s a link to instructions on finding a UDID – useful for emailing to new testers.

Building an ipa file

This is the most fun part.

An ipa file is a file testers can install in two steps:

1. Double-click on the ipa file, which loads it into iTunes for them; and then
2. Sync their device with iTunes.

In some cases, they might have to check the box next to your app’s name under the Apps tab amongst the list of their sync settings for their device.

Generating that ipa file, however, is a bit of work.  Here’s an overview of what you need to do.

These steps are for the Provisioning Portal, available to all Apple developer program members through developer.apple.com.  I’m assuming you’ve already done some usually-previously-required steps like creating your certificates and an app ID.  If not, see the links below.

1. In the Devices section, add the tester’s device using the UDID they sent you.
2. In the Provisioning section, click on the Distribution tab, and either add a new profile (if this is your first tester), or click Edit next to the existing profile for your app.  This will be an ad hoc distribution profile, and the other options are straightforward – always select all the devices you want to be able to run the app on.
3. Download the provisioning profile from the previous step, also from within the Provisioning section.
4. In Xcode, create a build configuration called “Ad Hoc Distribution” (the name does not affect the build, so you can change this string to suit your needs).  Modify the build settings to sign the binary using the new profile (iPhone Distribution, under Code Signing Identity).
5. In the build configuration drop-down (always visible in the top-left of the main Xcode window), choose Device, and then Ad Hoc Distribution.
6. From the Build menu, choose Build and Archive.
7. From Xcode’s Organizer window (ctrl-cmd-O), click on your app’s name under Archived Applications on the left.
8. Choose the latest archived version (if this is your first time through, there is no choice here), click on Share…, and then Save to Disk…
9. Save the file where-ever you like, and email it to your testers.  Done!

I’ve actually been relatively terse in the above list.  There are two much more detailed lists you can walk through if this is all new and scary.  The first is from Apple directly.  Once you’re logged in to developer.apple.com/ios, go to the Provisioning Portal, click on Distribution on the left, and read through each of the items starting with “Obtaining your iOS Distribution Certificate” – each of those items is a link that expands when clicked (link).  Those steps are official and comprehensive.

There’s also a nice stackoverflow answer with a detailed walkthrough posted by Arne Evertsson.

Tips for getting feedback

Actually getting useful feedback from testers is harder than you may think.  Expect a significant portion of testers to not follow through.  Those who do send feedback may easily send non-useful tips without careful forethought.  A few things can go wrong:

• Testers won’t know what to try to do.  Often, they might not be exactly your target audience, so you have to give them goals, such as “try to search for books on math.”  Even if they wouldn’t normally do this, it can be informative to see what they try, and how they feel about the results.
• You give over-detailed instructions, and nothing new is discovered.  If testers have step-by-step instructions, they’ll follow them exactly, and the results will always be the same.  The point of testing is to put testers in the position of real users, who have taken the time to download your app, but otherwise know nothing about it.
• Testers have too many choices, and end up not trying the primary use case.  If your test app is too open-ended and abstract, it may be hard for others to even know what your app is really good at.  I suggest you avoid treating testers as percolators of vision, and consider them more as an extremely low-bandwidth window into the minds of end users.  Consider yourself as getting a few bits back at a time.

There is a whole field of research on testing software, so this barely scratches the surface.  Another good post is Joel on beta testing.

Testing in person

Having explained so much about remote testing, I’d like to say that in-person testing is still the easiest and most in-depth way to go.  When you are interacting directly with someone trying your app, you can see first-hand how bored or excited they are, and understand more easily which buttons they’re pushing and how they build their internal model of your interface.

Test, tweak, repeat, – and converge your design to greatness!