Analytics in design

Proposal

Our proposal is to integrate evaluation into the design process. We already do conduct user-testing before a release. The question is, how to expand our design research to understand not only whether users can perform immediate tasks but also how our designs are working post-release.

To conduct the evaluation, we can use a combination of methods ranging from user testing and surveys to Google Analytics. We may find we need to develop custom analytic tools to answer our evaluation questions.

Strategy

To get stared, it is crucial that we are explicit about the goals for each project. Whether for a new element of a site or a new site, we need to articulate the intention of the design. We document some specifications in project proposals, so this effort will only augment and expand what we already do in early stages of project development. When the goals are stated, appropriate way to benchmark and analyze the success of the project naturally follow.

For a new site, assess our and the clients’ goals:

• What is the purpose of the site?
• What would we like users to get from the site?
• What would we like users do?
• Who is this site for?

For a new function:

• What is this upgrade meant to achieve?
• What behavior are we trying to affect?

For a more detailed approach to this method you can download surveys and the worksheets by gotomedia.
Although this sounds basic, it is easy to assume that clients and designers agree on the goals of the project without explicitly discussing our assumptions.

Once we know the goal of the project, we can agree on ways to operationalize our evaluation.

User testing is a part of this process - we do this early stage evaluation before we finish the design and launch it.

Analytics is a continuation of this evaluation. Using a combination of Google Analytics and internally generated tools we can record the appropriate measures. I outline a couple of concrete examples to get us started.

Specific proposals

The first step, I propose, is to measure the impact of new functionality we are currently designing on the behaviors we are trying to effect. There have already been some examples of this. Emina, for example, looked at how many people posted comments before and after the design decision was made to require people to add titles to a comment. I just looked at early results of changing the registration on PUMA creative sites on registration rates.

Projects to be analyzed

1. Notifications. Notifications have the possibility of impacting how often people use the site. To understand if our notification upgrades work, we need a baseline measure of the frequency of visits.
2. Edit pages. Edit page upgrades are intended to make it easier to finish adding things to Anymeta. We may want to look at average level of completeness before implementing this. This would require work on a completeness score.
3. Login + registration upgrades. Are we getting more people to register now that the registration is easier?

General analytics

1. Measure conversion rates on our sites. Our sites work well to turn up on Google search results. We have a lot of organic traffic. Is this traffic functioning to get new users? How many people generally register? We cannot currently track conversion rates because our login and registration happen in dialog boxes. Google analytics tracks paths to particular urls.

Next Steps

Build simple tools to upgrade how we track behaviors of interest

• Add some code to track important behaviors. People now register and login through dialog boxes. To track logins and registration through Google Analytics, we need to add code to some templates.
• With this code, we can set up goals and funnels in Google Analytics

Open questions:

• Are clients interested in who is using their sites? Is there any value in that information?
• Do clients have specific target goals for usage?

Some References:
Introduction on Design Research by Mike Kuniavsky
Nice synthesis of methods by: Bas Leurs, Peter Conradie, Joel Laumans, Rosalieke Verboom
Ezri's thesis