Perform a Card Sorting Exercise

Overview

Card sorting is an exercise to help build and/or test a proposed information architecture and/or navigation scheme.

As a part of the exercise, volunteers organize digital or physical topical cards into categories, according to their impression. Their decisions are then recorded, analyzed, and used to make decisions on site structure, storage mechanism, URLs and navigation (amongst many other future decisions).

Goals

Card sorting will help you understand expectations and clarity of topical content across your site, helping you:

• Build the underlying structure of your site
• Build or test plans for the top navigation of your site
• Develop a deeper understanding on the behavior of your user base

Step 1: Choose an Open, Closed, or Hybrid Sort

Open Card Sort

In an Open Card sort, volunteers are given cards and asked to group them into categories which they define. This approach gives the user more influence and can provide more insight. It also requires more time and attention in both the session(s) and analysis because in addition to trends in cards vs. category, trends in category titles must be analyzed, consolidated, interoperated, and, in some cases, eliminated.

Closed Card Sort

In a Closed Card sort, volunteers are given cards and asked to group them into pre-defined categories. This approach allows the agency to test a proposed information architecture.

Hybrid Card Sort

In a Hybrid Card sort, volunteers are given both pre-defined categories and the ability to create their own if they deem it necessary. It is suggested that this is only attempted when performing a digital card sort as it creates an additional layer of complexity when collecting and analyzing results.

You may consider performing two rounds of sessions, with separate participants in each; an Open sort first and a Closed sort second. The first round allows you to learn what goes together. At that point, you can build Categories and test using a Closed Sort.

Step 2: Choose a Card Sorting Technique

Step 3: Select Tasks to be Used in the Exercise

From your list of Top Tasks, select the top 50 tasks, ranked by level of importance. Be sure to skip tasks which do not require card sorting, such as those which will end up categorized under the About Us section of your website (e.g. Address/Location, Mission Statement, Contact Form, Policies). Once you have 50 tasks, ensure that each task is a simple phrase or idea, which uses concise, plain language. Consider how your customers describe a task, instead of how your agency describes the task. More consideration and guidance on this topic is available in the Identifying Top Tasks document.

Step 4: Prepare the Cards

While it is tempting to want the participant to sort all of your content, you must always be mindful of participant fatigue. Therefore, it is recommended that a maximum of 20 to 30 cards be assigned to each volunteer at the most, especially for an Open Sort.

For the digital sessions, follow the software’s directions on creating category cards, creating all 50 cards. Enable randomization and a limiter, ensuring only 20 to 30 cards are presented to each volunteer. Do note that all cards will still have equal representation in the study, even though a reduced number is shown to each volunteer.

For in-person sessions, write each task on a separate card, building decks of 20-30 cards. The easiest way to ensure each task has a matching amount of representation is to go straight down the list of tasks, in order, starting a new deck at the with the same count, resuming in-list.

In addition, for physical decks, it is advisable to use a word processor to maintain consistency and legibility. Consider printing each physical deck of cards as a different color to prevent mixing ups. One great feature of Optimal Sort is its ability to let you quickly create digital cards and then download and print them for testing in person. On the back of each card, write a unique number to aid in referencing each category.

Step 5: Prepare the Categories

If you plan to perform a physical Open sort, create a deck of six blank cards for each deck of tasks.

If you plan to perform a Closed sort, you must build a set of categories. Either sort the cards yourself, using as many members of your team as possible, or perform an Open sort amongst your agency. Using these results, plus subject matter expert knowledge, agency’s mission statement/goals, examples of other states’ navigation, create category cards which are your best attempt at identifying top-level navigation of no more than six categories.

Just as with the card decks, when creating the category cards, write each category on a separate card. It is advisable to use a word processor to maintain consistency and legibility. On the back of each card, write a unique letter to aid in referencing each category.

Step 6: Prepare for Data Collection

In-Person Sessions

In Excel, create a standardization grid spreadsheet, listing the cards in the rows and the categories in the columns. From your card sort results, you will determine how participants grouped the cards into the categories, then add the raw counts to your spreadsheet, as shown below in Figure 1.

Remote / Digital Sessions

In the majority of software, data collection is done automatically; however, you may need to indicate what types of result sets you are interested in seeing.

Step 7: Set up and Lead the Session

In-Person Sessions

Before the session begins, make sure each volunteer has a deck of cards, category cards (be it pre-defined or blank), and enough room at their desk to spread their cards on the table, or tape them up on a wall.

Introduce the session by reading a prepared script that describes the goals of your test and explains what you’re trying to achieve.  Show them the deck of cards, explaining that you are asking for help to find what categories of information should be on the site’s homepage what those categories should be called. Provide the participants with an estimate of how long the card sort will take before beginning the session to help them better gauge the required time and effort. It takes approximately 20 minutes for 30 items, 30 minutes for 50 items, and 60 minutes for 100 items.

In an open card sort, explain that you want to see what groupings of cards make sense to the participant, and that you will ask for a name for each group of cards once the participant has grouped them. In a closed card sort, explain that you want to see how the participant thinks the cards fit within the defined groups. Emphasize that there are no wrong answers.

In one-on-one sessions explain that you will be just taking notes, and encourage them to think out loud so you can learn from their thought processes about the tasks and categories. Once they begin sorting cards, minimize your interruptions and take notes. Let them work through it at their own pace and write down all your observations.

For open card sorts, if, in the end, the participant has too many groups for the homepage, ask if some of the groups could be combined. Allow the participant to put cards aside to indicate topics the participant would not want on the site. If they ask for help, suggest that they create a new group name or say “I’m not sure, what do you think?”

Always thank your participants for their time and ask if they would like to be informed when the new website goes live. You can either photograph the sort or use the numbers on the cards to quickly record what the participant did. Either photograph or write down the names of the participant for each grouping and the numbers of the cards the participant included under each lettered category, then reshuffle the cards for the next session.

Remote / Digital Sessions

Distribute card-sorting links to participants. This can be done in several ways, including:

• Email targeted participants; this may come from an existing distribution list.
• Post a link to the test on your social media pages.  When you post it, ask people to share the link to their networks.
• Post a link to the test on your website.

Set a desired end-date for the exercise and check on the progress of the study every day. If you’re not getting the results you need, broaden the distribution of the test. Once you start seeing trends in your results, you can identify areas for improvement or conclude the test. For example, if after 20 people take the test, there are no patterns forming, you may need to consider reworking the categories you’ve pre-selected.  If you are conducting an open test, you may need to reduce the number of tasks, or change your requirement to sort all cards. 

Step 8: Analyze your Results

Open, In-Person Sessions

The first step is to standardize categories created by volunteers, consolidating similar categories a consistent name. This will make analysis easier as you can combine groups where participants used the same basic concept but a slightly different label. This is a tricky step and requires a lot of judgment.

Start by identifying with the obvious candidates for standardization; there will usually be a group of categories used by every participant. With the remainder, standardize (consolidate) the category names if you are confident the concept is the same. Only create a standard category if you are sure two labels mean the same thing.

If you find where a user had created a combination category name (e.g. “Data and Graphs”), use the combined group, choose the most likely category, or consider amending the sort to break it into two group. Only do the latter if it is clear which cards belong to which group. Also, be careful not to standardize different categories just because participants have combined similar cards under the same heading.

Once you have created standardized columns, follow the directions stated above for Closed, In-Person Sessions.

Closed, In-Person Sessions

Once the session is over, review the photographs or other data recorded for each session and record the data in a standardization grid, modeled after Figure 1.

Analyze quantitative information based on which cards appeared together most often and how often cards appeared in specific categories.

• Calculate the Percentages

Replace the raw counts with percentages. Each percentage is the number of participants who sorted certain cards into a particular group divided by the total number of participants. Once you’ve calculated the percentages, keep only the percentages that are substantial enough to take into serious consideration. To quickly determine the meaningful results and gain insights from our recent card sort, use a 10-percent threshold, deleting all lower percentages from the spreadsheet. This threshold can vary based on the scale of your card sort and the goals of your study.

• Group the Cards

Starting with Category A, reorder the cards according to which cards have the highest percentages among all of the groups. Once you have determined which cards have their highest percentages under each group, rearrange the cards so all of the cards with the highest percentages are together in the matrix under each group.

Analyze qualitative information based on user comments.

Once you have analyzed the distribution of percentages across the groups. For each group, go through each card under the group and compare their percentages, then consider why participants made their judgments.

Remote / Digital Sessions

In typical card sorting software, the following examples of results are common:

Open & Hybrid Card Sorts

• Dendrograms

  Presents data clusters that represent participant agreement about the card categories.

• Similarity Matrix

  Presents a table that shows the percentage of participants who agree with each card pairing.

• Participant-Centric Analysis

  Shows the top three most acceptable submissions by participants, as tested against all the others.

• Standardization Grid

  Presents a table that shows the distribution of cards across the categories.

Closed Card Sorts

• Results Matrix

  Shows the number of times each card was sorted into each category.

• Popular Placements Matrix

  Shows the percentage of participants who sorted each card into each category. It attempts to propose the most popular groups based on each individual card's highest placement score.

  Consider consulting articles on the software vendor’s support website for detailed explanation of each result set.

Step 7: Make Your Decisions

After you analyze the data from card sorting, you should have useful information for structuring the information architecture of the site. You should use the results of your card sort to help you define the navigation of your site.