Letterboxd Usability Testing
Letterboxd is a social networking service for sharing and discussing films. Users can create profiles and use them to track and rate films they've watched, as well as write and read reviews. Letterboxd also allows users to follow other users and engage in discussions with them. Additionally, the platform provides information on upcoming releases, popular films, and personalized recommendations based on user preferences. Letterboxd has both a free and paid version, with the latter offering additional features such as custom lists and advanced statistics.
University of Texas at Austin, Masters of Science in Information Studies (MSIS)
Course: INF 385P - Usability, Spring 2023
Professor: Natalie Dare
Team: Alex Locke, Karishma Makhija, Shatayu Mondhe, Yichen (Sara) Wang, and Yiwei Wu
If you prefer a condensed version of this report, you can access our final presentation through the link below.
Introduction
Our usability class in the spring 2023 semester was tasked with conducting a semester-long usability evaluation project on a company of our choice, which included a Heuristic Evaluation, Competitive Analysis, and Moderated User Testing. The objective of the project was to gain an in-depth understanding of primary UX methods relevant to product development and learn the principles of other important UX tools/methods, including their scientific underpinnings, advantages, and disadvantages.
Focused on the desktop version of Letterboxd and performed a Heuristic Evaluation, Competitive Analysis, and Moderated User Testing.
Our usability project aimed to identify and address usability issues on the Letterboxd website for both novice and expert users.
The final deliverable was a comprehensive report that included evaluations, data analysis and synthesis, along with key insights and actionable recommendations to enhance the overall user experience.
High-level overview of our process
Heuristic Evaluation
Our group conducted a heuristic evaluation on Letterboxd. This evaluation is a usability inspection method that involves reviewing the interface or product based on a set of heuristics or guidelines for good UX design principles. We chose to use Jakob Nielsen's 10 Usability Heuristics for User Interface Design and focused on four of them:
Match Between System and the Real World
User Control and Freedom
Consistency and Standards
We evaluated each heuristic using a severity scale to determine if changes were needed urgently or not. Individually, we took screenshots and in FigJam synthesized our evaluations using all 10 heuristics, and then as a group, we narrowed down to the four that brought the most value to the evaluation.
Jakob Nielsen's 10 Usability Heuristics
Severity Rating Key
Check out the link below for a more in-depth breakdown of our heuristic evaluation.
Competitive Analysis
A competitive analysis is a process of evaluating a business's competitors, including their products or services, pricing, marketing strategies, distribution channels, and customer base. The main goal is to gain insights into the market and competition, which can help a business develop effective marketing and business strategies for achieving a competitive advantage. This is an ongoing process that requires businesses to stay up-to-date with changes in the market and their competitors' strategies.
Our Approach
Letterboxd Community Survey
Our approach to the competitive analysis of Letterboxd involved first discovering the most used features of the platform by creating and sending out a survey to the Letterboxd community, which garnered over 130 responses. We used these features to create a matrix that we then used to evaluate the desktop sites of each direct and indirect competitor.
SWOT Analysis
Additionally, we conducted a SWOT analysis for each competitor, analyzing their Strengths, Weaknesses, Opportunities, and Threats.
Synthesized and Final Recommendations
Finally, we synthesized our findings and provided recommendations for how Letterboxd could continue to gain a competitive advantage.
Direct Competitor Matrix
Indirect Competitor Matrix
Check out the link below for a more in-depth breakdown of our competitive analysis.
Moderated User Testing
Objectives & Goals
Objectives
This study aims to test the usability of the Letterboxd website.
Despite its popularity as a social platform for sharing opinions on movies, Letterboxd has several usability issues that we aimed to uncover through this test.
The study focused on the experience of both novice and expert users, while using the desktop version of the Letterboxd website.
Goals
Our goal was to identify any users pain points while using the Letterboxd website and provide recommendations to address them, with the ultimate aim of enhancing the overall user experience.
Research Design
Recruiting
We recruited participants who matched our target demographics through a 16-question screener. We designed a Google form and distributed it across various Letterboxd Discord servers, the Letterboxd subreddit on Reddit, and LinkedIn. The form requested information on participants' gender, age, country of residence, industry, research study participation experience, frequency and recency of app use to gauge their expertise level, and other related apps and services used. We added a control answer to ensure authenticity of responses, and also asked for their laptop's operating system and internet access to ensure compatibility with Zoom calls. To test their competency and filter out inauthentic candidates, we included an open-ended question asking for their favorite movie.
Out of the 17 responses we received, we used an eligibility score to narrow down the pool to 12 participants and sent them an email congratulating them on their eligibility. The email contained a link to a Calendly app to schedule a time slot for the interview, asked them to fill out a consent form and a pre-test questionnaire, and confirmed their Zoom compatibility. We sent the pre-test questionnaire based on our experience from a pilot study that helped us identify the time gap. We ultimately interviewed 7 participants, and only one of them did not show up.
While we tried our best to get an even mix of gender participants, we unfortunately had a larger turnout of male participants than female participants. This is one area we wish we could have changed.
Participant Breakdown
Research Design
A pre-test questionnaire was shared with participants in the interview invite to be filled before the interview.
1 warm up question was asked to learn about the user’s first impressions of the website.
6 scenarios were presented to the participants to test the usability of the platform.
A post test questionnaire was used to gain qualitative insights which was then analyzed and created into an affinity diagram.
A SUS survey was presented to the participants and the results were visualized into a heat map.
The participants were requested to think out loud while carrying out the tasks.
Testing Environment
A pilot interview was conducted to identify gaps in our testing approach.
The interviews were conducted remotely via Zoom and the participant’s audio, video and screen share was recorded.
All tasks were carried out using a dummy account on the Letterboxd desktop site.
Each interview had 1 moderator and 1-2 observers.
Observers took notes in Google Sheets and tracked task success and number of attempts.
Research Design & Testing Environment
Screenshots from the Remote Moderated User Testing
Task List
We designed a series of questions for each participant consisting of a warm-up question and six task scenario questions. These questions were specifically selected based on our findings from the heuristic evaluation and competitive analysis, with the aim of assessing the main features utilized by Letterboxd users. We hypothesized that the search function may be an area of interest, particularly with regards to potential pain points experienced by users when searching for movies with specific keywords such as "the", "knight", or "night" or anything numerical in title. Additionally, we aimed to ask questions that would help us better understand the mental models of novice and expert users for certain tasks.
The task list comprised the following:
Task 0 – First Impressions
Before we begin, we would like to know your initial thoughts on the website. What are your first impressions?
Task 1 – Add to Watchlist
Please add the following movies to your watchlist:
The Dark Knight (2008)
M3GAN (2022)
Task 2 – Rate & Review
Please rate and review The Dark Knight (2008) and M3GAN (2022). You can make up a brief review.
Task 3 – Create a List
Create a new list and add the following movies to it:
The Dark Knight (2008)
M3GAN (2022)
One Hundred and One Nights (1995)
Task 4 – Locate List & Sort
Find the list titled "Official Top 250 Narrative Feature Films" and sort by newest released films.
Task 5 – Like a Review
Like the review of M3GAN (2022) from the user Kársten.
Task 6 – Settings Toggle
You want to receive email notifications when you get a new follower. How would you do this?
Task List
Data Analysis
Affinity Diagram
Following the interviews, our group took the time to listen to each recording again and write down our observations for each participant and task in a FigJam board. We then analyzed these observations to identify common themes, which we prioritized as a group to arrive at the top 1-3 insights for each task. We also assigned a severity rating to each observation to help us better understand its impact on the user experience. We repeated this process for the open-ended post-test questionnaire and uncovered additional insights that informed our final recommendations.
Affinity Diagram
Task by Task Analysis
To create our final report, we first captured screenshots for each task and annotated our observations, including the corresponding severity rating score and the related heuristic principle (using the same severity rating scale we previously used in our heuristic evaluation). Additionally, we recorded the task completion rates and included a quote from a participant that reinforced our findings. Finally, we provided a list of usability recommendations for each task, based on our observations and the participants' feedback, aimed at improving the user experience of the platform.
Task 0 – First Impressions
Task 1 – Watchlist
Task 2 – Rate & Review
Task 3 – Create a List
Task 4 – Locate List & Sort
Task 5 – Like a Review
Task 6 – Settings Toggle Icon
Post-test Questionnaire Analysis
As mentioned earlier, we listened to the interviews and recorded our observations, categorized them, and prioritized them as a group. Based on our analysis, we identified the following key insights from the post-test questionnaire:
Question: How would you describe your overall experience of using Letterboxd?
Insight: The website appears cluttered and overwhelming, particularly for novice users who may find it difficult to navigate.Question: What do you like the most about Letterboxd?
Insight: Users appreciate the social features of Letterboxd and find them engaging and enjoyable.Question: What did you find most difficult and why?
Insight: Navigation on the website is challenging, and users often struggle to find what they are looking for.Question: What (if anything) surprised you while completing your tasks?
Insight: There are inconsistencies in the user experience, with some features being straightforward and others unexpectedly complex.Question: What (if anything) are some of the changes you would like to see on the website? Why?
Insight: While providing multiple ways to perform a task can be helpful, there may be too many options available, which can confuse users.Question: If you have used any other similar platforms, how would you compare letterbox with the platforms you’ve used before?
Insight: Some users who are familiar with similar platforms find the social aspect of Letterboxd to be too forced and artificial.
Post-Test Questionnaire
SUS Analysis
The SUS (System Usability Scale) survey is a commonly used method for evaluating a system or product's perceived usability. It involves a 10-item questionnaire that utilizes a five-point Likert scale to assess a user's satisfaction with the system's usability. The SUS survey offers a standardized and quantitative approach to evaluate the overall usability of a system and compare the results across various studies and products.
At the end of the interview, participants were sent a link to the Google form containing the SUS survey and asked to complete it. After gathering all the scores, we used a template to calculate the results.
Our key findings from the SUS survey were as follows:
Letterboxd has a strong fan base, and existing members generally enjoy using the platform.
Experienced users are adept at navigating the platform and have developed "hacks" to facilitate their use.
Letterboxd may not be as user-friendly for novice users.
Letterboxd's SUS score of 65 was slightly lower than the industry average of 68.
Expert users gave Letterboxd an average SUS score of 83.
Novice users gave Letterboxd an average SUS score of 42.
SUS Analysis
Key Insights and Final Recommendations
In conclusion, our team gained several key insights from the user testing, which led us to generate a set of recommendations for Letterboxd to improve their website and address user pain points. These recommendations are aimed at enhancing the overall user experience.
Key Insights
Expert users prefer mobile app over the website.
Novice users find Letterboxd challenging due to poor navigation, cluttered content, unclear UX writing, and too many options available.
Users appreciate the social features of Letterboxd, including the ability to find reviews from like-minded people and interact with them.
Finding new movies and keeping track of movies you've watched are important features for users.
Users find ads, poor navigation, list editing, inconsistent CTAs, poor layout, and weak search function to be issues with the website. Especially search.
Managing preferences in settings is straightforward.
Users struggle with navigation and finding what they need.
Some users find the social aspect of Letterboxd to be too forced and artificial.
Letterboxd has a loyal fan base.
Expert users can navigate Letterboxd easily by "hacking" their way around it.
Novice users have a lower average SUS score compared to expert users. Indicating that Letterboxd isn’t well catered to the novice user.
Users have mixed expectations about the cost of using Letterboxd. Some expected the cost to be higher, while some assumed with the current price offering. The current pricing is fair.
Final Recommendations
Improve the search function to recognize spelling variations and common abbreviations, using fuzzy search or auto-correct options if necessary. Prioritize search results based on relevance and importance.
Add a search function within the list feature and a search bar to the list page to allow users to easily search for specific items within their lists. Consider adding more prominence to the Top 250 list on the list page.
Add a search function within a user's profile to allow other users to search for reviews by that user more easily. Consider adding filters or sorting options to assist users in finding what they are looking for.
Improve ad placement frequency, relevancy and promote Pro and Patron offerings.
Improve algorithms to show relevant content, give users more control over preferences.
Simplify the homepage design, focus on the most important content, and test different versions with users.
Improve the process for creating and accessing watchlists, and make it more intuitive and user-friendly. Provide guidance to novice users on how to use the feature.
Clarify the CTA (+LOG) in the global navigation and use more descriptive language. Add explanatory tooltips to assist users.
Make the reviewer's name more visible and easier to read in the "popular reviews" section by increasing the font size.
Reorganize the navigation within settings to make it more intuitive and user-friendly.
Conclusion
Thank you for taking the time to read our report. Our team found this project to be a valuable learning experience, and we developed many skills throughout this process. If you have any further questions or would like to discuss our findings in more detail, please do not hesitate to contact me through my contact page.