About the Project
Overview: The Pratt Institute Library website supports graduate students, undergraduate students, and scholars in their research and learning endeavors. The library undoubtedly is a vital resource for the varying needs of Pratt's student and faculty body.
The purpose of this usability test is to evaluate the effectiveness of the current map functionality in helping Pratt students to find items in the schools Brooklyn and Manhattan Libraries. Through evalutative user research we hoped to answer the following questions:
- How users understand and interact with the map?
- How users understand the role of Reference Desk?
- Identify what can be improved and provide actionable recommendations
Duration of study: Approximately 4 weeks
Discipline: Prototyping, Usability Testing, Eye Tracking
Tools: Tobii Pro, Adobe Premiere Pro, and Sketch
Team
My Role
- Conduct stakeholder interviews
- Plan and conduct usability testing on 2 target users
- Analyze usability test results to uncover problems/issues
- Consolidating recommendations to tackle the usability problems/issues
- Craft a presentation deck, problem list, and highlights reel video
- Present findings and recommendations to the client/stakeholders
Deliverables
The following final deliverables were developed and provided to the client at the end of the study.
- Presentation Deck - A report in slide deck form which summarizes the study goals, approach, findings, detailed feedback and recommendations.
- Problem List - A document outlining each usability problem with a short text and assigned severity (Nielsen Norman Group’s Severity Ratings for Usability Problems)
- Highlights Reel Video - A video reel containing our findings with supporting video recordings of users during usability test.
Overview: The Pratt Institute Library website supports graduate students, undergraduate students, and scholars in their research and learning endeavors. The library undoubtedly is a vital resource for the varying needs of Pratt's student and faculty body.
The purpose of this usability test is to evaluate the effectiveness of the current map functionality in helping Pratt students to find items in the schools Brooklyn and Manhattan Libraries. Through evalutative user research we hoped to answer the following questions:
- How users understand and interact with the map?
- How users understand the role of Reference Desk?
- Identify what can be improved and provide actionable recommendations
Discipline
User Experience (UX), Prototyping, Usability Testing, Eye Tracking
Team
- Marta Miklaszewicz
- Wenjun (Juno) Zhou
- Yiqiao (Karen) Li
- Armon Burton
My Role
- Conduct stakeholder interviews
- Plan and conduct usability testing on 2 target users
- Analyze usability test results to uncover problems/issues
- Consolidating recommendations to tackle the usability problems/issues
- Craft a presentation deck, problem list, and highlights reel video
- Present findings and recommendations to the client/stakeholders
Deliverables
The following final deliverables were developed and provided to the client at the end of the study.
- Presentation Deck - A report in slide deck form which summarizes the study goals, approach, findings, detailed feedback and recommendations.
- Problem List - A document outlining each usability problem with a short text and assigned severity (Nielsen Norman Group’s Severity Ratings for Usability Problems)
- Highlights Reel Video - A video reel containing our findings with supporting video recordings of users during usability test.
Overview: The Pratt Institute Library website supports graduate students, undergraduate students, and scholars in their research and learning endeavors. The library undoubtedly is a vital resource for the varying needs of Pratt's student and faculty body.
The purpose of this usability test is to evaluate the effectiveness of the current map functionality in helping Pratt students to find items in the schools Brooklyn and Manhattan Libraries. Through evalutative user research we hoped to answer the following questions:
- How users understand and interact with the map?
- How users understand the role of Reference Desk?
- Identify what can be improved and provide actionable recommendations
Duration of Study: Approximately 4 weeks
Discipline: Prototyping, Usability Testing, Eye Tracking
Tools: Tobii Pro, Adobe Premiere Pro, and Sketch
Team
My Role
- Conduct stakeholder interviews
- Plan and conduct usability testing on 2 target users
- Analyze usability test results to uncover problems/issues
- Consolidating recommendations to tackle the usability problems/issues
- Craft a presentation deck, problem list, and highlights reel video
- Present findings and recommendations to the client/stakeholders
Deliverables
The following final deliverables were developed and provided to the client at the end of the study.
- Presentation Deck - A report in slide deck form which summarizes the study goals, approach, findings, detailed feedback and recommendations.
- Problem List - A document outlining each usability problem with a short text and assigned severity (Nielsen Norman Group’s Severity Ratings for Usability Problems)
- Highlights Reel Video - A video reel containing our findings with supporting video recordings of users during usability test.
Our Process
To capture and collect data during this study we used an eye tracker tool called Tobii Pro Lab which accurately captures and tracks users' eye movement. With this tool we were able to observe and understand where and how users are looking.
As with any test/study, prior to testing with actual users it is important to run a pilot test on all the tasks to ensure things run accordingly. This test was no different. The intial pilot test proved valuable in tweaking the test, procedures, and tasks.
One thing we found to be helpful during the test sessions, is asking the user to 'think aloud' as they worked through the various task during the test. Encouraging users to verbalize what they were thinking is a great way to capture what they are thinking/feeling and helps to add context to they what/why of what they do. Overall, this approach provided us with richer qualitative insights backed up with real time user reactions.
To capture and collect data during this study we used an eye tracker tool called Tobii Pro Lab which accurately captures and tracks users' eye movement. With this tool we were able to observe and understand where and how users are looking.
As with any test/study, prior to testing with actual users it is important to run a pilot test on all the tasks to ensure things run accordingly. This test was no different. The intial pilot test proved valuable in tweaking the test, procedures, and tasks.
One thing we found to be helpful during the test sessions, is asking the user to 'think aloud' as they worked through the various task during the test. Encouraging users to verbalize what they were thinking is a great way to capture what they are thinking/feeling and helps to add context to they what/why of what they do. Overall, this approach provided us with richer qualitative insights backed up with real time user reactions.
Dual-monitor setup: Participant is sat steadily facing the main monitor; Moderator sits at rear side to participant so as not to encourage conversation, Observer sat further by the side to observe facilitator's monitor and take notes
Designing the Study
The Participants
The target demographics were identified during the client meetings and the participants were recruited based on their recommendations:
- Pratt Institute Students
- Brooklyn Campus Students
- Undergraduate Students
Creating Task Scenarios
The following tasks were designed to gain insight into a user experience using the libraries map features and to identify the issues they would potentially face.
After each task we asked them to grade the test on a scale of 1 (easy) to 5 (difficult), how do you evaluate the difficulty of the task? Also while the task was still fresh in their minds, we gave them the opportunity to provide any suggestions or observations based on your experience with this task?
- Task 1 - Go to: https://library.pratt.edu/ and search for the Information Visualization book by Colin Ware.
- Task 1.1 - What information can you find on the website that will help you locate where this books is exactly?
- Task 1.2 - Now imagine you would like to retrieve this book personally at the library. Can you explain us step by step the process of retrieving the book?
- Task 2 - You are a student and you are looking for a thesis written by Nicholas Dease. Please search for this thesis and tell us how would you retrieve it.
After the user completed the task we wanted to a brief wrap up and Post-Test Questionnaire of their experience and get any futher insights they would like to share about the study, tasks, and/or Library website. During this wrap up we asked them the following questions:
- Do you find map functionality useful?
- If you were to retrieve a book in the library, would you use a map in the future again?
- Do you prefer the map or looking by circulation number?
- Any additional comments on what you liked and what you disliked about the map?
The Participants
The target demographics were identified during the client meetings and the participants were recruited based on their recommendations:
- Pratt Institute Students
- Brooklyn Campus Students
- Undergraduate Students
Creating Task Scenarios
The following tasks were designed to gain insight into a user experience using the libraries map features and to identify the issues they would potentially face.
After each task we asked them to grade the test on a scale of 1 (easy) to 5 (difficult), how do you evaluate the difficulty of the task? Also while the task was still fresh in their minds, we gave them the opportunity to provide any suggestions or observations based on your experience with this task?
Task 1 - Go to: https://library.pratt.edu/ and search for the Information Visualization book by Colin Ware.
Task 1.1 - What information can you find on the website that will help you locate where this books is exactly?
Task 1.2 - Now imagine you would like to retrieve this book personally at the library. Can you explain us step by step the process of retrieving the book?
Task 2 - You are a student and you are looking for a thesis written by Nicholas Dease. Please search for this thesis and tell us how would you retrieve it.
After the user completed the task we wanted to a brief wrap up and Post-Test Questionnaire of their experience and get any futher insights they would like to share about the study, tasks, and/or Library website. During this wrap up we asked them the following questions:
- Do you find map functionality useful?
- If you were to retrieve a book in the library, would you use a map in the future again?
- Do you prefer the map or looking by circulation number?
- Any additional comments on what you liked and what you disliked about the map?
The Participants
The target demographics were identified during the client meetings and the participants were recruited based on their recommendations:
- Pratt Institute Students
- Brooklyn Campus Students
- Undergraduate Students
Creating Task Scenarios
The following tasks were designed to gain insight into a user experience using the libraries map features and to identify the issues they would potentially face.
After each task we asked them to grade the test on a scale of 1 (easy) to 5 (difficult), how do you evaluate the difficulty of the task? Also while the task was still fresh in their minds, we gave them the opportunity to provide any suggestions or observations based on your experience with this task?
Task 1 - Go to: https://library.pratt.edu/ and search for the Information Visualization book by Colin Ware.
Task 1.1 - What information can you find on the website that will help you locate where this books is exactly?
Task 1.2 - Now imagine you would like to retrieve this book personally at the library. Can you explain us step by step the process of retrieving the book?
Task 2 - You are a student and you are looking for a thesis written by Nicholas Dease. Please search for this thesis and tell us how would you retrieve it.
After the user completed the task we wanted to a brief wrap up and Post-Test Questionnaire of their experience and get any futher insights they would like to share about the study, tasks, and/or Library website. During this wrap up we asked them the following questions:
- Do you find map functionality useful?
- If you were to retrieve a book in the library, would you use a map in the future again?
- Do you prefer the map or looking by circulation number?
- Any additional comments on what you liked and what you disliked about the map?
Findings
I found the in-person usability test to be extremely insightful on the potential issues users would face while using the map feature on Pratt Institutes Library Website. More importantly the Eye-Tracking tool provided quantitive (heatmaps and gazemaps) and qualitative data (observations and recordings) to support our recommendations. At a High level we found the following:
100%
of users recognized crucial elements on the map - such as pin marking the location of a book or a Reference Desk
87%
participants would like to use map functionality in future. Also this percentage of users identified issues with way finding on the map
100%
of users were able to successfully complete the tasks
100%
of users quickly found Map button - all users naturally proceed to click on it
Finding 1: Users had difficulties with wayfinding while using the maps.
Finding 1: Users had difficulties with wayfinding while using the maps.
Users had a hard time wayfinding while viewing the maps. They had a difficult time orienting themselves with the map, locating the door, and utimately could not figure out how to navigate to the pin (book/shelf) from the enterance.
Finding 2: Users asked to include the stack and book details on the map.
While the map provides a visual way to find the book, it currently lacks detailed information such as call numbers, stack information in a central place that could further assist the user find what they are looking for in the library. Instead this information is scattered or buried throughout the map and was often missed by the users.
Finding 3: Users were not familiar with the library terms used on the site.
Finding 3: Users were not familiar with the library terms used on the site.
While our users have been to the library several times before, they still were unfamiliar with some of the roles and terms used on the Library website. For instance the role of the Reference desk and what they could do there was unclear to users. Also users are not sure what and/or how to use book call numbers.
Finding 4: Users were confused about the meaning and labeling of some of the icons.
Finding 4: Users were confused about the meaning and labeling of some of the icons.
During the testing, users pointed out that some of the icons and labeling were incorrect and not consistent throughout the site. This could potentially cause confusion while a user is browsing the cataloge searching for what they need.
Finding 5: Users expressed an interest in being able to save the maps on their phone.
Adding additional functionalty to support a users experience is always a plus. A user mentoined she typically likes to save (or take screenshots) of maps to her phone just in case she can go back to them.
Recommendations
As we were analyzing the finding and developing the recommendations, we found there was some crossover between the recommendations and one or more findings. There for we decided to group the recommendations together instead of breaking them individually.
Recommendation 1: Clarify and provide easier wayfinding on the map.
We recommended to clarify wayfinding using arrows and paths, make key information more prominent, and to further distinguish what's most relevant to users goals, providing contexts and directions at a glance.
Current Design
Our Recommended Design
Recommendation 2: Provide users with additional information that is instructional and helpful.
We merged Call Number from the top and direction instruction from the bottom to a right-sided panel. We also added the item's title and the author's name, as well as stack and call number responding to users needs. A shelf number was not available at this point, but was recommended to implement in the future. All the relevant information that users might need help for will there be centralized and readily accessible while minimizing scrolling.
Current Design
Our Recommended Design
Recommendation 3: Create meaningful and revelant icons.
We suggested use icons specific for each library and keep only the ones that are relevant. Since reference desk and circulation desk are situated at the crucial points of interaction with students, users should be clearly clued in that these involve helps from a human.
Our Recommended Design
Outcomes
Happy clients!
The usability study of the Pratt Institute library was a great success and our client was extremely happy with the findings and recommendations we provided. They were specifically excited to have supporting data such as video recordings, heat maps, gaze maps to back up and support our recommendations. They especially appreciated how simple, on point and actionable our solutions were, and expressed interest in implementing as soon as possible.
The usability study of the Pratt Institute library was a great success and our client was extremely happy with the findings and recommendations we provided. He was specifically excited to have supporting data such as video recordings, heat maps, gaze maps to back up and support our recommendations. It was especially satisfying from our client that all of our recommendations were actionable and will can implemented shortly our presentation.
Through this study and course, I've learned that the more you can tell a complete story and by having videos of the user and seeing them work through the tasks and running into issues is extremely impactful.
What I'm proud of...
We were able to come up with recommendations that both satisfied our clients and met our users needs.
We managed to coordinate and work through the process with actual users and a fairly complex technology within a really tight timetable. For example, all of our test sessions were conducted within one week.
We harnessed the power of letting users tell the story. Neither the visuals alone nor us explaining or outlining could tell the full story of the issue. Being able to hear and see are rather more effective and impactful, and as a team we brought this to life.
What I wish I did better...
We have encountered some technical difficulties retrieving quantitative data, which constrained our analysis to Times of Interests. It would otherwise be interesting to explore other methods such as Areas of Interest (specific components of an interface) and dive deeper into various metrics.
I wish we could have pilot tested using retrospective think aloud method (participants watch a video of their gaze replay and explain the thoughts and feelings while they were carrying out the task) too and seen how it might work differently to eventually decide which one was better for our actual testings.
One last thing...
References
- Nielsen, J. (1994). Usability Inspection Methods. CHI’94, Boston,Massachusetts, USA, 413-414.
- Rogers, Y., Sharp, H., & Preece, J. (2011). Interaction Design: Beyond Human-Computer Interaction (3rd Ed.). Chichester, UK: John Wiley & Sons, Inc.
- Nielsen, J. (1994). Usability Inspection Methods. CHI’94, Boston,Massachusetts, USA, 413-414.
- Rogers, Y., Sharp, H., & Preece, J. (2011). Interaction Design: Beyond Human-Computer Interaction (3rd Ed.). Chichester, UK: John Wiley & Sons, Inc.