Usability Evaluation

This project covers usability testing conducted on specific tasks in the MyUHS portal, the University of Texas at Austin’s online booking and health records management system. The goals of the usability testing included establishing a baseline of user performance, establishing and validating user performance measures, and identifying navigation concerns to be addressed in order to improve the efficiency, productivity, and end-user satisfaction of both students and University Health Services staff.

Executive Summary

University Health Services is uniquely challenged by a patient population that is navigating the healthcare system for the first time without parental guidance. UHS uses Point and Click Solutions (PNC) as its Electronic Health Records and Practice Management platform. This allows UHS to manage patient records electronically, send secure messages, collect pre-visit questionnaires, and book appointments online. The online booking process, utilized by 57% of students, is tailored to generate appointment options related to the specific reason a student reports they want to see a provider. If a student is scheduling a well-visit, they will be given appointment options that may be a few weeks away with any of the providers authorized to complete a well-visit. If a student is experiencing non-debilitating knee pain, they will be given appointment options within a few days with a provider who can effectively diagnose joint-related issues.

However, UHS reports students are not booking appointments online correctly, and the errors are not being discovered until the student in question arrives for his or her appointment, reporting symptoms that differ from the appointment slot they were given. This means the pre-visit questionnaire they completed online during the booking process does not ask the correct questions and the provider they will be seeing may not be appropriate for the ailment the student wants to have treated. Worse yet, some students are booking appointments online and waiting several days for an appointment, when they should have been seen immediately because the issue they have is one that warrants immediate attention.

This study, conducted with two other group members and the cooperation of UHS, explored the ideal routes to get students to the correct services, as outlined by UHS, and identified the points where users diverged from the ideal path. While it is understood that corrective action may not be possible due to the limitations of the PNC platform, it was hoped that the client could use data captured in this study to negotiate with PNC for future improvements in the platform, or alternately, use it as a baseline for seeking a replacement vendor.

Methodology

A test participant pool of eight subjects was tasked with booking appointments online using the University Health Services online booking system and completing five distinct scenarios / test tasks. Participants were encouraged to speak aloud as they worked through each process. Our team also identified the ideal outcome to be followed by working with UHS staff – as such each task had a highly specific goal in mind during the test.

Our test plan originally included a subjective evaluation questionnaire. After subsequent meetings with UHS discussing the inherent limitations of the PNC software, we eliminated this from our efforts. Instead, we focused our testing efforts on the participants’ process of navigating the system. Whenever possible, we captured commentary related to the page content and the order in which instructions appear, which UHS can modify if necessary.

Task	Description	Ideal outcome
Task 1	What are the steps you would take to find health center office hours, phone number and online booking information?	UHS encourages online booking for most appointments that are not acute issues
Task 2	You have a persistent cough that is keeping you and your roommate up at night. Walk us through your process to be seen for this issue.	Acute cough should call for immediate appointment
Task 3	You twisted your knee playing soccer and it hurts periodically. Walk us through your process to be seen for this issue.	This issue should book with sports medicine
Task 4	You need to book your annual well woman visit. Walk us through your process to be seen for this issue.	This issue should book with women’s Health
Task 5	One of your professors invited you to participate in an Antarctic research expedition – this requires a physical before you can be cleared to join the research team. Walk us through your process to book the appointment.	This issues should book with general medicine -> Antarctica physical

Evaluation Tasks/Scenarios

For the purposes of this test, our team worked closely with UHS staff (specifically the UHS nursing director) to determine the tasks that would identify common pain-points in the booking process, which had proven to be problematic, leading to frequent errors and incorrect submissions. For the purpose of the test scenario, we asked the user to imagine they were a new undergraduate student at UT Austin, and enrolled in the Student Health insurance plan provided by the University. The user was asked to seek out an on-campus medical professional via MyUHS, and to:

Find health center office hours, phone number and online booking information
Use the system to book an appointment for a persistent cough
Book an appointment for an injury sustained while playing soccer on a non-UT affiliated team
Book an annual well women’s health exam.
Book a physical for a specific study abroad/research project

Results

Task Completion Success Rate
Task completion was measured by the tester as the the ability to conclude a task without encountering a critical error. Tasks 4 and 5 were not tested on participants 7-8 and average completion rate has been adjusted accordingly. It is worth noting that task 2, which had the lowest completion rate, was always tested in the same order. It was the first encounter for users with the online booking process, which could point to a possible learnability issue that should be corrected in subsequent testing by randomizing task order.

Time on Task
The tester was responsible for documenting the time on each task for each participant. Some tasks were inherently more difficult to complete than others, which is reflected by the average time on task.

Task 1 required participants to discover the best way to contact MyUHS, using the Healthy Horns primary site (http://healthyhorns.utexas.edu). This task took the longest amount time to complete (mean = 198.2 seconds). In contrast, Task 3 took the shortest amount of time (mean = 109.3 seconds), due primarily to a large number of participants who opted to just call UHS and abandon the MyUHS booking system. These tasks were marked as 30 seconds each, to accommodate the time to read the prompting text on the page and dial the UHS booking line.

Errors
Errors were captured by the tester in the survey of their target sample. A non-critical error is an error that does not prevent successful completion of the scenario. In the case of the MyUHS booking system, there are certain scenarios that would ideally be resolved by a phone call. Therefore, in the instance of tasks 1 and 5, a phone call is considered a successful conclusion to the task. In the case of tasks 2-4 it is considered a critical error.

Summary of Data

The table below displays a summary of the test data.

Task	Task Completion	Non-Critical Errors	Critical Errors	Average Time on Task
1	10	2		204.5
2	4	9	5	194.3
3	6	4		129.1
4	6	1		111.3
5	5	5	1	197.5

Recommendations

This test plan was executed with the understanding that UHS was beholden to a contractor, PNC, and would be limited in their ability to execute massive changes to the existing service deployment. The ultimate goal was to identify and provide recommendations for:

Small changes that could improve the usability of the system in its current deployment
Language clarifications that could help more effectively direct users to the ideal outcomes
Larger changes that UHS could leverage in order to strengthen its position in the renegotiation of the PNC relationship

Severity Rating

The severity rating below is based on two factors: impact and frequency.

Impact is the ranking of the consequences of the problem by defining the level of impact that the problem has on successful task completion. There are three levels of impact:

High: Prevents the user from completing the task (critical error)
Moderate: Causes the user difficulty, but the task can be completed (non-critical error)
Low: Minor problems that do not significantly affect task completion (non-critical error)

Frequency is the percentage of participants who experience the problem when working on a task. There are three levels of frequency:

High: 30% or more of the participants experience the problem
Moderate: 11% – 29% of participants experience the problem
Low: 10% or fewer of the participants experience the problem

Change	Justification	Severity
Increased visibility on language urging participants to call	A recurring theme in the MyUHS experience is the inconsistency of branding and unclear calls to action – this was noted both in user comments and test results. Adding images and iconography to call-outs urging users to opt-out of the online booking system when appropriate would help reduce the number of urgent care appointments that are incorrectly booked online, when in fact they should be escalated to urgent care.	High
Continue v. Cancel placement / visual representation and left-to-right reading	Most of the non-critical errors found in the testing process related to the placement of progress buttons in the booking process. The ‘continue’ button in the lower left side of the page contradicts the standard (albeit western), left-to-right reading pattern of most users. A change in this placement would likely reduce these errors.	High
Shorten text overall (tl;dr)	The booking site is overly text-heavy, with over 40% of the participants seeming to click buttons before completely reading the instructions (this metric was difficult to test accurately). While UHS is limited in what they can control in the PNC system, it appears basic HTML and type hierarchy should be within their control to edit or request changes to. Much like the recommendations around urging participants to call, UHS could utilize images, iconography and columns layouts to more accurately direct their clients to the proper destination.	Moderate
Improved Calendar/Appointment View	In initial conversations with UHS staff, the nursing director mentioned wanting an ‘Orbitz style’ view of the available appointments, the goal of which was unclear to the testing team. However, following the testing process it became clear that the scheduling component of the booking system lacked any of the modern web design functions of sorting and filtering (similar to those found on modern travel sites). The addition of sort, search and filter features would greatly enhance the MyUHS experience.	Moderate
Spell out acronyms	9.2% of UT’s student body in 2013 came from outside the United States. 25% of our test participants were not U.S. citizens, and did not understood the abbreviations and terms used in the booking process (see Participant Feedback and Recommendations section above). This is a very quick and easy fix that can improve the process for a large portion of the student population.	Moderate
Taxonomy of ailments	Several users noted the lack of depth in the specificity of symptoms, a lack of descriptive language, as well as the inability to select multiple options or selections that seem indistinguishable (i.e. cough and respiratory symptoms). UHS should undertake further testing to understand how its target audience is approaching its booking experience, with a specific focus around language used and increased specificity with regards to the symptom	Low

Conclusion

As a group, participants found the MyUHS site cumbersome, dated, unintuitive, and not particularly user-friendly. An excessive amount of text throughout made it difficult to immediately understand the appropriate next step(s), and resulted in participants skimming for crucial information, and often overlooking it. User feedback indicates that MyUHS should consider trimming repetitive content and focusing on clear directions to help the user quickly and correctly advance through the system.

Unconventional button placement coupled with indistinct graphic design — several users commented that the buttons were hard to notice at first, as they look so similar to the text, with the same font and background color — caused participants to fail to proceed correctly, or to feel unsure of the next step. Users also often clicked the wrong button, because they instinctively assumed the button on the right would take them forward in the process. The fact that this error was repeated by multiple users, and often multiple times, indicates that switching the “Continue” and “Cancel” button placement would help to improve usability, and by extension, the user experience.

Limited language / descriptive text and selection options in terms of the various ailments a user can seek medical care for led to some confusion among participants, with multiple users choosing the incorrect provider for the condition in question, or prioritizing the wrong symptom (since the system only allows the user to select one ailment or symptom). Adding descriptive text for each ailment could help users make more accurate selections when booking online appointments, and ultimately reduce instances of users accidentally booking appointments with the wrong providers.

Each of these issues contributed to various task failures and a general sense of frustration and confusion. These results provide UHS with a general, big-picture view of their average user’s experience, as well as insights into some of the pain points and roadblocks that frequently prevent successful online appointment booking. The data and user feedback collected in this study can also be used to facilitate discussions with the service provider around the current system configuration, as well as future service negotiations.

In the short term, continuing to gather feedback from their users, and implementing some of the simple recommendations provided here — minimizing unnecessary text, making button placement more visible and intuitive, and providing more detail around specific ailments and symptom classifications — will help to improve the overall MyUHS user experience.

Team

Chrystal Keogh
Amy Rothbaum
Wade Treichler

Completed: May 6, 2015