Background
I led research and user testing for the Identity Ecosystem Framework (IDEF) Registry as part of the National Strategy for Trusted Identity in Cyberspace (NSTIC), an Obama era, White House initiative. The IDEF Registry is a digital identity standard assessment tool created by the Identity Ecosystem Steering Group (IDESG). The standard covers the privacy, security, interoperability and usability of identity tools. The outcome of an attestation is the ability for a company to display a trustmark on their product or service.
User Research Objective
The goal of the user study was two-fold: first, to ensure that the assessment form was understandable to those users who wish to list their products and that it included sufficient and expected information needed to complete the form accurately, and second, to ensure that the registry listing itself was usable, accessible and understandable to users who are seeking identity solutions. The following images represent the 2016 alpha launch of the website and the site that was redesigned as of Fall 2018.
My Role
The IDESG Self-Attestation Listing Service (SALS) launched an alpha version of an identity ecosystem registry on June 6, 2016. I had been a volunteer member and 2015 vice chair of the User Experience Committee, working on developing the standards for usability. Because development of the alpha version of the attestation form was ongoing and a series of tight deadlines looming, the SALS team invited me to resign my seat on the committee to join as a paid researcher.
The SALS team decided to form an agile process with the goal to iterate improvements after the public launch. I worked directly with a contracted project manager, third party marketing and design companies, the Chair of the IDESG User Experience Committee and other members of the IDESG.
Method
Test participants included primarily IDESG members and observers who provide identity services, including certification, authentication, authorization, registration and transaction intermediation, or who rely on identity services in their own internal systems and commercial products. We selected users who were likely to complete the attestation form on behalf of their companies or products because have a high level of understanding of the privacy, security, interoperability and usability of their own products.
Through several phases of development, we employed a number of user test methods, beginning with needs assessment interviews and open ended conversations to more formal user tests and later usability/user acceptance tests. We had planned to do end user tests with a general population of internet users, but did not have enough funding, which is a major challenge of publicly funded research and is described in detail below.
Needs Assessment Interviews
Tests included needs assessment interviews of 12 prospective users, followed by additional user tests of seven users. For the needs assessment, I interviewed 12 prospective study participants about their needs for identity standards assessment and how the current IDEF Registry assessment tool compares to similar industry and government standards.
I wanted to understand if their current frameworks addressed all of their concerns about privacy, security, interoperability and usability. Many are subject to various kinds of identity standards such as NIST 800.63, HIPPA or FICAM. I wanted to find out whether there were any gaps in the current processes for evaluating product trust, and to see how the planned IDEF registry might serve their needs.
General findings were presented in a Google slide presentation showing typical responses to eleven study questions, suggested improvements to the proposed registry design and the standards themselves, and the impact the improvements may have on the user acceptance. These were discussed over two, 2- hour meetings of the IDEF Registry Working Group.
Pre-Alpha User Tests
After delivering my needs assessment findings to the development team, the design team began to prepare wireframes and prospective trustmark and scoring graphics. I designed usability tests based on the wireframes and initial website graphics.
I employed an observational walkthrough of proposed and completed designs, an expert heuristics review, user surveys and follow-up interviews with seven registry users. I utilized card sorts, preference tests, a speak-aloud user test of the website wireframes.
The tests were conducted remotely using conferencing software with screen sharing. I had arranged one in-person, moderated test during a trip to the Information Architecture Summit in Atlanta, but the participant was stuck in traffic and was not able to connect at our allotted time.
Findings were presented in several conference calls via screen sharing and Google Drive. Privacy and security was a key need for identity providers and those who relies on identity services. Interoperability was an issue for some but not all participants, and most participants had not considered usability though a few had done some research beyond user acceptance testing.
Participants indicated that a self-attestation was desirable, both as a way to publicly demonstrate conforming to standards but also as a way to test products in development for optimal privacy and security. Action based navigation was preferred.
Alpha Testing
There was some concern about how the trustmark score would be presented. Our second set of tests focused on the presentation of scored attestations. The original model used a similar registry presentation to Consumer Reports, indicating the level of completion of each of the standards by color-coded icons and a pie chart graphic. The listings could be expanded so that registry users could view how the service scored based on the type of identity product represented.
User tests included pre- and post-test survey of seven alpha site users. Some participants were interviewed as they completed the attestation form on the alpha website, while others were interviewed after they completed the process. The goal of these tests was to see if the attestation form and instructions were understandable and relatively easy to complete, to get a sense of how many people were involved with completing the form and to see if the user flow made sense.
Users found the scoring and graphics on the registry listing page to be understandable on the surface, but confusing when digging in more deeply. Many could not understand why some services seemingly attested to fewer standards, even though they were reporting for services that covered fewer operational categories. There was also some disagreement around when a standard could be listed as “Not Applicable” versus “Not Implemented”.
Based on the findings I developed recommendations for improvements. Insights included a need for detailed instructions for both attesters and registry visitors. Because the attestation might require input from technical, legal and product teams, participants wanted to have the ability to download a copy of the form and instructions, similar to how the IRS provides printable versions.
Site content needed to address registry visitors more. I determined that the registry represented a two-sided market and advocated for addressing the information needs of relying parties, or those who might use the registry to evaluate trusted identity services. I recommended testing prospective registry visitors to make sure that we understood how both audiences might navigate through the process.
There also seemed to be a lot of difficulty understanding the steps of the process from start to finish. Some minor issues included a mismatch between how the color of icons reflected the level of completion.
Expert Review
I engaged four members of the User Experience Committee, all usability experts, to participate in a heuristic analysis using Nielsen-Norman Group’s 10 usability heuristics and Abby Covert’s IA Heuristics. These expert users primarily evaluated the assessment form, but also provided input on the usability of the registry listings themselves, as a proxy for typical registry listing users.
User Testing for IDEF 2.0
In Fall 2017 I was reengaged to complete a user research study of the language in the standard requirements. This included telephone and in-person interviews of six IDEF Registry participants. We reviewed the responses to the attestation with each assessor and asked whether and what changes were made to their product as a result.
We launched a design sprint to update and clarify the trustmark graphics and website information architecture. I worked with members of the Agile Advisory Team and design contractor, Early Adopter, to advise and evaluate updates to the website, attestation form, and knowledge base.
We created sets of graphics representing the registry website responsive layout and icons, for both mobile and desktop. We designed several variations of the trustmark and score visualizations to replace the current graphics which users had found confusing.
I prepared additional user tests of these materials as well as graphics representing the trustmark and score received by service providers who register their products.
Formal testing began in June 2018. I successfully petitioned to attend the Identiverse conference in Boston, knowing that many of the global identity providers that we wanted to include in our study would be attending. I took a train up for the day and was able to get six more in-person interviews, including identity officers at international, federated identity providers and visualization companies.
Challenges
UX Research occurred under a much reduced budget in 2017 and 2018. Funding for Obama era initiatives have been eliminated or cut drastically under the current administration and this project was no exception: as of December 2017, IDESG was operating under a quarter of its original funding. The IDESG responded by restructuring the committees and seeking new sources of funding. The Kantara Intitiative, which had itself created a set of identity standards agreed to absorb the IDESG assets and working groups. The User Experience Committee continued its work as an agile advisory committee, but no additional funding is currently anticipated.
To address these challenges, the management committee set up an Agile Advisory Team to tighten the project scope and manage outcomes. I was invited to participate in these discussions in Fall 2018 as an unpaid advisory member.
Outcome
As of late December 2018, ten companies completed assessments. The updated website has been launched with the updated trustmark and score graphics. Several deliverables have been released on the public wiki and the development phase is in process, along with additional, iterative user testing.
Since Usability was a major component of the assessment, I also developed a set of user experience guidelines and metrics for service providers to use in evaluating usability requirements of the attestation. The guidelines include general information about performing usability tests and user research. It also contains tests specifically for addressing trust and identity products. These have been incorporated into the Usability section of the assessment guidance documents on the IDEF Registry website.
Public Deliverables:
- IDEF Registry Interviews: File:Post-Alpha User Research Report.docx
- UX Guidelines: File:Noreen-UX Guidelines-w Requirements.pdf
- Evaluation of Requirements: File:Evaluation of Requirements-User Research Requirements and Scope.docx
- IDEF Registry Final Analysis: File:IDEF Registry Interviews-Requirements-Analysis.xlsx
The IDESG Usability Guidelines and Metrics document has been made public in the IDEF Registry Knowledge Base:
https://wiki.idesg.org/wiki/index.php?title=Talk%3AUser_Experience_Guidelines_Metrics.
Detailed information about the project, the assessment and the User Experience Committee is available on the public IDESG Wiki.
Since this project, IDESG’s assets have been absorbed and republished by the Kantara Initiative Educational Foundation.
Announcement:
The IDEF Registry: an open invite to commit to trusted digital identity solutions
Resources:
Identity Ecosystem Steering Group (IDESG)
IDEF Registry
Identity Ecosystem Framework – Baseline Functional Requirements