Category: usability

Client: OASIS/Identity Ecosystem Steering Group
Visit Website

My Role

I led user testing for the Identity Ecosystem Framework (IDEF) Registry as part of the National Strategy for Trusted Identity in Cyberspace (NSTIC), a White House initiative. The IDEF Registry, a digital identity standard assessment tool, launched its alpha version on June 6, 2016. Because development of the alpha version of the attestation form was ongoing, I was brought into an agile process with the goal to iterate improvements after the public launch. I worked directly with a contracted project manager, third party marketing and design companies, the Chair of the IDESG User Experience Committee and members of the IDEF Registry working group.

User Research

The goal of the user study was two-fold: first, to ensure that the assessment form was understandable to those users who wish to list their products and that it included sufficient and expected information needed to complete the form accurately, and second, to ensure that the registry listing itself was usable and understandable to users who are seeking identity solutions.

Test participants for the first goal included IDESG members and observers who provide identity services, including certification, authentication, authorization, registration and transaction intermediation, or who rely on identity services in their own internal systems and commercial products. We selected expert users because we expect that those who will be completing the attestation form have a high level of understanding of the privacy, security, interoperability and usability of their own products.

Tests included needs assessment interviews of 12 prospective users, followed by additional user tests of seven users. For the needs assessment, I interviewed 12 prospective study participants about their needs for identity standards assessment and how the current IDEF Registry assessment tool compares to similar industry and government standards. I wanted to understand if the IDEF tool addressed all of their concerns about privacy, security, interoperability and usability and to get a sense of whether the planned registry served their needs. General findings were presented in a Google slide presentation showing typical responses to eleven study questions, suggested improvements and the impact on the user expereince. These were discussed over two, 2- hour meetings of the IDEF Registry Working Group.

Usability Tests

After delivering my findings to the development team, I began to design usability tests. I employed an observational walkthrough of proposed and completed designs, an expert heuristics review, user surveys and follow-up interviews with seven registry users. I utilized card sorts, preference tests, cognitive walkthrough of wireframes and a live website, as well as observations and survey feedback of seven alpha site users as they completed the attestation form on the alpha website to develop recommendations for improvements.

I engaged four members of the User Experience Committee, all usability experts, to participate in a heuristic analysis using Neilsen-Norman Group’s 10 usability heuristics and Abby Covert’s IA Heuristics. These expert users primarily evaluated the assessment form, but also provided input on the usability of the registry listings themselves, as a proxy for typical registry listing users. Due to the early stage of development, the client did not wish to

Results

The results showed that while the IDEF was rigorous, the implementation of the assessment and registry listings needed improvement, particularly to address situations where more than one person or company department might need to be involved. There were a number of issues with the interface including layout and data visualizations that could use improvement. Since Usability was a major component of the assessment, I also developed a set of user experience guidelines and metrics for service providers to use in evaluating usability requirements of the attestation. These will be incorporated into the Usability section of the assessment guidance documents.

UPDATE (5/22/2017): As of late Spring 2017, nine companies have completed assessments. The website remains in alpha with my recommendations set for implementation when the next round of grant funding is approved. Should I be reengaged, the next studies will include user tests of participants seeking identity services.

UPDATE (6/15/2019): The Registry is currently 65% complete and has transferred to the Kantara Initiative’s Education Foundation as of December 2018. I am continuing to serve on an agile advisory team and am working on use cases for health care. I presented the registry and participated in roundtable discussions at the 2019 Health Information Summit in Washington, DC on June 4, 2019.

Note: I signed a Non-Disclosure Agreement and am unable to share any images aside from those made public at idecosystem.org and idefregistry.org. Detailed information about the project, the assessment and the User Experience Committee is available on the public IDESG Wiki. Some of the documents including a draft rewrite of the Usability Guidelines and Metrics have been made public at: https://wiki.idesg.org/wiki/index.php?title=Talk%3AUser_Experience_Guidelines_Metrics

Announcement:
The IDEF Registry: an open invite to commit to trusted digital identity solutions

Resources:
Identity Ecosystem Steering Group (IDESG)
IDEF Registry
Identity Ecosystem Framework – Baseline Functional Requirements

Announcement:
The IDEF Registry: an open invite to commit to trusted digital identity solutions

Resources:
Identity Ecosystem Steering Group (IDESG)
IDEF Registry
Identity Ecosystem Framework – Baseline Functional Requirements