Workshop: Pinterest as Exhibition

Date: February 10, 2016
Facilitator: Noreen Whysel
Venue: Metropolitan Library Council of New York, New York, NY.
Event: NYC Digital Humanities Week
URL: https://nycdh.org/dhweek/event/pinterest-as-exhibition-gallery/

Slides: http://www.slideshare.net/nwhysel/dh-week-workshop-pinterest-as-exhibition

Pinterest offers a unique way to display and interrelate digital assets with a wider world of interconnected materials and activity. Learn how UK-based research group, Architecture_MPS promotes its journal articles, conferences and online resources by exposing its relationships with other research, exhibitions, and imagery.

As part of our engagement with scholarly communication AMPS provides current listings supported by additional materials relevant to both academics and discipline information professionals. Since 2014, we have used Pinterest for curating collections of images and articles on topics related to our published journal articles. The boards function as a resource guide or reference to current books, films, exhibits, conferences, lectures and competitions related to the AMPS remit.

Related Content

Architecture_MPS Pinterest Board: https://www.pinterest.com/architecturemps

Fire and Light: Illumination in Religious Art: https://metillumination.wordpress.com

Pinterest as Digital Archive poster presented on May 7, 2016 at the ASIS&T 2016 Information Architecture Summit, Atlanta, GA.: http://www.slideshare.net/nwhysel/pinterest-as-digital-archive-ia-summit-2016-atlanta

Press Mention: https://studentwork.prattsi.org/dh/2016/02/18/noreen-whysels-pinterest-as-an-exhibition-gallery-at-metropolitan-new-york-library-council-21016/

Information Architecture Gateway

I have been working with Shari Thurow and Bev Corwin on developing information architecture services and research for the science gateways community. To date, we have presented two posters and two workshops for Science Gateways Community Institute’s (SGCI) annual Gateway conference and The Information Architecture Conference. We also provide consulting services to members of the SGCI community. Our completed work is available at the Open Science Foundation.

Shari, Bev and I co-chair the Information Architecture Community Group at the World Wide Web Consortium. The group is for anyone interested in Information Architecture and standards, and to participate, discuss, share, support, develop, and learn about them

Science Gateway Research

Accessibility Compliance and Assessments for Gateway Websites in Life Sciences: Toward Inclusive Design

Contributors: Noreen Whysel, Shari Thurow, Bev Corwin
Event: The Information Architecture Conference
Date: April 20, 2021 (updated)

Contributors: Noreen Whysel, Shari Thurow, Bev Corwin
Event: Gateways 2020
Date: October 12, 2020

Abstract: Information architects and accessibility experts examined the accessibility of 50 randomly selected gateway websites in the Life Sciences category in the Science Gateways Community Institute (SGCI) catalog, using both manual and automated methodologies. None of these sites produced an accessible website as per W3C, WCAG 2.1, and Section 508 standards. The most common accessibility success in these websites was URL structure, which enables web browsers and search engines to access content. Both manual and automatic methodologies are needed to fully assess website accessibility. For example, a website can meet W3C accessibility standards and not be fully accessible. However, automated tools are useful for identifying and troubleshooting accessibility issues for expert reviews.

This research is currently being updated with a comparison to 50 earth science gateways for SGCI’s Gateways 2022 and was selected to be presented in San Diego on October 18, 2022.

https://osf.io/fkrcb/

Tutorials

Designing Effective Search Experiences

Date: May 24, 2023
Facilitator: Noreen Whysel
Event: UX-LX: UX Lisbon, FIL Meeting Center, Lisbon, Portugal
URL: https://www.ux-lx.com/speakers/noreen-whysel.html#workshop

Accessible Gateways: Communicating Inclusive Design Practices with Pattern Libraries

Date: October 26, 2023
Facilitators: Shari Thurow, Noreen Whysel
Event: Science Gateways 2023, Online
URL: https://www.theiaconference.com/

Accessible Gateways: Communicating Inclusive Design Practices with Style Guides

Date: October 27, 2023
Facilitators: Shari Thurow, Noreen Whysel
Event: Science Gateways 2023, Online
URL: https://www.theiaconference.com/

Sense-Making, Search Systems & Site Optimization Workshop

Contributors: Shari Thurow, Noreen Whysel
Event: IAC 21: The Information Architecture Conference
Date: April 20, 2021
URL: https://www.theiaconference.com/session/sense-making-search-systems-site-optimization-2/

How to Create and Maintain an Effective Information Architecture and Navigation System for Science Gateway Websites

Contributors: Shari Thurow, Noreen Whysel
Event: Gateways 2019
Date: October 30, 2019
URL: https://osf.io/p6e79/

Part 1 of this tutorial provides actionable tips and illustrates important IA topics such as:

  • Understanding Information architecture (what it is & why it is important)
  • Information hierarchy: Data > Information > Knowledge
  • Wireframing (blueprints for web pages)
  • Qualitative & quantitative methods to measure content findability
  • Applying IA to gateway tasks

Part 2 of this tutorial introduces and illustrates:

  • Five types of navigation schemes
  • When to use them
  • How to arrange & format navigation (no HTML skills required)
  • Navigation metrics for gateway sites
  • When to revise navigation

At the end of this tutorial, attendees will:

  • Know what to put in design templates to make gateway content more findable before & after people arrive on the site
  • Identify & measure significant architecture & navigation metrics
  • Learn various methods to identify, evaluate, & fix common IA/navigation issues
  • Communicate with web professionals, usability/UX professionals, & information architects in an informed way

This tutorial includes exercises, downloadable checklists (PDFs) and a bibliography.

Presentation: Architecture-Navigation-SGCI-2019.pdf

Checklist: Website-Navigation-Checklist-Omni.pdf

Additional Output

Whysel, N. (October 27, 2021). Information Architecture Community Group. (Host), W3C TPAC 2021. World Wide Web Consortium. [conference website]

Gesing, S. et al. A Vision for Science Gateways: Bridging the Gap and Broadening the Outreach. (July 19-22, 2021). Practice & Experience in Advanced Research Computing (PEARC21). Association for Computing Machinery. [conference website] [recording]

Varon, A. & Whysel, N. (May 5, 2021). Evaluating Science Gateways for Accessible Design. 34th Semi-Annual Dr. Janet Liou-Mark Honors & Undergraduate Research Poster Presentation. New York City College of Technology, Brooklyn, NY.

More Information Architecture projects

Internet Safety Labs

Continuing the work of the Me2B Alliance, Internet Safety Labs is on a mission to ensure safety and human dignity in connected products and services. We measure the behavior of technology to catalyze product teams, policymakers, and researchers to take action.

My Role

My role at Internet Safety Labs is twofold: As Director of Validation Research, l developed research operations to provide user experience and other research services to the various working groups, exploring questions around the consumer experience of their relationship with digital technology.

Secondly, I developed the product integrity testing framework (UX audit) for digital technologies, in particular mobile apps and websites. This framework, based on information architecture heuristics and coupled with data integrity and security testing, makes up the requirements for ISL Safe Software Specification.

In 2023, I continued working with ISL as a consulting researcher. I am currently working on a study funded by the Internet Society of consumer perceptions of safety testing of internet-connected products. We expect to publish a research report by year-end.

Methods

Product Integrity Testing

I continued to develop the UX Integrity framework for the ISL Safe Software Specification, a new version of the Me2B Safe Technology Specification. This framework was based on an application of Abby Covert’s Information Architecture Heuristics to ensure that notices of data collection, use and sharing is Clear, Findable, Accessible, Credible and Communicative, or understandable by a wide audience of human and machine readable or accessible devices. I trained a research staff member/developer on how to perform a UX audit.

Data Privacy Audit

I conducted a study of several usability testing and survey packages, using the ISL framework, to understand how these software programs collect, store and whether they share participant data with third parties. I presented my findings in a lightning talk on informed consent at the 2022 Symposium on Usable Privacy and Security.

Qualitative and Quantitative Research

I conducted interviews, focus groups and surveys to discover attitudes and behaviors related to the safety of software and software-enabled products. I documented insights from two focus groups of technology consumers on technology relationships, privacy policies and terms of use. I developed several surveys to further explore insights from qualitative research to understand whether the attitudes discovered in interviews were widely held. The research helped to evolve our understanding of how consumers experience their relationship and risks with respect to digital technologies. I also interviewed and surveyed product developers to discover their attitudes and procedures for product safety.

Sample research questions:

How do people describe their relationship with technology?

Under what conditions is it acceptable for a website or mobile app to know one’s location?

Do people change how they interact with a website when they are familiar with the legal terms of the website? Do they understand who and what these policies protect?

Do people want the ability to send their own legally binding Information Service Agreement to service providers? Do they want (and would they use) a data management dashboard for managing the personal information that gets shared with all service providers?

Tools

Usability Software: SurveyMonkey, Prolific.io, Optimal Workshop, Usertesting.com, Userinterviews.com

Survey Software: SurveyMonkey, Google Forms, Microsoft Forms, Typepad

Collaboration: Microsoft Teams, Zoom, Sharepoint, Excel, OneNote, Trello, Monday

Research Artifacts

Spotlight Report #7: Who Is Responsible for Internet-Connected Product Safety?. (Pending Publication, 2023). Internet Safety Labs.

Spotlight Report #6: Proffering Machine-Readable Personal Privacy Research Agreements: Pilot Project Findings for IEEE P7012 WG. (June 2, 2022). Internet Safety Labs.

Spotlight Report #5: Me2B Alliance Validation Testing Report: Consumer Perception of Legal Policies in Digital Technology. (January 18, 2022). Internet Safety Labs.

Spotlight Report #3: Me2B Alliance Validation Research: Consumer Sensitivity to Location Tracking by Websites and Mobile Apps (November 5, 2021). Internet Safety Labs.

Specifications

ISL Safe Software Specification for Websites and Mobile Apps V1.1. (July 5, 2022). Internet Safety Labs.

Me2B Safe Specification v1.0. (April 29, 2020). Internet Safety Labs.

Presentations

Preventing Digital Harm in Online Spaces (May 26, 2023). UX LX: UX Lisbon.

Informed Consent: Are Your Participants Aware of What They Share? (August 7-9, 2022). SOUPS2022: Symposium on Usable Privacy and Security. USENIX.

Statement on Dark Patterns at the California Privacy Protection Agency Stakeholders Meeting. (May 5, 2022). Internet Safety Labs.

Safe Tech Audit: IA as a Framework for Respectful Design (April 23, 2022). IAC22: Information Architecture Conference.

The Future of the Metaverse with Vint Cerf and Noreen Whysel. (April 12, 2022). Disruptive Technologists.

Shedding Light on Dark Patterns: A Case Study on Digital Harms. (April 28, 2021).
IAC21: Information Architecture Conference.

Webinar: Me2B Research: Consumer Views on Respectful Technology. (January 28, 2021). Internet Safety Labs.

Future Plans

Internet Safety Labs is seeking grant funding to continue its mission to keep tech safe. Visit at https://www.internetsafetylabs.org.

More research projects

Identity Ecosystem Framework Registry

Association for Computational Linguistics

Decision Fish Financial Wellness App

Me2B Alliance

Background

The Me2B Alliance develops safety specifications for connected technology. It is comprised of software engineers, policy analysts, UX experts, business and philanthropic leaders who are committed to giving individuals more say in how technology treats people. We are setting up a rigorous independent testing and certification program for websites, apps and connected devices. The Me2B Alliance is comprised of working groups for Me-s (the consumer), B-s (the business) as well as the Policy and Legal and Certification working groups. Together, we are setting the standard for Respectful Technology.

“Me2B” is a term of art in the data privacy community and is not a trademarked word. It is a play on the term B2C, where the needs of the individual—the “Me”—come first.

My Role

My role at the Me2B Alliance was twofold: I led the Validation Research practice to provide user experience and other research services to the various working groups, exploring questions around the consumer experience of their relationship with digital technology.

Secondly, I developed the product integrity testing framework (UX audit) for digital technologies, in particular mobile apps and websites. This framework, coupled with data integrity and security testing, makes up the requirements for a proposed “Me2BA Safe” certification.

I managed two junior researchers in the development of the specification and in report production for the Spotlight reports. I collaborated with data privacy experts on working groups and interviewed donors and SMEs, and advised on website redesign and persona development.

User Research Methods

Qualitative Research

I engaged consumers in one on one conversations about their relationship with technologies they use in their day to day lives. Research questions ranged from their understanding of privacy policies, terms of use agreements and other agreements they make implicitly by using a technology. for example, do users change how they interact with a website when they are familiar with the legal terms of the website? And would a score make a difference?

Preference Testing

I performed a series of tests of the certification mark to be used as a symbol of trust in connected digital technologies. This included interviews, focus groups, unmoderated 5-Second preference tests and surveys.

Interviews

I conducted interviews of subject matter experts, donors, policy experts, and prospective Me2B Alliance members to develop personas for the redesign of the me2balliance.org website. Additional interviews with general technology consumers explored their relationship with technology generally, their reactions to Me2B concepts and proposed website iconography, and their understanding of legal policies governing website and mobile app use.

I interviewed subject matter experts to understand their reaction to a proposed implementation of the IEEE P7012 standard for machine-readable personal privacy agreements. This research was conducted with a grant from the IEEE and published as a Spotlight report.

Focus Groups

I conducted two focus groups of general technology consumers comprising six sessions over a period of three months in 2021. Discussion topic included their relationship with technology, the design and naming of our research reports and guides, and the proposed specifications for safe internet technology. The research helped to evolve our understanding of how consumers experience their relationship and risks with respect to digital technologies. These insights were applied to the development of the Me2B Safe Technology Specification.

Information Architecture Methods

Product Integrity Testing

I developed a UX Integrity framework for the Me2B Safe and Respectful Technology Framework (now published as the Me2B Safe Specification). This framework was based on an applicaiton of IA heuristics to ensure that notices of data collection, use and sharing is Clear, Findable, Accessible, Credible and Communicative or understandable by a wide audience of human and machine readible or accessible devices.

Tools

Interviews and Focus Groups: Zoom, UserInterviews.com, SurveyMonkey

Preference Tests and 5 Second tests: UserInterviews.com, SurveyMonkey

Collaboration: Microsoft Teams, Zoom, Microsoft365, Trello, Monday

Artifacts

Spotlight Report #6: Proffering Machine-Readable Personal Privacy Research Agreements: Pilot Project Findings for IEEE P7012 WG. (June 2, 2022). Me2B Alliance.

Statement on Dark Patterns at the California Privacy Protection Agency Stakeholders Meeting. (May 5, 2022). Me2B Alliance.

Spotlight Report #5: Me2B Alliance Validation Testing Report: Consumer Perception of Legal Policies in Digital Technology. (January 18, 2022). Me2B Alliance.

Spotlight Report #3: Me2B Alliance Validation Research: Consumer Sensitivity to Location Tracking by Websites and Mobile Apps. (November 5, 2021). Me2B Alliance.

Me2B Safe Specification v1.0. (April 29, 2020). Me2B Alliance.

Presentations

Safe Tech Audit: IA as a Framework for Respectful Design (April 23, 2022)
Conference Presentation: Information Architecture Conference.

The Future of the Metaverse with Vint Cerf and Noreen Whysel. (April 12, 2022). Disruptive Technologists.

Shedding Light on Dark Patterns: A Case Study on Digital Harms. (April 28, 2021)
Conference Presentation: Information Architecture Conference.

Webinar: Me2B Research: Consumer Views on Respectful Technology. Me2B Alliance.

Future Plans

In July 2022, the Me2B Alliance changed its name to Internet Safety Labs. “We’ve changed our name, not our mission.” Visit at https://www.internetsafetylabs.org.

Association for Computational Linguistics

Background

The Association for Computational Linguistics’ daily operations have been led by a single Business Manager for over two decades. As she neared retirement, estimated in two to three years, the ACL required documentation of her businesses processes related to the day-to-day operations of the ACL so a new Business Manager will be able to continue operations with minimal disruption.

My Role

I was hired as a contractor to perform ethnographic research to understand ACL business policies and processes to ensure a successful transfer to new management upon the future retirement of the association’s Business Manager.

Method

The goal is to document the ACL Busines Manager’s daily workflows and office environment. I used ethnographic study techniques including interviews, observations, video and audio recordings. Audio and video recordings of the sessions are saved in MP4 format. Detailed notes with timestamps for the start and end of each recorded session are saved as a Microsoft Word file. A formal transcription was produced using the dictation tool in Microsoft Word. Because of the coronavirus travel restrictions, the observations took place as a series of Zoom calls over the course of a week beginning September 11, 2020 through September 18, 2020, with an additional call on November 2, after which an updated, final report was completed.

Research Questions

The following are the main research questions studied:

  • What are the day to day duties while managing business for the ACL?
  • How might the ACL ensure continuity of operations when the current Business Manager retires?

Outcome

The final report was delivered to the ACL Treasurer to support the development of office management policies.

More research projects

Internet Safety Labs

Identity Ecosystem Framework Registry

Decision Fish Financial Wellness App

Decision Fish

Background

Decision Fish applies the latest research in decision science, behavioral economics, corporate finance and user experience design to build a suite of apps that help people save money and reduce risk when making high-stakes, decisions, that align with personal values and create happiness. We are addressing an urgent social mission: to help people make better financial decisions, avoid debt and live happier, more secure lives.

My Role

Decision Fish is a small startup with three founders and a contracted designer and developer. My role at Decision Fish is Chief of Operations. While this role incorporated oversight of financial, facility and HR duties, I also developed the UX design and testing strategy, hired contract designers and wrote and carried out user research.

User Research Methods

Competitive Review

I performed a competitive review of similar applications available in the market, researched the problems and markets they addressed and evaluated gaps in prospective user needs. Many tools available for budgeting and planning focus on creating a budget based on prior expenditures and tracking and categorizing purchases as they happen. Most provide graphic tools for reviewing past performance (monthly or annual) and some have additional support for setting and tracking goals. Many assumed a certain level of financial literacy and do not provide education except for more basic financial concepts. Programs that provided a deeper knowledge require in class learning, which may not be retained, or ongoing personal coaching which can be costly.

Focus Groups and Workshops

I ran focus groups and financial wellness workshops to observe people discussing money issues and planning their finances. The focus groups took place at a career outplacement centers, local meetups and in our office. These incorporated informal discussions and affinity diagramming to understand how users feel while making different kinds of financial decisions.

Semiotics Web/Information Architecture Workshop on financial wellness. We created affinity diagrams of attendees biggest financial decisions and their feelings around these decisions.
Semiotics Web/Information Architecture Workshop on financial wellness. We created affinity diagrams of attendees biggest financial decisions and their feelings around these decisions.
Financial Wellness Workshop at Impact Hub NYC, featuring a paper prototype of our app.
Financial Wellness Workshop at Impact Hub NYC, featuring a paper prototype of our app.


As we developed our web app, we also hosted formal financial wellness workshops at Impact Hub NYC and Hunter College School of Social Work. The workshops use mindfulness exercises, play money and a paper version of the app as we were developing it to help people understand how to budget and evaluate personal goals.

Financial Wellness goals
Photo of Worksheet from Financial Wellness Workshop

Insights: Both groups discussed and documented their financial goals and fears as well as ways to develop empathy for their future selves. These in turn gave us insight and empathy for the varied goals and needs of potential users. The insight formed the basis of our trademarked Happy Budget model, a framework for spending that aligns what you spend money on with the things in life that make you happy.

Moderated User Tests

Our initial user tests included a series of online and in-person walkthroughs of a pre-alpha home affordability application developed by the CEO/product manager in Excel. We did not screen participants a this stage as we were still learning what types of users to target and wanted to have a broad group. It was important at this stage to use Excel as our prototyping tool to have a realistic representation of a participant’s financial picture while saving on development costs. The test began with an interview about financial decisions and habits and general demographics and continued with a test of the Excel prototype. Participants spoke aloud as they went through the initial home affordability module.

The feedback from early testing indicated that a home affordability calculator was interesting, but would not be used very often. Insights from interviews indicated that people wanted more help planning and keeping on top of their budget. They liked the just-in-time learning and that the interactive design, though somewhat clumsy in Excel, was approachable.

Insights: Our user research findings and competitive analysis during this phase of development allowed us to pivot from a home affordability focus to a broader financial planning model, incorporating a trademarked framework for budgeting that helps align spending with a user’s personal values.

Testing continued with Financial Planning and Budgeting modules developed again in an interactive Excel model. We invited prior participants and recruited from our newsletter subscriber list, members of our co-working space at Impact Hub and started to reach out to benefits providers, nonprofits and other potential customers.

One of the most common responses to the web app was “I really need this.” But we noted some issues with the workflow that overlooked some users’ situations, such as data fields and content that referenced children, which many, particularly older, female participants didn’t want to see. We also noted the dependence on a savings figure defined early in the program prevented some low income users and those with a lot of debt to continue to the budgeting module.

Insights: As we gathered insights, we refined the user flow and made notes on changes for the developer to make in future versions of the web app. We continued both in-person moderated tests, remote tests moderated via Zoom or Skype and began remote, unmoderated alpha tests.

To do an unmoderated user test, and to provide a realistic program for B2B clients, we needed a working web app. Our developer team was entirely remote and I realized that the Axure prototype would be difficult to share, so we moved to an InVision prototype to allow better, real time collaboration. I worked on user flows, content and a style guide. We hired a designer to produce high resolution screens in Sketch and used a combination of Slack, Trello and InVision to discuss and document changes. The resulting web app became our private beta product and we were ready to continue.

Unmoderated User Tests

During the private beta launch of the Decision Fish web app, we began screening participants. We focused on employed users in low to middle income levels ($30,000 to $80,000 per year) and invited prior participants to sign up for an online account. We also targeted users through our Mailchimp campaigns and began inviting prospective partners, who serve or employ people in our target group.

Remote sessions were recorded via Hotjar’s session recording tool, which we like because of its GDPR data privacy compliance. Users were aware that we would track their usage but that the recorder would not show the data they entered. We added a Hotjar feedback button to encourage users to enter thoughts on any page as they were experiencing problems or to report on things they liked.

Insights: We found it was important to many participants that we were careful with potentially sensitive personal information. Participants liked that we used an API to connect to financial data at their bank. When participants completed a module they were invited to complete a survey.

Tools

Prototypes: Paper, index cards, InVision, Axure, Excel

User Testing: Hotjar recordings and feedback forms, Google Forms, Surveymonkey, Craig’s List

Collaboration: Join.me, Zoom, Google Docs, Trello, Slack, Telegram

Artifacts

Early Prototype Screenshots

Below are prototypes from an early Azure version. The prototype interaction is overlaying an Excel based module created by the product manager.

IMG_1333
IMG_1335
IMG_1336
IMG_1338

I also created mobile screens in Axure. The following screen shows the ability to add fields based on the number of children the user wishes to include.

IMG_1326
IMG_1327

Collaborative Prototypes

InVision Prototype 1
InVision Prototype 2

Private Beta

The following are screenshots from the web app launched in February 2016.

Desi says representative screenshot
Big decisions... representative screenshot
Happy Budget... representative screenshot

Recognition

As COO, I am involved with developing partner and funding prospects, financial and HR operations and competitive market research. I was selected as a 2018 Fellow in the Startup Leadership Program and secured spots in the NYU StartEd Incubator for 2017 and Impact Hub’s 2018 30 for 30 Cohort.

2018 MetLife Inclusion Plus Semifinalist
2018 Impact Hub 30 for 30 Cohort
2017 NYU StartEd Incubator
2017 Best for NYC Changemaker Award, B Lab and NYC EDC
2016 Semi-Finalist, Village Capital FinTech

Future Plans

We are continuing to seek B2B collaborators to conduct pilot tests. An ideal pilot partner would be an employer or affinity organization serving individuals in our target population, particularly low-to-middle income, employed individuals.

We are developing financial education content and pathways to address needs of immigrants, single parents, unmarried and divorced women, veterans and other potentially underserved populations.

We plan to add a learning tool or online class, and are researching AI chatbots and online forums. We plan to create an interactive forum where users can discuss their goals based on their Happy Budget profile.

More research projects

Internet Safety Labs

Identity Ecosystem Framework Registry

Association for Computational Linguistics

IDEF Registry

Background

I led research and user testing for the Identity Ecosystem Framework (IDEF) Registry as part of the National Strategy for Trusted Identity in Cyberspace (NSTIC), an Obama era, White House initiative. The IDEF Registry is a digital identity standard assessment tool created by the Identity Ecosystem Steering Group (IDESG). The standard covers the privacy, security, interoperability and usability of identity tools. The outcome of an attestation is the ability for a company to display a trustmark on their product or service.

User Research Objective

The goal of the user study was two-fold: first, to ensure that the assessment form was understandable to those users who wish to list their products and that it included sufficient and expected information needed to complete the form accurately, and second, to ensure that the registry listing itself was usable, accessible and understandable to users who are seeking identity solutions. The following images represent the 2016 alpha launch of the website and the site that was redesigned as of Fall 2018.

IDESG Website Screenshot 2016
IDESG Website 2016
IDESG Website Screenshot 2018
IDESG Website 2018

My Role

The IDESG Self-Attestation Listing Service (SALS) launched an alpha version of an identity ecosystem registry on June 6, 2016. I had been a volunteer member and 2015 vice chair of the User Experience Committee, working on developing the standards for usability. Because development of the alpha version of the attestation form was ongoing and a series of tight deadlines looming, the SALS team invited me to resign my seat on the committee to join as a paid researcher.

The SALS team decided to form an agile process with the goal to iterate improvements after the public launch. I worked directly with a contracted project manager, third party marketing and design companies, the Chair of the IDESG User Experience Committee and other members of the IDESG.

Method

Test participants included primarily IDESG members and observers who provide identity services, including certification, authentication, authorization, registration and transaction intermediation, or who rely on identity services in their own internal systems and commercial products. We selected users who were likely to complete the attestation form on behalf of their companies or products because have a high level of understanding of the privacy, security, interoperability and usability of their own products.

Through several phases of development, we employed a number of user test methods, beginning with needs assessment interviews and open ended conversations to more formal user tests and later usability/user acceptance tests. We had planned to do end user tests with a general population of internet users, but did not have enough funding, which is a major challenge of publicly funded research and is described in detail below.

Needs Assessment Interviews

Tests included needs assessment interviews of 12 prospective users, followed by additional user tests of seven users. For the needs assessment, I interviewed 12 prospective study participants about their needs for identity standards assessment and how the current IDEF Registry assessment tool compares to similar industry and government standards.

I wanted to understand if their current frameworks addressed all of their concerns about privacy, security, interoperability and usability. Many are subject to various kinds of identity standards such as NIST 800.63, HIPPA or FICAM. I wanted to find out whether there were any gaps in the current processes for evaluating product trust, and to see how the planned IDEF registry might serve their needs.

General findings were presented in a Google slide presentation showing typical responses to eleven study questions, suggested improvements to the proposed registry design and the standards themselves, and the impact the improvements may have on the user acceptance. These were discussed over two, 2- hour meetings of the IDEF Registry Working Group.

Needs Assessment presentation to SALS Advisory Team

Pre-Alpha User Tests

After delivering my needs assessment findings to the development team, the design team began to prepare wireframes and prospective trustmark and scoring graphics. I designed usability tests based on the wireframes and initial website graphics.

I employed an observational walkthrough of proposed and completed designs, an expert heuristics review, user surveys and follow-up interviews with seven registry users. I utilized card sorts, preference tests, a speak-aloud user test of the website wireframes.

The tests were conducted remotely using conferencing software with screen sharing. I had arranged one in-person, moderated test during a trip to the Information Architecture Summit in Atlanta, but the participant was stuck in traffic and was not able to connect at our allotted time.

Findings were presented in several conference calls via screen sharing and Google Drive. Privacy and security was a key need for identity providers and those who relies on identity services. Interoperability was an issue for some but not all participants, and most participants had not considered usability though a few had done some research beyond user acceptance testing.

Participants indicated that a self-attestation was desirable, both as a way to publicly demonstrate conforming to standards but also as a way to test products in development for optimal privacy and security. Action based navigation was preferred.

Alpha Testing

There was some concern about how the trustmark score would be presented. Our second set of tests focused on the presentation of scored attestations. The original model used a similar registry presentation to Consumer Reports, indicating the level of completion of each of the standards by color-coded icons and a pie chart graphic. The listings could be expanded so that registry users could view how the service scored based on the type of identity product represented.

User tests included pre- and post-test survey of seven alpha site users. Some participants were interviewed as they completed the attestation form on the alpha website, while others were interviewed after they completed the process. The goal of these tests was to see if the attestation form and instructions were understandable and relatively easy to complete, to get a sense of how many people were involved with completing the form and to see if the user flow made sense.

Users found the scoring and graphics on the registry listing page to be understandable on the surface, but confusing when digging in more deeply. Many could not understand why some services seemingly attested to fewer standards, even though they were reporting for services that covered fewer operational categories. There was also some disagreement around when a standard could be listed as “Not Applicable” versus “Not Implemented”.

IDESG 2016 Provider Listing with old scoring

Based on the findings I developed recommendations for improvements. Insights included a need for detailed instructions for both attesters and registry visitors. Because the attestation might require input from technical, legal and product teams, participants wanted to have the ability to download a copy of the form and instructions, similar to how the IRS provides printable versions.

Site content needed to address registry visitors more. I determined that the registry represented a two-sided market and advocated for addressing the information needs of relying parties, or those who might use the registry to evaluate trusted identity services. I recommended testing prospective registry visitors to make sure that we understood how both audiences might navigate through the process.

There also seemed to be a lot of difficulty understanding the steps of the process from start to finish. Some minor issues included a mismatch between how the color of icons reflected the level of completion.

Expert Review

I engaged four members of the User Experience Committee, all usability experts, to participate in a heuristic analysis using Nielsen-Norman Group’s 10 usability heuristics and Abby Covert’s IA Heuristics. These expert users primarily evaluated the assessment form, but also provided input on the usability of the registry listings themselves, as a proxy for typical registry listing users.

User Testing for IDEF 2.0

In Fall 2017 I was reengaged to complete a user research study of the language in the standard requirements. This included telephone and in-person interviews of six IDEF Registry participants. We reviewed the responses to the attestation with each assessor and asked whether and what changes were made to their product as a result.

We launched a design sprint to update and clarify the trustmark graphics and website information architecture. I worked with members of the Agile Advisory Team and design contractor, Early Adopter, to advise and evaluate updates to the website, attestation form, and knowledge base.

We created sets of graphics representing the registry website responsive layout and icons, for both mobile and desktop. We designed several variations of the trustmark and score visualizations to replace the current graphics which users had found confusing.

IDESG Provider Listing with flyout scores
IDESG 2018 Provider Listing with updated flyout scores

I prepared additional user tests of these materials as well as graphics representing the trustmark and score received by service providers who register their products.

Formal testing began in June 2018. I successfully petitioned to attend the Identiverse conference in Boston, knowing that many of the global identity providers that we wanted to include in our study would be attending. I took a train up for the day and was able to get six more in-person interviews, including identity officers at international, federated identity providers and visualization companies.

Challenges

UX Research occurred under a much reduced budget in 2017 and 2018. Funding for Obama era initiatives have been eliminated or cut drastically under the current administration and this project was no exception: as of December 2017, IDESG was operating under a quarter of its original funding. The IDESG responded by restructuring the committees and seeking new sources of funding. The Kantara Intitiative, which had itself created a set of identity standards agreed to absorb the IDESG assets and working groups. The User Experience Committee continued its work as an agile advisory committee, but no additional funding is currently anticipated.

To address these challenges, the management committee set up an Agile Advisory Team to tighten the project scope and manage outcomes. I was invited to participate in these discussions in Fall 2018 as an unpaid advisory member.

Outcome

As of late December 2018, ten companies completed assessments. The updated website has been launched with the updated trustmark and score graphics. Several deliverables have been released on the public wiki and the development phase is in process, along with additional, iterative user testing.

Since Usability was a major component of the assessment, I also developed a set of user experience guidelines and metrics for service providers to use in evaluating usability requirements of the attestation. The guidelines include general information about performing usability tests and user research. It also contains tests specifically for addressing trust and identity products. These have been incorporated into the Usability section of the assessment guidance documents on the IDEF Registry website.

Public Deliverables:

The IDESG Usability Guidelines and Metrics document has been made public in the IDEF Registry Knowledge Base:
https://wiki.idesg.org/wiki/index.php?title=Talk%3AUser_Experience_Guidelines_Metrics.

Detailed information about the project, the assessment and the User Experience Committee is available on the public IDESG Wiki.

Since this project, IDESG’s assets have been absorbed and republished by the Kantara Initiative Educational Foundation.

Announcement:
The IDEF Registry: an open invite to commit to trusted digital identity solutions

Resources:
Identity Ecosystem Steering Group (IDESG)
IDEF Registry
Identity Ecosystem Framework – Baseline Functional Requirements

More research projects

Internet Safety Labs

Association for Computational Linguistics

Decision Fish Financial Wellness App

Empathy Jam 2017 – JourneyApp

Empathy Jam 2017 was a one-day UX Design hackathon, held on October 28 at Fordham University, Lincoln Center. Participants formed teams of up to five people and selected from two challenges.

Challenge: The JourneyApp team addressed Challenge B. How might we create tools and platforms that utilize AI for New Yorkers to learn and practice soft skills for a JOB SEARCH?

Problem Statement: Tom is a job seeker who wants to keep his professional hard/soft skills relevant to stay competitive in a job market that is becoming more automated. Tom needs a way to assess, collaborate, and identify needed hard/soft skills so that he can build connections with others, and leverage/improve upon existing skill sets.

Role: My role was to serve as the Lead UX Researcher. I led the team through research exercises including developing assumptions about potential users to test in the field and synthesizing our assumptions with our research findings. We split into two field study teams and my group, including Jen Green and Daniel Cho interviewed people on the plaza at Lincoln Center and on the Fordham campus about their use of technology for self improvement and job search and their concerns about potential displacement from automation.

Target users
Target users

Persona


We found that while most people we interviewed were unconcerned about automation, and in fact had a rather positive view of AI and assistive technologies, they were concerned about improving their soft skills and wading through an abundance of information to find the right programs. Interviews seemed to confirm that an AI would be a welcome tool to help sort through opportunities.

Assumptions
Assumptions

As a team, we ideated features of our application, which would employ an AI to interview users about their skills and experience and the types of new skills and occupations they would like to pursue. We decided the AI would then offer a roadmap for developing skills to help the job seeker grow into professions that interest the user, based on the interview and iterative feedback. The AI would suggest courses, professional networks, mentors and communities, and track completion of suggested tasks.

From these exercises, a teammate created a proposed user flow for the app. The next step was to map the user journey to our proposed user task flow. I created a journey map for onboarding as well as the overall tasks a user completes in the application. We then went outdoors again for more field testing to validate our assumptions.

User Journey Map - Assessment
User Journey Map – Assessment

After refining our user flow, my teammate Kristin Bodkin created a mock-up of the web application while Vanessa Sanchez prepared the presentation deck for the final demos.

Here is a twitter post from @EmpathyJam showing our presentation.

Empathy Jam 2017 – JourneyApp Demo

We didn’t win any prizes, but we learned a lot about the UX research and product ideation processes, made new friends and had a wonderful day of developing empathy for our local community.

Map Mosaic: From Queens to the World

Amy Jeu and I curated a weekend exhibit, Map Mosaic: From Queens to the World, on October 29-30, 2017 at the Queens Museum celebrating the map-making community. The event featured talks and demonstrations as well as a hall dedicated to paper and digital maps submitted from the private collections of members of the GISMO community. These maps represent a wide range of themes including the diverse Queens neighborhood and demographics, urban planning, environmental studies, election analysis and more.

Visitors at Map Mosaic: From Queens to the World, Queens Museum, NY
Mezzanine Level with map exhibit and children's activity tables at Map Mosaic: From Queens to the World, Queens Museum, NY
Mezzanine Level with map exhibit and children's activity tables at Map Mosaic: From Queens to the World, Queens Museum, NY
Five maps with placards at Map Mosaic: From Queens to the World, Queens Museum, NY
Table with flyers and Dr. Suess book for children's story hour at Map Mosaic: From Queens to the World, Queens Museum, NY
Four visitors at interactive map station, one wearing 3D glasses at Map Mosaic: From Queens to the World, Queens Museum, NY
Interactive map station with 3D glasses at Map Mosaic: From Queens to the World, Queens Museum, NY

My Submissions

For my contribution to the exhibit, I created a cutout map of the 1964 World’s Fairgrounds to teach children how map layers work in GIS. This series of maps, printed on acrylic transparency sheeting can be stacked to show through various layers: Base Map, Parks, Buildings, Streets/Paths. We also provided additional paper and colored pencils for children to use. This activity helped younger visitors to understand the concept of map layers in GIS.

Because the event was held over Halloween weekend, I also contributed a set of themed maps with Halloween parade routes and a “Crime of the Century” story map retelling the activities from the 1934 Ice House Heist in Brooklyn and Upper West Side Manhattan. The piece included reproductions of aerial photographs from the time period.

Documentation

Each item in the exhibition included a placard indicating the name of the mapmaker, the materials used and a brief description of the subject. We used icons to indicate whether an interactive version was available at the computer stations or that the mapmaker is also a speaker in our forum.  

interactive

Interactive Map

speaker

Speaker

Amy Jeu created the flyer and copy for the exhibit which was published on the Queens Museum website and the signage used for the exhibit and presentations. I created the placards and the online exhibit catalog.

Archive

The Map Mosaic event was privately curated. Queens Museum published an announcement and the exhibit catalog and list of interactive maps are available at GISMO’s Website. The acrylic manipulative work is located in the GISMO archive. All maps produced by the NYC Office of Emergency Management were donated to the Queens Museum and all other, individual artwork was returned to the artists.

Queens Museum Website Announcement
Exhibit Catalog
Interactive Maps

Unite for Humanity Hackathon

In July 2016, I was on the organizing team for the Unite for Humanity Hackathon, one of 30+ Open Camps held at the United Nations over three weeks. My main task was planning logistics and preparing challenges based on the United Nation’s 17 Sustainable Development Goals (SDGs). The Hackathon was unusual in that instead of the usual overnight event, this one was held over an entire week of activities, including guest speakers and ideation sessions, plus an overnight sprint at the UN General Assembly Building in New York City.

United Nations Sustainable Development Goals

My co-producer, Amine Tourki and I divided responsibilities. Amine focused on working with Hackathon teams, coordinating communications, via a Gitter Channel, preparing rooms and equipment and advising teams, while I focused on booking speakers, helping sponsors create challenges and setting up resources on the Unite for Humanity Github site.

Screenshot of Unite for Humanity Hackathon Github site
Unite for Humanity Hackathon Github site

Our opening event was held in a conference room in the UN General Assembly building. Speakers included our host Radia Funna, Head of Innovation at the UN Office of Information and Communications Technology, who presented on the OITC Crowdsourcing Platform, Humana Khan, Executive Director of Muflefun.org and Mubin Shaikh, a Counterterrorism Subject Matter Expert. Khan and Shaikh discussed counter-terrorism activities via Skype, followed by presentations by other challenge sponsors, including Wendy Brawer of Greenmap System, Sonia Hamel fro DOable, Marcus Iannozzi
of B Lab on Being a Tech Focused B-Corp, and others.

Screenshot from Unite for Humanity EtherPad
Unite for Humanity EtherPad

It was really good to hear which of the UN goals people were most fired up about and that also helped people form teams. One woman said she was a UX designer and could do great pitch decks. And since I was building the event repository, I offered to help with research or datasets.

Over the course of the week, we met daily with teams on the 27th floor of the UN Secretariat building. We worked with teams through project ideation all week up to our final weekend sprint at UN General Assembly. Some of the artifacts created included affinity diagrams, sketches and workflows.

Target SGDs, Unite for Humanity Hackathon, United Nations WHO: Affinity diagram of actors, Unite for Humanity Hackathon, United NationsGraph of actors, climate change vs counterterrorism, Unite for Humanity Hackathon, United Nations
Loop and Benefit Metrics, Unite for Humanity Hackathon, United Nations

#ProjectScarlett was the first place winner of the Unite for Humanity Hackathon. Additional project descriptions are available at the Unite for Humanity Github.

Team photo. #ProjectScarlett, winner of the 2nd Annual Unite for Humanity Hackathon
#ProjectScarlett, winner of the 2nd Annual Unite for Humanity Hackathon. Team leader: Mario Salazar (front row, center)