Research Operations Community, ReOps+, published my article, Informed Consent: Vetting Research Software for Privacy, in the Research Operations’ Medium publication. I am a member of the ReOps+ board and they invited me to contribute this article, which discusses a study of several research platforms and how user experience researchers can be sure participants understand what is happening to their personal data. The information can help researchers protect their liability* and improve trust among your participants.
We’d like to be sure that the data about our research participants stays between us and the test participant, but are our participants fully aware of the data sharing agreements underlying their use of the testing tools? The confidentiality agreement they have with us is only part of the picture.
In this article, I’ll discuss how to ensure that your participants know how their data is collected and how it might be used or shared beyond the scope of the covered research product. I’ll focus on a mini audit of several user testing software packages that we performed based on the 10 attributes for respectful Me2B commitments that underlie the Internet Safety Lab’s ISL Safe Software Specification:
Clear data processing notice
Viable permission
Identification minimization
Data collection minimization
Private by default
Reasonable data use & sharing / Me2B deal in action
Data processing behavior complies with data subject’s permissions and preferences
“Me2B” is a flipping of the traditional shortcut, B2C or Business to Consumer, relationship and is designed to put the individual first.
“Me2T” is your relationship with the technology itself.
To understand the background let’s take a brief look at the data privacy legal landscape in the US. I’m not a lawyer, so this is really just a broad brush overview. Any legal questions should be discussed with your corporate counsel.
Data Governance
Participant data may be collected in a number of ways, such as entering numbers or text directly into forms, entering it into an account profile (if you have one) or via an aggregated profile obtained from third party data brokers. Behavioural data also may be collected from third parties or your own app use.
Those of us who collect, use and share data from our research participants are becoming subject to a greater and greater number of data protection laws. Each law has varying degrees of requirements, usually based on where the data subject lives, so you want to be sure to get your data governance policies right. And it’s fair to expect the same from usability software that collects and controls data from you and your participants.
Data Handling in Practice
Researchers collect and store data with a number of different tools that in turn use underlying technology that may also access this data. Knowing what entities might have access to data through the testing platform’s relationship with these underlying tools can help you to evaluate whether you are exposing your team or your participants to risks that come with these technologies. We like to call this the “Me2T” relationship and it is largely hidden from the user.
Lack of notice and consent to share data present significant risks.
Notice of data sharing and consent are key components of many of the data privacy laws that govern which data we can and cannot save, use or share. While the risk to the researcher is similar to those of the user testing platform, the platform also bears responsibility for ensuring that anyone participating in a test on their platform has an appropriate level of notification that the data is being collected and shared, and subsequently allow the participant control over whether they continue using it.
Data Safety Audit
Researchers collect and store data with a number of different research tools, and that creates that Me2T relationship between the individual and the technology. We created a mini audit based on our safety specification. It is not a scientific study, i.e., we didn’t do a randomised sample and it only reflects the software packages that either we use in our own research or those that we’ve documented from forums that we participate in. However, the results brought up some interesting questions. (As a note, these are all companies that I have used and am comfortable using).
Table 1: Data sharing by vendor
Source: Internet Safety Labs. Note that Usability Hub is now Lyssna.
You’ll notice from this list that most of the software we looked at shares data with Google and other external vendors. One shared data with Facebook’s ad network and two shared with Amazon and Microsoft (including Microsoft Forms).
In Table 2, you can also see that just for these eight vendors, there are a few dozen companies or company assets that are receiving data. The ones in bold are advertising or tracking software, which often have agreements to sell the data they collect through data brokers. Many of these tools aren’t necessarily exploiting user data, but they are doorways to entities that now have some access to your participants’ data and your participants should know about that.
Table 2: Third party data vendors discovered in this study
Source: Internet Safety Labs
Methodology
To do the analysis, we used a tool from Evidon called Trackermap that exposes tags that allow data sharing between entities. What you’re seeing below is a map of the underlying technologies that expose data from Google Forms and Microsoft Forms. Trackermap is a paid platform that is bundled with Evidon’s Tag Auditor product, but there are free tools, like Augustine Fou’s Page Xray, that maps server and data tracking requests.
Results
Trackermap scans for various requests by external sites. We were particularly interested in advertising (blue), analytics (red), and trackers (gold), as these are most likely to be integrated into a data broker network.
Fig. 1: Google Forms Trackermap. Source: Internet Safety Labs.Fig. 2: Microsoft Forms Trackermap. Source: Internet Safety Labs.
We started with Google Forms and Microsoft Forms because they are popular, free tools that don’t require a lot of expertise to set up. While we expected to see a lot of sharing within their own advertising networks, we only saw Microsoft sharing with Bing Ads. Google Forms did not share data with their advertising network.
Can the participants see this? Well, Google doesn’t require it, but researchers can add an additional description with information about the study and details for informed consent, if they choose to. Significantly, most of the form-based surveys that we reviewed didn’t actually do this.
A savvy user may see that Google has its own privacy policy at the bottom of the form. That’s one potential relationship, but the Google Forms survey we reviewed also indicated that there was another company involved, a panel recruiter called SurveySwap. This is another Me2T relationship. This means that there are a few third party technologies in play here (Google and the panel recruiter), but no reference to the consent practices for any of these underlying relationships other than Google’s privacy policy link. So maybe Google Forms doesn’t share much, but in this case, the participants in this survey are potentially exposed to data sharing by the panel company (see the Tracker Map results formSurveySwap below).
Fig. 3: Surveyswap Trackermap. Source: Internet Safety Labs.
We ran a few other tests. The table below shows the number of trackers, ad networks and analytics packages for several products commonly used in user research.
Table 3: Ad networks, data trackers and analytics packages by vendor
Source: Internet Safety Labs. Note that Usability Hub is now Lyssna.
Below are the tracker maps from live tests at the usability testing platforms that we examined, and you can see that these platforms share to both DoubleClick and Google Analytics:
Fig. 4: Trackermap results for Usability Hub’s (now Lyssna) Usability Test and First Click Test and Optimal Workshop’s TreeJack tree test and Optimal Sort card sort. Source: Internet Safety Labs.
The survey vendors we examined tended to have a smaller number of tracking vendors:
Fig. 5: Trackermap for TypeForm and SurveyMonkey forms. Source: Internet Safety Labs.
The third group that we looked at was panel recruiters, where we saw a lot of data sharing with entities like Facebook Ads, DoubleClick, Microsoft Marketing and Adobe Metrics:
Fig. 6: Trackermap results from UserInterviws and Prolific.io. Source: Internet Safety Labs.
…you should be asking yourself whether your participants are aware of these relationships and whether … [vendors] have access to the data they provide to you.
It’s important to note that panel recruiters create a relationship with the participant at the time when the participants create an account with the recruiter, usually before they sign up for your study. It’s not a relationship you control, and it is not likely that your research data is shared with the recruiter unless you use their platform to run the survey.
When you look at these results, you should be asking yourself whether your participants are aware of these relationships and whether they are aware that these entities might have access to the data they provide to you. We feel it’s a good idea to remind participants of any Me2T consent relationships that they have already entered into when they participate in your study.
What else can you do?
Product development is flawed. Often there is no consent at all when testing with potential users. What are some of the other things that you can do to ensure that you are fulfilling your role as a data collector?
Researchers should be advocating for informed consent, highlighting all of the potential recipients of the participant’s data, and referencing in the informed consent document any additional data policies underlying the usability, platform, software or panel recruitment programs that are in use. And you should make all of this part of your vendor selection process.
Software testing platforms should take a closer look at their data protection responsibilities and make a greater effort to inform participants and test creators of the data sharing policy, not just once, but every time they use your software.
…and if the 75 min read warning on LinkedIn scares you (it’s mostly charts anyway) jump to the intro and discussion to see what you really should be concerned about as digital makers. This is important information that every product designer and engineer should know.
Some interesting findings about product safety attitudes:
* When it comes to product safety, there’s a double standard among consumers for connected vs. unconnected products.
People expect product makers to be responsible for the safety of things like home goods, cars, cleaning products and the like. But they don’t have the same expectation when it comes to websites, Smart TVS and mobile apps.
* Many consumers appear unaware of the causal connection between personal and societal harms such as physical, emotional, reputational, and financial damage and the systemic loss of privacy tied to connected products and services.
Product consumers are subjecting themselves to more harms than they think when they trust digital product makers to take proper care of their personal information.
* Even though survey respondents didn’t score mobile apps as the “least safe” option—websites, smart automobiles and smart homes got that dubious honor—consumers expressed more concern about the safety of apps than the safety of other internet-connected products.
If you find that last point interesting, you will find Internet Safety Lab’s AppMicroscope educating. App Microscope displays Safety Labels for mobile applications. Currently, App Microscope contains over 1700 apps studied in the ISL 2022 K-12 EdTech safety benchmark.
In May, I was invited to speak at UX Lisbon, on Preventing Digital Harm in Online Spaces. At the main event, I presented the Internet Safety Lab’s framework for preventing digital harm in connected products. This included a discussion of the relationship technologies have with consumers. I demonstrated techniques designers should adopt to mitigate the digital harms and dark patterns that could potentially violate that relationship. You can download my presentation below.
On the first day of the event, I ran a half-day, pre-conference workshop titled “Designing Effective Search Strategies.” In this session, I introduced a new framework using observation as a powerful tool to understand site search behavior. To explore this, we broke into seven groups and worked on empathy maps, search personas and mapping the user journey. I also introduced including group personas (2 of the groups took as a hint to discover cocktail lounges in Lisbon). As a takeaway, all participants received a toolkit for crafting these artifacts and a step-by-step process to enhance product search. We got to eat yummy Portuguese snacks, too!
“Noreen … made the interesting point that if we build an accessible design we’ll also be solving many search problems.”
What a wonderful event, interesting and welcoming people and an absolutely unforgettable time!
I am available to teach your team preventing or mitigating digital harm. Or lead a workshop on how to understand user search behavior. I can lead workshops solo or with my colleagues at Information Architecture Gateway. Let me know if we can help.
In a session yesterday of the NSFCyberAmbassadors leadership training program, my breakout group were tasked with discussing a case study of a potential ethics violation in research data privacy. The Code of Conduct that we were to use to determine if a violation occurred was the Association for Computing Machinery’s (ACM).
The case study involved a research scientist who had made software to analyze three sets of participant data, including DNA records, medical records and social media posts. There was a problem with the program and the scientist wanted to be able to do a crowdsourced code review. They asked their ERB team to review whether they could release the codebase to the public to crowdsource the problem. The ERB approved the request as long as no participant data was also released or could be reidentified. The case expressed a statement that there was a risk of reidentifying data but didn’t say specifically how. Just that the request was approved.
My first impression was that the research scientist was hiding behind item 2.6 in the ACM Code of Conduct, which says to only do work within your area of competence. The way we read it, the researcher relied on the Ethics Review Board (ERB) to make the ethical determination. Since the ERB approved the study, was the researcher in the clear?
Conversation ensued about how a data analytics program that didn’t include test data could be tested, or whether it could be tested with dummy data and a sample of open social media posts/hashtags, etc. but that was actually aside from our real interest, which was the idea that technology developers, including those with less funding, but also those with fewer guardrails, may not be competent to or interested in make ethical decisions.
Someone brought up AI. People working in AI today or really any large, complex model affecting global populations, are often making decisions way outside of their area of competence. They may do well, in one or two disciplines, but understanding and unraveling the externalities of what the thing will do once it’s in the world is of lesser interest since they aren’t ethicists.
In fact, not all companies have ERBs and many big names, youknowwho, have quietly and unceremoniously disbanded their ethics teams. In a world of move fast and break things, it’s not their area of competence.
How do people locate and discover information online? Well, they type keywords into a search engine and then select items from the search results, right? This is the current mental model of how search/retrieval works for most users. But it’s not the only way people search, nor is it necessarily the most effective for the information seeker.
In this workshop, you will learn about ”Sense-making,” a search behavior that information architects, user experience (UX) and usability pros should not ignore. You will learn how individuals (and groups) plan and carry out search activities. How a searcher’s goals affect their sense-making tasks. And how accessible design and information architectures improve search performance. At the end, you you will understand how to optimize the user experience of your products and search engine results pages, so people get the information they need with less frustration.
Topics covered:
Approaches to sense-making & information seeking behavior
Searcher goals that affect sense-making tasks
How accessible design and information architecture improve search performance
Where & how to implement search-related sense-making in user personas/profiles & customer journeys
How to optimize individual search listings for findability & sense-making
Search strategies for apps, video, voice and ChatGPT
Exercises:
Individual and group search exercise
Analyze a selected web page for accessible design and search optimization
Incorporate search behavior characteristics into personas and JTBD
App, video and voice search optimization
Discussion of new and emerging forms of search experiences
Attendees will learn:
How to identify search behaviors and incorporate them in personas and JTBD tasks
How to architect & optimize different types of search experiences
How accessible design can improve search experiences for everyone
How search strategy differs for websites, apps, voice, video and emerging experiences
Are you creating products that respect the needs and autonomy of your users? Are you concerned about how GDPR, CPRA and the proposed ADPPA data privacy regulations might affect your digital product? Would you like to learn how to evaluate your digital products for safe technology behavior?
It’s possible to measure ethical behavior of technology, and Information Architecture heuristics provide a useful testing framework. Building on the Safe Tech Audit presented at IAC22, this full day workshop will discuss the role of usability and application of Abby Covert’s Information Architecture Heuristics in technology development standards from the 2011 National Strategy for Trusted Identity in Cyberspace to the Internet Safety Labs’ ISL Safe Technology Audit. You will learn to use the framework to perform a product integrity test of a web technology of your choice.
In Part One, you will learn:
What are technology specifications and how are findability, understandability, accessibility and usability factors in developing them?
How do the various data privacy laws affect the use and control of personal data?
How can Information Architecture inform the design of safe and respectful apps and websites?
In Part Two, you will
Perform Safe Technology audits to measure product safety of a website and compare different kinds of technologies (news sites, social media, university websites, retail stores, etc.)
Select a website or app of your own to test and develop a plan to address any negative results (you won’t be required to share the results if you don’t wish to)
As a group, we will create an artifact to present on IAC23 Poster Night.
“Tools for Accessibility” by Noreen Whysel. AI generated art produced at NightCafe Studio
I was on a call the other day where we were discussing identity services for underserved populations. Someone brought up Diversity, Equity, and Inclusion (DEI) as a framework for ensuring accessible services for all.
DEI, as applied to product and service design, is a three-pronged philosophy that asks if you are assuring that diverse perspectives and lived experiences are being considered in the design of the service; whether access to the design or service is fair to all categories of people; and whether those—whose diverse experiences are considered—feel safe, welcome and included in the service and its outcome.
We discussed DEI in our group, but one person became uncomfortable, insisting that it doesn’t matter who is using the services as long as everyone can use it. He was concerned that focusing on DEI might mean that the unique needs of people, like the parent of a disabled person, would be excluded from consideration in the design of a product or service.
I thought this was an odd framing. He isn’t wrong to worry that caregivers may not have the best-designed experiences, which is why Universal Design, or design that everyone can use without impediment, is so important as a framework.
But rejecting conversations about DEI outright seems short sighted.
As a framework, I like DEI because it offers a reminder that there are people who get forgotten in the design process. It asks questions like “Who are we including?” and “Who are we leaving out?” So, my colleague’s concern about addressing the needs of the parent of a disabled person is exactly the type of inclusion issue that a DEI framework can help to identify.
It is also an area that I have been focusing on at IA Gateway with Shari Thurow and Bev Corwin. We are working on a model for a group persona that addresses the search needs of caregivers and people with a medical concern, whether a family member, acquaintance or someone in guardianship care.
Earlier this week, the Kantara Initiative Leadership Council approved a new Charter for the Resilient Identifiers for Underserved Populations work group (RIUP WG). This work group combines two legacy work groups (WGs) from the Identity Ecosystem Steering Group (IDESG). IDESG formed in 2011 to provide a trust registry under the White House’s National Strategy for Trusted Identity in Cyberspace and absorbed by Kantara in 2018. As a member of the IDESG UX Committee, I wrote the User Experience Guidelines and Metrics document for the ID Ecosystem Framework Registry.
Under the new charter, two work groups, Federated Identifiers for a Resilient Ecosystem (FIRE WG) and Healthcare ID Assurance (HIAWG) will combine to form the RIUP WG. This group will address identity assurance concerns for underserved people, often referred to as “vulnerable populations” by healthcare sector.
1) WG NAME (and any acronym or abbreviation of the name):Resilient Identifiers for Underserved Populations Work Group (RIUP WG)
(2) PURPOSE: The purpose of the Work Group is to support vulnerable and underserved populations in America. At a high level, these populations include those with physical and cognitive disabilities, or who are homeless, impoverished, senior citizens, immigrants, incarcerated, institutionalized and otherwise underserved minority groups that need digital credentials to access online resources; particularly, online healthcare and financial resources. Without an easily reusable identifier, it is nearly impossible for these individuals to gain secure access to the resources and services that may be available to them.
We will work, in collaboration with other private sector and public agencies towards establishing identifiers and access management (IAM) solutions that respect privacy, promote efficiency, limit redundancy, reduce barriers to use/adoption, increase interoperability, improve security, enhance safety and trust, eliminate identification errors, support resiliency, and achieve greater empowerment across the entire spectrum of online transactions. The RIUP WG will identify, coordinate, innovate and harmonize with ongoing and emerging identity initiatives, standards, and technologies, and communicate our findings to all relevant stakeholders, both in the US and, selectively, with other countries, under the leadership of the Kantara Initiative.
(3) A SCOPE – Guidelines for Cultivating a User-Centric Trust and Promoting Adoption within Underserved Communities
About “Underserved Populations”
Why does the RIUP WG use “underserved” rather than “vulnerable” when discussing the needs of healthcare populations? The US Health and Human Services tends to use “vulnerable” or “vulnerable and/or underserved” when discussing needs of people who require healthcare services but do not reflect the typical healthcare technology user.
In human subject testing, the category generally includes the elderly, poor, pregnant women, children, and infants, and recently, incarcerated people. But for the purposes of access to healthcare services, it also includes rural populations, those with permanent and temporary disabilities, indigenous peoples and others who may object to being described as vulnerable. In these cases, people need services that may be difficult to find, therefore rendering them “underserved.”
I had a conversation with Dana Chisnell, a founding member of the US Digital Service now serving as Deputy Design Director at US DHS, who convinced me to use “underserved” as a descriptor for identifiers. While there will still be “vulnerable populations” requiring special services, “underserved” puts the onus of care on the service provider rather than the traits of an individual which may or may not reflect their needs, abilities or level of personal agency. This work follows my research interest at the Internet Safety Lab where we are changing the conversation around digital harms, where the outcome of a service or lack of service can be harmful.
What’s Next?
RIUP WG will begin by creating guidelines for cultivating a user-centric trust registry and promoting adoption within Underserved Communities. We will publish a Use Case for Trusted Identifiers for underserved populations. And with a universal design strategy we will emphasize, highlight and prioritize user scenarios/stories from vulnerable and underserved populations to improve services for all users. We will test the use case and user stories across different verticals and persons of varying backgrounds and cultures. And we will create a dictionary that is harmonized with industry terminology.
There are a lot of initiatives that we will be watching. NIST is drafting 800-63-4 Digital Identity Guidelines, so we will work on comments on how to incorporate the needs of underserved people. The HSS Office of the National Coordinator (ONC) referenced trust registries in its work on Social Determinants of Health for Medicaid and we are participating in its information forums. We also plan to update the MAAS draft to incorporate recommendations from these efforts.
Three women in a liminal space. Digital art generated by Night Cafe.
I have been working from home for a couple decades, so a number of things were new for me during the COVID pandemic, and the hardest was probably having everyone else at home with me. My husband and my eldest who was in college and studying remotely, and occasionally my youngest who was in off-campus housing at college in New Orleans, but ended up back at home due to COVID and storm evacuations.
During this time, I needed to change my routine. Sharing my office with my husband was difficult because he took frequent calls that broke my concentration and he’s a noisy typer. I moved my “office” to my son’s room and had to negotiate when I could and couldn’t socialize or use the kitchen, since everyone’s lunch schedules were all a bit different.
The other thing that was different was that my normally “out of the house” activities were also back in house. I teach at a local college that was online for several semesters. (I’ve been teaching on campus for the past semester, with some online weeks, but a lot of students are still in hybrid classes). The evening meetups and other professional networking that I used to go out for a night or two a week were still happening from home, which can be awkward when the family expects to eat and relax together normally at that time.
Posting the schedule on the door or through a shared calendar has been helpful for coordinating my family activities. My husband or one of my kids cook dinner and feed the cat on the nights when I have an evening meetup or when I am teaching online, just as they would have done when I wasn’t here. Taking calls outside when the weather cooperates was also helpful, though not ideal. The alternative to evening meetups was to find similar activities during the day, and I found quite a lot that fit my schedule, but it meant negotiating midday things again.
My students at CUNY City Tech, who are mostly college junior and seniors, are also only now getting to classes on campus. They have been under a lot of stress about finding internships and post-college jobs and generally negotiating their own living spaces with family. Having to attend school from home was one of the added stresses that pandemic lockdown caused for students. And negotiating hybrid schedules can be exhausting.
A few semesters ago at the beginning of the pandemic, I had a student I’ll call Fatima who was attending my class from home. Her brother was also at attending school from home and space was tight. Fatima complained to me that her mom was always on her to clean up and complaining about why she wasn’t doing her part. It was causing her a lot of stress worrying about sick relatives, schoolwork, and extra home chores on top of it all, when she would normally be taking classes on campus and have had the “excuse” of not being at home.
I think what was going on at her house and in the homes of many of my students at the time was similar to what was happening with household supply shortages at the beginning of pandemic lockdown, where things you would normally use at work or school were now being purchased and used at home. Only for Fatima, instead of a shortage of toilet paper and bleach, there was a shortage of liminal space, the time and physical passages between her school and home life, that allowed her to adjust to and negotiate the activities that happen in those spaces.
Fatima was at home. Her brother was at home. Her mom and dad were also at home. The stress of being the busy student, helpful daughter, and goofy friend crammed in one space was exhausting. When her mom was complaining about the mess, Fatima was operating in busy student mode, not helpful daughter mode, which caused conflict.
What seemed to click for Fatima and a lot of my students was the idea of the lost “Third Places” or those special places and conditions outside of home or school or work that define different aspects of our being, but that are now collapsed into a less private and more awkward home/Zoom life. Interestingly, Fatima’s mother was losing her third place, as well. She used to have space at home that was her own to be who she was when she was not surrounded by all the extra activity, people and messes that everyone, her children, her husband, living in the same space all the time created.
And with hybrid classes and workspaces continuing to conflate our work, home, social, spiritual, and mental lives, conflict and negotiation will likely continue as we sort out these disparate spaces. It is important to recognize these conflicts, and to be a lot more forgiving of ourselves and others as we negotiate this new way of living. Because we aren’t back to “normal” yet, if we ever will be.
