Research Operations Community, ReOps+, published my article, Informed Consent: Vetting Research Software for Privacy, in the Research Operations’ Medium publication. I am a member of the ReOps+ board and they invited me to contribute this article, which discusses a study of several research platforms and how user experience researchers can be sure participants understand what is happening to their personal data. The information can help researchers protect their liability* and improve trust among your participants.
We’d like to be sure that the data about our research participants stays between us and the test participant, but are our participants fully aware of the data sharing agreements underlying their use of the testing tools? The confidentiality agreement they have with us is only part of the picture.
In this article, I’ll discuss how to ensure that your participants know how their data is collected and how it might be used or shared beyond the scope of the covered research product. I’ll focus on a mini audit of several user testing software packages that we performed based on the 10 attributes for respectful Me2B commitments that underlie the Internet Safety Lab’s ISL Safe Software Specification:
Clear data processing notice
Viable permission
Identification minimization
Data collection minimization
Private by default
Reasonable data use & sharing / Me2B deal in action
Data processing behavior complies with data subject’s permissions and preferences
“Me2B” is a flipping of the traditional shortcut, B2C or Business to Consumer, relationship and is designed to put the individual first.
“Me2T” is your relationship with the technology itself.
To understand the background let’s take a brief look at the data privacy legal landscape in the US. I’m not a lawyer, so this is really just a broad brush overview. Any legal questions should be discussed with your corporate counsel.
Data Governance
Participant data may be collected in a number of ways, such as entering numbers or text directly into forms, entering it into an account profile (if you have one) or via an aggregated profile obtained from third party data brokers. Behavioural data also may be collected from third parties or your own app use.
Those of us who collect, use and share data from our research participants are becoming subject to a greater and greater number of data protection laws. Each law has varying degrees of requirements, usually based on where the data subject lives, so you want to be sure to get your data governance policies right. And it’s fair to expect the same from usability software that collects and controls data from you and your participants.
Data Handling in Practice
Researchers collect and store data with a number of different tools that in turn use underlying technology that may also access this data. Knowing what entities might have access to data through the testing platform’s relationship with these underlying tools can help you to evaluate whether you are exposing your team or your participants to risks that come with these technologies. We like to call this the “Me2T” relationship and it is largely hidden from the user.
Lack of notice and consent to share data present significant risks.
Notice of data sharing and consent are key components of many of the data privacy laws that govern which data we can and cannot save, use or share. While the risk to the researcher is similar to those of the user testing platform, the platform also bears responsibility for ensuring that anyone participating in a test on their platform has an appropriate level of notification that the data is being collected and shared, and subsequently allow the participant control over whether they continue using it.
Data Safety Audit
Researchers collect and store data with a number of different research tools, and that creates that Me2T relationship between the individual and the technology. We created a mini audit based on our safety specification. It is not a scientific study, i.e., we didn’t do a randomised sample and it only reflects the software packages that either we use in our own research or those that we’ve documented from forums that we participate in. However, the results brought up some interesting questions. (As a note, these are all companies that I have used and am comfortable using).
Table 1: Data sharing by vendor
Source: Internet Safety Labs. Note that Usability Hub is now Lyssna.
You’ll notice from this list that most of the software we looked at shares data with Google and other external vendors. One shared data with Facebook’s ad network and two shared with Amazon and Microsoft (including Microsoft Forms).
In Table 2, you can also see that just for these eight vendors, there are a few dozen companies or company assets that are receiving data. The ones in bold are advertising or tracking software, which often have agreements to sell the data they collect through data brokers. Many of these tools aren’t necessarily exploiting user data, but they are doorways to entities that now have some access to your participants’ data and your participants should know about that.
Table 2: Third party data vendors discovered in this study
Source: Internet Safety Labs
Methodology
To do the analysis, we used a tool from Evidon called Trackermap that exposes tags that allow data sharing between entities. What you’re seeing below is a map of the underlying technologies that expose data from Google Forms and Microsoft Forms. Trackermap is a paid platform that is bundled with Evidon’s Tag Auditor product, but there are free tools, like Augustine Fou’s Page Xray, that maps server and data tracking requests.
Results
Trackermap scans for various requests by external sites. We were particularly interested in advertising (blue), analytics (red), and trackers (gold), as these are most likely to be integrated into a data broker network.
Fig. 1: Google Forms Trackermap. Source: Internet Safety Labs.Fig. 2: Microsoft Forms Trackermap. Source: Internet Safety Labs.
We started with Google Forms and Microsoft Forms because they are popular, free tools that don’t require a lot of expertise to set up. While we expected to see a lot of sharing within their own advertising networks, we only saw Microsoft sharing with Bing Ads. Google Forms did not share data with their advertising network.
Can the participants see this? Well, Google doesn’t require it, but researchers can add an additional description with information about the study and details for informed consent, if they choose to. Significantly, most of the form-based surveys that we reviewed didn’t actually do this.
A savvy user may see that Google has its own privacy policy at the bottom of the form. That’s one potential relationship, but the Google Forms survey we reviewed also indicated that there was another company involved, a panel recruiter called SurveySwap. This is another Me2T relationship. This means that there are a few third party technologies in play here (Google and the panel recruiter), but no reference to the consent practices for any of these underlying relationships other than Google’s privacy policy link. So maybe Google Forms doesn’t share much, but in this case, the participants in this survey are potentially exposed to data sharing by the panel company (see the Tracker Map results formSurveySwap below).
Fig. 3: Surveyswap Trackermap. Source: Internet Safety Labs.
We ran a few other tests. The table below shows the number of trackers, ad networks and analytics packages for several products commonly used in user research.
Table 3: Ad networks, data trackers and analytics packages by vendor
Source: Internet Safety Labs. Note that Usability Hub is now Lyssna.
Below are the tracker maps from live tests at the usability testing platforms that we examined, and you can see that these platforms share to both DoubleClick and Google Analytics:
Fig. 4: Trackermap results for Usability Hub’s (now Lyssna) Usability Test and First Click Test and Optimal Workshop’s TreeJack tree test and Optimal Sort card sort. Source: Internet Safety Labs.
The survey vendors we examined tended to have a smaller number of tracking vendors:
Fig. 5: Trackermap for TypeForm and SurveyMonkey forms. Source: Internet Safety Labs.
The third group that we looked at was panel recruiters, where we saw a lot of data sharing with entities like Facebook Ads, DoubleClick, Microsoft Marketing and Adobe Metrics:
Fig. 6: Trackermap results from UserInterviws and Prolific.io. Source: Internet Safety Labs.
…you should be asking yourself whether your participants are aware of these relationships and whether … [vendors] have access to the data they provide to you.
It’s important to note that panel recruiters create a relationship with the participant at the time when the participants create an account with the recruiter, usually before they sign up for your study. It’s not a relationship you control, and it is not likely that your research data is shared with the recruiter unless you use their platform to run the survey.
When you look at these results, you should be asking yourself whether your participants are aware of these relationships and whether they are aware that these entities might have access to the data they provide to you. We feel it’s a good idea to remind participants of any Me2T consent relationships that they have already entered into when they participate in your study.
What else can you do?
Product development is flawed. Often there is no consent at all when testing with potential users. What are some of the other things that you can do to ensure that you are fulfilling your role as a data collector?
Researchers should be advocating for informed consent, highlighting all of the potential recipients of the participant’s data, and referencing in the informed consent document any additional data policies underlying the usability, platform, software or panel recruitment programs that are in use. And you should make all of this part of your vendor selection process.
Software testing platforms should take a closer look at their data protection responsibilities and make a greater effort to inform participants and test creators of the data sharing policy, not just once, but every time they use your software.
…and if the 75 min read warning on LinkedIn scares you (it’s mostly charts anyway) jump to the intro and discussion to see what you really should be concerned about as digital makers. This is important information that every product designer and engineer should know.
Some interesting findings about product safety attitudes:
* When it comes to product safety, there’s a double standard among consumers for connected vs. unconnected products.
People expect product makers to be responsible for the safety of things like home goods, cars, cleaning products and the like. But they don’t have the same expectation when it comes to websites, Smart TVS and mobile apps.
* Many consumers appear unaware of the causal connection between personal and societal harms such as physical, emotional, reputational, and financial damage and the systemic loss of privacy tied to connected products and services.
Product consumers are subjecting themselves to more harms than they think when they trust digital product makers to take proper care of their personal information.
* Even though survey respondents didn’t score mobile apps as the “least safe” option—websites, smart automobiles and smart homes got that dubious honor—consumers expressed more concern about the safety of apps than the safety of other internet-connected products.
If you find that last point interesting, you will find Internet Safety Lab’s AppMicroscope educating. App Microscope displays Safety Labels for mobile applications. Currently, App Microscope contains over 1700 apps studied in the ISL 2022 K-12 EdTech safety benchmark.
In May, I was invited to speak at UX Lisbon, on Preventing Digital Harm in Online Spaces. At the main event, I presented the Internet Safety Lab’s framework for preventing digital harm in connected products. This included a discussion of the relationship technologies have with consumers. I demonstrated techniques designers should adopt to mitigate the digital harms and dark patterns that could potentially violate that relationship. You can download my presentation below.
On the first day of the event, I ran a half-day, pre-conference workshop titled “Designing Effective Search Strategies.” In this session, I introduced a new framework using observation as a powerful tool to understand site search behavior. To explore this, we broke into seven groups and worked on empathy maps, search personas and mapping the user journey. I also introduced including group personas (2 of the groups took as a hint to discover cocktail lounges in Lisbon). As a takeaway, all participants received a toolkit for crafting these artifacts and a step-by-step process to enhance product search. We got to eat yummy Portuguese snacks, too!
“Noreen … made the interesting point that if we build an accessible design we’ll also be solving many search problems.”
What a wonderful event, interesting and welcoming people and an absolutely unforgettable time!
I am available to teach your team preventing or mitigating digital harm. Or lead a workshop on how to understand user search behavior. I can lead workshops solo or with my colleagues at Information Architecture Gateway. Let me know if we can help.
“Tools for Accessibility” by Noreen Whysel. AI generated art produced at NightCafe Studio
I was on a call the other day where we were discussing identity services for underserved populations. Someone brought up Diversity, Equity, and Inclusion (DEI) as a framework for ensuring accessible services for all.
DEI, as applied to product and service design, is a three-pronged philosophy that asks if you are assuring that diverse perspectives and lived experiences are being considered in the design of the service; whether access to the design or service is fair to all categories of people; and whether those—whose diverse experiences are considered—feel safe, welcome and included in the service and its outcome.
We discussed DEI in our group, but one person became uncomfortable, insisting that it doesn’t matter who is using the services as long as everyone can use it. He was concerned that focusing on DEI might mean that the unique needs of people, like the parent of a disabled person, would be excluded from consideration in the design of a product or service.
I thought this was an odd framing. He isn’t wrong to worry that caregivers may not have the best-designed experiences, which is why Universal Design, or design that everyone can use without impediment, is so important as a framework.
But rejecting conversations about DEI outright seems short sighted.
As a framework, I like DEI because it offers a reminder that there are people who get forgotten in the design process. It asks questions like “Who are we including?” and “Who are we leaving out?” So, my colleague’s concern about addressing the needs of the parent of a disabled person is exactly the type of inclusion issue that a DEI framework can help to identify.
It is also an area that I have been focusing on at IA Gateway with Shari Thurow and Bev Corwin. We are working on a model for a group persona that addresses the search needs of caregivers and people with a medical concern, whether a family member, acquaintance or someone in guardianship care.
THE OCCASIONAL MENTOR A monthly column based on questions I’ve answered on Quora, heard on Slack groups, and other career advice I’ve given over the prior month. Hope you like it, but feel free to challenge me in the comments, if you have a different experience. Below are questions I answered in August.
To find the answer look at the labels. Data or users? Are you more comfortable working with data or with people?
Data scientists work with data sets and computational analysis, while UX designers focus on people and their needs and behaviors.
Data scientists work with tabular data, charts, graphs, statistics/graphics programs like R and computer languages like Python, JSON and SQL. Their subject matter expertise is mathematics.
UX designers work with drawing and wireframing software, Post-Its, whiteboards and Sharpies. And lots of discussion, interviewing, observation, surveying and feedback. Their subject matter ultimately is people who use the products they design.
In some companies there may be an overlapping of the roles. For example a data scientist may work with user generated data, such as usage logs, to analyze behavior. A UX designer may help the data scientist test a visualization that is understandable to the users. So if you are interested in both you may be able to find roles that focus on your area of expertise, but give you some exposure to the other disciplines.
What Is the Best Way to Become Successful User Experience/User Interface Designer and Promote Yourself for Someone Who Is Completely New to this Career Path
Be sure to read a wide variety of subject matter. Read about philosophy, cognitive science and behavioral economics. Daniel Kahneman’s Thinking Fast and Slow and Thaler/Sunstein’s Nudge are good ones to start. Also read in areas where you have particular subject matter expertise or interest as you are most likely to succeed in getting a job, and enjoying it, in a product area you can be passionate about. I’m currently reading Gary A. Klein’s Sources of Power, a book that focuses on high stakes decision-making by military and emergency personnel and Planning for Everything, by Peter Morville, who coauthored Information Architecture for the Web and Beyond.
Watch: There are a lot of great conferences and talks that post their materials online that you can watch for free or for a small fee. I like IxDA’s Interaction Conference, Enterprise UX from Rosenfeld Media and Jared Spool’s UIE conferences. UIE collects talks in an “All You Can Learn” Library that are very good quality.
You can also find video courses on platforms like Udemy and Vimeo. I am currently taking a Cooper design course at Udemy taught by Alan’s Cooper, whose company Cooper.com, a user experience design and strategy firm offers design training. IDEO also has online design courses though these can be pricy for someone just starting out.
Listen: If you search “top ten UX podcasts” you’ll find most of the good ones. UX Podcast is the most cited. I like Postlight’s Track Changes. It has the banter of Car Talk and isn’t always so serious.
Also, since UX is all about the user, really build your listening muscle by listening to what people around you are saying about the products and services they use. What kind of language to they use when describing their experiences? What common problems or complaints do people have? Are they articulate or vague? Sometimes the vague ones are the most interesting to explore.
Talk: Find UX and Design related Meetups in your area and get out and talk to Designers. Ask them questions. What do they do? What do they love and hate about it? What are their most interesting or wicked challenges. Meetups are wonderful opportunities to network with UX designers, hiring managers and other likeminded people who can serve as mentors and travel buddies on your UX journey.
THE OCCASIONAL MENTOR A monthly(ish) column based on questions I’ve answered on Quora, heard on Slack groups, and other career advice I’ve given over the prior month. Hope you like it, but feel free to challenge me in the comments, if you have a different experience. Below are questions I answered in June.
How are the user experience design and data science professions connected with each other?
“Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from data in various forms, both structured and unstructured, similar to data mining.”
A data scientist is a person who is skilled in quantitative research and can formulate a study, analyze the results and create reports to inform other people about the topic of study. They may work with spreadsheets, statistical programs, graphical interfaces, and programming languages like Python, Java, JSON, R, SQL, MATLAB, SAS, C and F#, among others. They may also work with text analysis software, geographic information systems (GIS) and visualization tools like Tableau and Gephi.
UX designers use the results of quantitative research, created by data scientists and UX researchers. The reports help the designers understand user behavior, based on data collected from digital product user logs, web analytics, or quantitative user research tests. These data may describe typical user paths and places where users tend to drop off or bounce away from the app. It could include the results of A/B tests, card sorts, heatmaps, user flow diagrams and demographic and conversion data.
UX designers may also use the output of data studies in the content of the products they are designing for. These studies would be relevant to the subject of the product, not user generated data. For example, an infographic or other visualization that illustrates aspects of the product: weather maps, income disparity charts, election results.
What is the difference between a content strategist and a UX writer?
A content strategist creates a plan for all of the company’s reusable content assets. This can include graphics, text, labels, photographs, charts, PDFs, videos, audio files, documentation, directories, etc. The content strategist creates policies and manages the programs that house and govern content. This could include inventory, storage, workflow and governance of content (such as who has access to what type of content, who is responsible for updating or archiving content, who can delete or create new content).
A UX writer prepares written content for use in any number of media, including advertising, apps and websites, video/audio/animation, PR, etc. with a focus on maintaining a consistent user experience across all channels. This can include articles, product descriptions, documentation, headings, headlines, labels, microcopy, essentially anything that needs to be written in words.
Is the digital humanities an enduring movement or a trend?
I think it will endure. Academics need to create original research. Digital projects and analysis represents an exciting way to discover new things about subjects that otherwise seem to be studied to death. Applied to art, literature, history and other subjects in the humanities, digital projects open up a whole frontier of analytics and visualization where computational study used to be rare. This can take the form of text analysis, network diagramming, geographic information systems, 3D printing and even the creation of virtual worlds.
Where it can hit a road block is the fact that people who pursue humanities don’t often have the skills or competence required to utilize computational tools in their research. This isn’t their fault, it just happens to be rare in humanities curricula. That is why many universities are investing in developing IT and library staff who have these skills.
Ultimately, schools will include more and more digital studies electives in humanities programs. So like art and art history programs now may include chemistry and material science in units on art preservation, and English departments will have more an more computer scientists on hand to help with digital humanities projects.
THE OCCASIONAL MENTOR A monthly-ish column based on questions I’ve answered on Quora, heard on Slack groups, and other career advice I’ve given over the prior month. Hope you like it, but feel free to challenge me in the comments, if you have a different experience. Below are questions I answered in May.
Is it helpful to get a UX certificate or go to a UX conference as a starting point for a college undergraduate who wants to work on UX later but has no experience yet?
I am going on be the contrarian and say absolutely go to a conference or a meetup that is aligned with your UX interest. A certificate program will probably get you some basic skills, but so would reading books and working on pro bono projects on your own. (See one of my previous answers on certificates). For someone just starting out, it’s the interaction with other attendees as much as the talks and workshops that help build your knowledge of what and who you need to know to get a job in the field. And most conferences offer student discounts or lower-cost workshops so you don’t necessarily have to pay full price to get a benefit. Depending on where you live, Meetups can be plentiful and free or cheap. Online interest groups like Designers Guild on Facebook or UX Mastery on Slack are also good ways to find a community. UX Mastery even has a mentoring program.
Keep in mind that the most valuable UX design skills are soft skills like communication, presentation and ability to make insights. Design tools are always evolving so what you learn at a boot camp may not be marketable in a few years.
Some positive things about taking a certificate course. You meet your competition and potential future coworkers. A formal program may be confidence-building if you fear you don’t have basic understanding of what UX designers do and how they do it and aren’t comfortable picking up these skills on your own. But do some research. Not all certificates or boot camps have a good reputation. Meetups and other UX events are good places to ask about programs in your area.
Even better if your university offers design courses that you can take as part of your degree. Also, look for intro level cognitive psychology and ethnography courses (typically anthropology classes that cover interviewing skills). If your school has business or entrepreneur programs, ask if they offer any design or customer discovery workshops. Sometimes these programs are open to students schoolwide.
Registry Is Key Step in Growing Healthy and Secure Online Identity Ecosystem
Marketwired Identity Ecosystem Steering Group (IDESG)
Jun 6, 2016 8:00 AM
WASHINGTON, DC–(Marketwired – Jun 6, 2016) – The Identity Ecosystem Steering Group (IDESG) — an independent, non-profit organization dedicated to creating the future of trusted digital identities — today announced a new service that empowers organizations to improve the way they handle identities. The Identity Ecosystem Framework (IDEF) Registry brings the digital identity community closer to realizing the White House’s vision for trusted identities in cyberspace.
Every organization involved in online identity transactions plays a key role in creating and sustaining a healthy online identity ecosystem. The IDEF Registry allows companies to independently assess their own identity management methods against common industry practices. Using the IDESG’s Identity Ecosystem Framework as a model, organizations can now master and build on commonly accepted criteria for interoperability, privacy, security and usability. Meeting milestones in these subject areas is essential to ensuring that digital identities are protected and trustworthy online.
“This is an essential step in creating a safer environment for online transactions,” said Salvatore D’Agostino, President of the IDESG and CEO of IDmachines, LLC. “By equipping organizations involved in online transactions with a tool to measure where they stand relative to accepted policies and best practices, we’re promoting a safer internet on two levels. We’re making industry-accepted best practices more accessible to organizations who want to meet them, and providing a structured benchmark to organizations and individuals that want to use safer protocols for their digital transactions and information management.”
The Registry is an actionable step in the Identity Revolution, and the first opportunity of its kind for online identity service providers and owners and operators of applications that register, issue, authenticate, authorize and use identity credentials to prove that they operate secure platforms for their customers. Those that voluntarily register with the Registry publicly demonstrate their dedication to best practices in identity management. In addition to increasing participating organizations’ value and trust in the marketplace, the Registry gives them access to their industry’s most cutting-edge methods for identity management.
Initial listers include some of the preeminent companies in the identity space, such as MorphoTrust and PRIVO.
“As a founding member of the IDESG, PRIVO understands the level of commitment, subject matter expertise and vision required to bring the Registry to life,” said PRIVO Co-founder and CEO Denise Tayloe. “We are very proud to be one of the first services to hold ourselves accountable to the IDEF requirements that support a privacy-preserving, interoperable, secure, easy-to-use credential for families we serve, in order to protect and enable young users to engage and transact online.”
The IDESG has a pipeline of applicants and anticipates significant demand to join these early adopters to complete the process. Listing in the IDEF Registry is currently free for those who self attest.
“An Internet built around the identity principles of the Identity Ecosystem Framework, is in the best interest of us all as individuals,” said Mark DiFraia, Senior Director of Market Development at MorphoTrust. “MorphoTrust is proud to be one of the first organizations to join the IDEF Registry because we made the investment to build our online identity solution from the ground up to deliver on the National Strategy for Trusted Identities in Cyberspace (NSTIC) Principles. It is our sincere hope that the combination of NSTIC principles, the IDEF and now the IDEF Registry apply the right amount of pressure to shape the behavior of online players for the benefit of us all.”
For more information on The Identity Ecosystem Framework Registry, visit IDEFRegistry.org.
About the Identity Ecosystem Steering Group (IDESG)
IDESG is a voluntary, public-private partnership dedicated to developing an Identity Ecosystem Framework (IDEF) and services to better online digital identity. The IDESG looks to advance the Identity Ecosystem called for in the National Strategy for Trusted Identities in Cyberspace (NSTIC). The NSTIC, signed by President Obama in 2011, envisions the identity ecosystem as an online environment where individuals and organizations will be able to trust each other because they follow agreed-upon standards and policies to obtain and authenticate their digital identities. Come see how IDESG is making this happen by joining us in the effort and taking advantage of our services at IDESG.org.
Contact:
Media Contact
Donna Armstrong
ConnellyWorks, Inc.
571-323-2585 x6140
donna@connellyworks.com
As part of the project, my design team conducted interviews, surveys and user testing of current residents, neighbors and prospective residents. These were divided into four test groups including families with children, younger residents (couples and singles), those aged 55 and over and those considering a move to the neighborhood. The block association reported that “The respondents to the survey requested that the website be reorganized to find information more easily with an emphasis on portraying a ‘neighborhood feel.'” Indeed, my team found a great need among the test population for findable and accessible information.
My specific user test population included site users aged 55 and older. Among the requirements of this group, legibility and safety information was the most important need, and descriptive pictures of the block’s people and activities were especially appreciated.
The Information Architecture community also got a shout out:
“We are very lucky to have an information architect on our block to develop our ‘new and improved’ website that will be informative for all our residents.”
It is an honor to be recognized by the block for our contribution, and I thank my teammates for doing such a great job. We expect to launch the new site in June.
