In order to answer these and other pressing questions that affect public policy, researchers at Stanford University and Penn State University say that we need to move beyond measures of screen time, and record and analyze everything people see and do on their devices. The researchers call for a Human Screenome Project that will provide a detailed approach to observing the complexities of current digital lives.

In a commentary published online today in the journal Nature, the researchers contend that such a project is necessary to test the widely held assumptions that digital media are, at least partly, both responsible for and provide solutions to many of our most pressing health and social problems. Until now, what we see and do on our screens has been mostly invisible to researchers.

As a result, most studies have relied on participants to self-report their screen use, which is often highly inaccurate. And even when it is measured more accurately using software, total screen time or time spent on one app or website or another does not reflect what is new about digital interactions – switching quickly between radically different types of content and contexts. One user’s hour on Facebook keeping up with social posts from friends may have very little in common with a neighbor’s hour on Facebook keeping up with political news, or a co-worker’s hour on Facebook stalking people or bullying strangers.

The researchers argue that examining screen time alone is no longer sufficient because modern screen behaviors are too complex and varied. “The research has not kept up with the changes in technology,” said co-author Byron Reeves, who is the Paul C. Edwards Professor of Communication at Stanford’s School of Humanities and Sciences. “A lot of the research we have is incomplete, irrelevant or wrong because we don’t actually know what people are doing in these complex digital environments.”

“No matter what you study, whether it’s politics, addiction, health, relationships or climate action, if you really want to understand peoples’ beliefs and behaviors, you really need to look at their ‘screenome,’ because so much of our lives is now filtered through our digital devices,” said co-author Thomas Robinson, the Irving Schulman, MD Endowed Professor in Child Health and professor of pediatrics and of medicine at Stanford. “Many of the things we once did face-to-face are now reflected and recorded on our screens, whether it is banking or deciding what to eat or making friends or playing games or dating or exercising or discussing politics, and so on.”

Creating a map of digital life

To create a multidimensional map of people’s digital lives, the researchers are developing the field of Screenomics. They use software, installed on a person’s personal smartphone or other devices with their consent, that records, encrypts and transmits screenshots to a secure research site automatically and unobtrusively every five seconds whenever the device is turned on. This results in unique records of media use that can be analyzed and zoomed in on, to observe moment-by-moment changes across different content and screens, or zoomed out of, to describe longer-term changes over days, weeks or even years.

The Stanford Screenomics Lab has collected over 30 million data points from over 600 participants, to date, and has demonstrated that most people cycle through vast amounts of material very quickly, switching from one segment to another every 10 to 20 seconds. There is also good evidence that media use is highly idiosyncratic, and contains threads of experience that cut across radically different content that may make sense only to individual users. No two users’ screenomes look alike and even a single individual’s screenome appears unique from hour to hour, day to day, and week to week.

“We now have a way in which we can observe all of that movement and begin quantifying and studying it,” said commentary co-author Nilam Ram, professor of human development and psychology at Penn State University. “Now we’re in a position to be able to launch the Human Screenome Project as a large-scale, interdisciplinary effort that brings all that technology and domain experts together.”

Encouraging healthier screen use

The name ‘Human Screenome Project’ is a reference to the Human Genome Project and other federally funded ‘-ome’ projects that have produced large, sharable databases while maintaining the privacy of subjects. Reeves and his colleagues say they want to pursue a similar, open-source blueprint for a project that studies screen activity data while respecting user privacy.

“One of the reasons to have a Human Screenome Project is to scale up the amount of data that can be available to study how people are dealing with media,” Reeves said. “We have hundreds of people generating millions of screenshots. We need thousands and tens of thousands of people generating even more screen data.”

The researchers are optimistic that the project’s findings can be used to encourage healthier screen use. “It’s not just giving people information about what they’re doing, but actually building interventions around it,” Robinson said. “For example, if a person is struggling with becoming more physically active, we can identify their digital media use associated with periods of sedentary behavior and steer them, using precisely timed and personally tuned interventions, towards healthier outcomes.”