Study on SOtL program winner in student poster competition

The goals of the Consortium for Science Policy and Outcomes Science Outside the Lab program are lofty.

Held in Washington, D.C. each summer for the past ten years, the two-week program exposes graduate science and engineering students to policy analysts, lobbyists, business people, decision-makers and program managers – the key players in the science policy process – in the hopes that the students will come away from the experience with a better understanding of their role in the very complex world of research funding, science policy and societal progress.

The program aims to increase students’ appreciation of the big picture of scientific progress and to challenge their thoughts, values and the way they think about the world – in essence, to open their minds.

But, does the program work? How exactly do you measure whether a mind has been opened, and to what extent? ASU’s Michael Bernstein and co-researcher Kiera Reifschneider set out to answer that question.

Bernstein is a research associate with the Center for Nanotechnology in Society at ASU and doctoral candidate in the School of Sustainability. Reifschneider is a former post-doctoral researcher at the center and now a senior physical scientist with the US Government Accountability Office in Washington, D.C.

The team knew that a more traditional assessment, such as a pre- and post-test of information or a simple survey wouldn’t be enough.

“We’re not interested in regurgitated knowledge,” said Bernstein. “It’s more about do they have a deeper and more profound understanding of the complexities of the science policy space?”

To get at assessing this more holistic outcome, Bernstein and Reifschneider put together a suite of three assessments:

• The first, a concept map, had students – before and after going through the program – write down the people, organizations and factors involved in shaping science policy and the connections among them.

• The second, a burst reflection, had students write down the first five words that came to mind in response to program discussions and activities. The words were scored for their emotional content related to happiness, control and excitement.

• Finally, the students filled out a survey before and after the program about their views on the roles of scientists and technical experts in society and of information and values in science policy.

Bernstein presented results of this novel assessment approach at the 2015 American Association for the Advancement of Science (AAAS) student poster competition, where he won the social sciences category. The achievement was announced in the April 3 issue of Science.

Determining whether alternative education experiences, such as Science Outside the Lab, work to improve science and engineering graduate students’ understanding of the science policy process is critical to the Center for Nanotechnology in Society’s broader goal of improving outcomes of emerging technologies in society.

“There seems to be a profound disconnect between what policymakers and decision-makers need and what the scientific community provides,” said Reifschneider, who first noticed the lack of understanding between the two groups while finishing her doctorate in biochemistry at ASU and working with the Consortium for Science Policy and Outcomes. “This research gets right at the heart of the issue by examining the beliefs and values that are driving a lot of our scientists, and whether those could be part of the disconnect.”

Jameson Wetmore, associate director of engagement with the Center for Nanotechnology in Society and another co-author of the poster, is an instructor in the Science Outside the Lab program. He has long wanted to illustrate with data the radical changes that students go through.

“They end up with a much better appreciation of how important government officials are to the future of science and of the many sets of expertise and issues that need to be taken into account in the innovation process,” said Wetmore. “Each student takes away something different, and each of their minds is expanded in a different way.”

Preliminary results of the assessments are promising. For example, the students’ concept maps included a more complex web of interconnected nodes after going through the program.

“All participants at the beginning of the program knew that government played a role in science policy, but just called it ‘government,’” said Wetmore. “After the program, many listed a much larger number of government groups and agencies and their individual roles.”

For his dissertation, Bernstein is developing assessments for this and two other alternative training programs for scientists and engineers. He and his co-authors also hope to conduct a study that looks at longer-term outcomes of Science Outside the Lab.

“The real payoff comes when these scientists and engineers are at a company or university and decide to sit down with policymakers to help decide the best research direction to make sure their work is useful,” said Wetmore. “Ultimately, we want them to understand the breadth of the world and to consider their impact on it as they do their work.”