Welcome to TALL!
Teaching the hard stuff
Learn more about our lab

People

Meet the people in our lab.

Picture of Jim Stigler

Jim Stigler

stigler@ucla.edu
Jim Stigler is professor of psychology at UCLA. His research interests include teaching and learning in complex domains, the impacts of culture and technology on learning, and applications of improvement science to education.

Picture of Karen Givvin

Karen Givvin

kbgivvin@g.ucla.edu
Karen Givvin is a researcher and adjunct professor of psychology at UCLA. Her research interests center on teaching, learning, and motivation in achievement domains.

Picture of Ji Son

Ji Son

jiyunson@gmail.com
Ji Son is professor of psychology at Cal State LA. She is interested in how basic cognitive and perceptual processes foster rich and transferable learning.


Picture of Adam Blake

Adam Blake

adamblake@ucla.edu
Adam Blake is CourseKata's director of technology . He is interested in the factors that influence students' judgments, decisions, and strategies related to learning, and how instruction and learning materials can improve these behaviors.

Picture of Claudia Sutter

Claudia Sutter

claudia@coursekata.org
Claudia C. Sutter is CourseKata's director of research. She is interested in understanding how students’ motivational beliefs affect their learning behaviors and outcomes and how learning contexts shape students’ opportunities to reach their full potential.

Picture of Caylor Davis

Caylor Davis

crdavis25@ucla.edu
Caylor Davis is CourseKata's director of implementation. She is interested in research and course design that makes math more accessible for non-STEM students.

Picture of Alice Xu

Alice Xu

alicex@g.ucla.edu
Alice Xu is a graduate student in Developmental Psychology at UCLA. She is interested in individual differences during learning, STEM learning in the college population, and children's cognitive and language development.

Picture of Asia Mullins

Asia Mullins


Asia Mullins is a graduate student at Cal State LA. She is interested in developing learning techniques that enhance student comprehension of data visualizations, as well as encourage student enthusiasm for statistics.

Picture of Icy (Yunyi) Zhang

Icy (Yunyi) Zhang

yunyi9847@g.ucla.edu
Icy Zhang is a graduate student in Developmental Psychology at UCLA. She is interested in the role of embodied learning in supporting students’ understanding of difficult content such as statistics and applying these theory-based interventions to real classrooms

Picture of Ben Winjum

Ben Winjum

bwinjum@ucla.edu
Ben Winjum is a staff member in the High Performance Computing group of UCLA’s Institute for Digital Research and Education, where he works on advanced computing and data analytics projects that support both education and research.

Alumni

Picture of Stacy Shaw

Stacy Shaw

stacy.tamsen.shaw@gmail.com
Stacy Shaw graduated from UCLA in 2020 and is now an Assistant Professor at Worcester Polytechnic Institute in the areas of Learning Science and Technology and Psychological and Cognitive Science. Her research focuses on learning in mathematics and statistics with a focus on flexible thinking and creativity.

Picture of Emma Geller

Emma Geller

egeller@ucsd.edu
Emma Geller is an assistant teaching professor at UC San Diego. She is interested in applications of cognitive psychology to support deep learning and understanding in multimedia environments, especially regarding the design of online video learning.

Picture of Mary Tucker

Mary Tucker

maryctucker@ucla.edu
Mary Tucker graduated from UCLA in 2022. Her research interests include the psychological and contextual factors that influence teaching and learning, the design of learning environments in STEM domains, and the application of technology and data analytics to understand and improve teaching and learning.

Picture of Laura Fries

Laura Fries

lauracfries@gmail.com
Laura Fries is a project scientist at UCLA. She is interested in the cognitive and affective factors that influence learning, and instructional strategies to promote transferable understanding and retention of complex domains.

Papers

Dig into what we've been working on.



How does expectancy-value-cost motivation vary during a semester? An intensive longitudinal study to explore individual and situational sources of variation in statistics motivation

Claudia C. Sutter, Delaram A. Totonchi, Jamie DeCoster, Kenneth E. Barron & Chris S. Hulleman (2024).

We measured expectancy, value, and cost 10 times over a 10-week introductory statistics course (N = 219) to examine their overall trajectory as well as individual (between-student) differences and situational (within-student) variability. First, our findings revealed an initial decline in expectancy and value and an initial increase in cost. Second, expectancy, utility value, and cost demonstrated individual and situational variability of comparable size, while intrinsic value had higher individual variability. Third, individual and situational variability in expectancy and value predicted variability in performance. Lastly, the relation of situational variability in expectancy and utility value with performance was stronger for Black, Latinx, and other racially marginalized students than for White/Asian students. Our findings provide empirical evidence for the situational nature of motivational beliefs and have implications for practitioners, course curriculum designers, and policymakers who aim to create more supportive and motivation-enhancing environments, particularly for statistics courses and students from racially marginalized and underserved backgrounds. Download a copy.

Memorization and Performance During Pandemic Remote Instruction - Evidence of Shifts from an Interactive Textbook

Jose L. Salas, Xinran (Wendy) Wang, Mary C. Tucker, & Ji. Y. Son (2024).

Students believe mathematics is best learned by memorization; however, endorsing memorization as a study strategy is associated with a decrease in learning (Schoenfeld, 1989). When the world changed with the onset of the COVID-19 global pandemic, instruction transitioned to fully remote instruction where many assignments and examinations became open textbook, open note, and even open Internet. In this new world, did students change their beliefs about the role of memorization in learning? Did academic performance change? And did the relationship between memorization beliefs and academic performance change? The current study takes advantage of data collected in an online interactive statistics textbook used by courses before (in-person) and after (remote) the declaration of the COVID-19 pandemic at three institutions, each representing a part of the California Master Plan for Higher Education (e.g., University of California, California State University, and California Community Colleges). Results from 2668 students who used the textbook showed that the UC institution had lower memorization belief scores compared to both the CSU and CCC institutions. Even when controlling for institution and chapter of the textbook, lower memorization belief scores were related to higher performance. Surprisingly, there were no significant differences in either memorization beliefs nor performance before and after transitioning to online remote instruction due to the pandemic. Although much of educational research is conducted in one institution, this kind of research can identify differences across institutional contexts to understand how learning can be affected by different teaching formats, including in-person and online/distance, brought on by disruptive social changes such as a global pandemic. Download a copy.

The “Better Book” Approach to Addressing Equity in Statistics : Centering the Motivational Experiences of Students from Racially Marginalized Backgrounds for Widespread Benefit

Claudia C. Sutter, Matthew C. Jackson, Karen B. Givvin, James W. Stigler & Ji Y. Son (2024).

Although improving racial equity in critical college courses such as introductory statistics is a laudable goal, making research-based progress toward that aim in a scalable manner remains a chal- lenge. To translate psychological insights to benefit racially marginalized students, we implemented the “Better Book” approach, where instructors, researchers, and developers work together to improve an online textbook used in introductory statistics. The “Better Book” approach to equity assumes that racially marginalized students are a “canary in the coal mine”, alerting us to systemic issues that can affect a broader array of students. We started by finding places in the textbook where racially marginalized students reported higher perceptions of costs (the effort and time required to learn the content) than non-marginalized students. Then we drew upon suggestions from users to redesign the textbook where gaps in cost perceptions peaked. We then analyzed data from both the original and redesigned versions of the textbook to evaluate the impact on students who were subsequently enrolled in the course. Results showed that perceptions of cost were dramatically reduced in the experience of racially marginalized students but also the redesign resulted in an improved experience for all students. Download a copy.

Watching videos of a drawing hand improves students’ understanding of the normal probability distribution

Icy (Yunyi) Zhang, Xiaohan Hanna Guo, Ji Y. Son, Idan A. Blank & James W. Stigler (2024).

Understanding normal probability distributions is a crucial objective in mathematics and statistics education. Drawing upon cognitive psychology research, this study explores the use of drawings and visualizations as effective scaffolds to enhance students' comprehension in a sequence of three studies. Download a copy.

Embodying Similarity and Difference - The Effect of Listing and Contrasting Gestures During U.S. Political Speech

Icy (Yunyi) Zhang, Tina Izad, Erica A. Cartmill (2024).

Study 1 examined two rhetorical devices—contrasts and lists—used by three politicians during U.S. presidential debates and asked whether the gestures produced during contrasts and lists differ. Gestures produced during contrasts were more likely to involve changes in hand location, and gestures produced during lists were more likely to involve changes in trajectory. Study 2 used footage from the same debates in an experiment to ask whether gesture influenced the way people perceived the politicians’ arguments. Download a copy.

Prediction versus Production for Teaching Computer Programming

Mary C. Tucker, Xinran (Wendy) Wang, Ji Y. Son, & James W. Stigler (2024).

Background: Most students struggle when learning to program. Aims: In this paper we examine two instructional tasks that can be used to introduce programming: tell-and- practice (the typical pedagogical routine of describing some code or function then having students write code to practice what they have learned) and prediction (where students are given code and asked to make predictions about the output before they are told how the code works). Sample: Participants were 121 college students with no coding experience. Methods: Participants were randomly assigned to one of two parallel training tasks: predict, or tell-and-practice. Results: Participants in the predict condition showed greater learning and better non-cognitive outcomes than those in the tell-and-practice condition. Conclusions: These findings raise a number of questions about the relationship between programming tasks and students’ experiences and outcomes in the early stages of learning programming. They also suggest some pedagogical changes to consider, especially in early introductions to programming. Download a copy.

Latine Students’ Motivational and Emotional Experiences Related to Their Introductory Statistics Course - Differences by Institution Type Necessitate Tailored Interventions

Claudia C. Sutter, Karen B. Givvin, Paige L. Solomon, & Ana Leandro-Ramos (2024).

The present study developed a representation-mapping intervention designed to help students interpret, coordinate, and eventually translate across multiple representations. We integrated the intervention into an online textbook being used in a college course, allowing us to study its impact in a real course over an extended period of time. The findings of this study support the efficacy of the representation-mapping intervention for facilitating learning and shed light on how to implement and refine such interventions in authentic learning contexts. Download a copy.

Representational-Mapping Strategies Improve Learning From an Online Statistics Textbook

Icy (Yunyi) Zhang, Maureen E. Gray, Alicia (Xiaoxuan) Cheng, Ji Y. Son, & James W. Stigler (2023).

The present study developed a representation-mapping intervention designed to help students interpret, coordinate, and eventually translate across multiple representations. We integrated the intervention into an online textbook being used in a college course, allowing us to study its impact in a real course over an extended period of time. The findings of this study support the efficacy of the representation-mapping intervention for facilitating learning and shed light on how to implement and refine such interventions in authentic learning contexts. Download a copy.

Listening for Common Ground in High School and Early Collegiate Mathematics

Gail Burrill, Henry Cohn, Yvonne Lai, Dev P. Sinha, Ji Y. Son, & Katherine F. Stevenson (2023).

Solutions to pressing and complex social challenges require that we reach for common ground. Only through cooperation among people with a broad range of backgrounds and expertise can progress be made on issues as challenging as improving student success in mathematics. In this spirit, the AMS Committee on Education held a forum in May 2022 entitled The Evolving Curriculum in High School and Early Undergraduate Mathematical Sciences Education. This article is a report on that forum by the authors listed above, who were among the organizers and presenters. Download a copy.

Don't force a false choice between algebra and data science

Ji Y. Son & James W. Stigler (2022).

Download a copy.

The Better Book Approach for Education Research and Development

James W. Stigler, Ji Y. Son, Karen B. Givvin, Adam Blake, Laura Fries, Stacy T. Shaw, & Mary C. Tucker (2019).

This paper describes a new approach for education research and development - the better book approach - and reports on our initial development and application of the approach in the context of introductory college-level statistics. Download a copy.

Lower Socioeconomic Status is Related to Poorer Emotional Well-Being Prior to Academic Exams

Danny Rahal, Stacy T. Shaw, James W. Stigler (2023).

People of lower social status tend to have greater emotional responses to stress. This study assessed whether lower social status was related to greater emotional responses in anticipation of a naturalistic stressor: academic exams among college students. As hypothesized, multilevel models (ratings nested within participants) predicting emotion indicated that students with lower mother’s education had less positive emotion, more depressive emotion, and more anxious emotion the day prior to academic exams than students with higher mother’s education (proportional reductions in variance [PRV] = .013–.020). Specifically, lower mother’s education was associated with poorer well-being before but not after the exam. Exploratory models revealed that differences in emotion by mother’s education were strongest for students with lower exam scores (PRV = .030–.040). Download a copy.

Teaching Statistics and Data Analysis with R

Mary C. Tucker, Stacy T. Shaw, Ji Y. Son & James W. Stigler (2022).

In this exploratory study, we characterize the attitudes and experiences of 672 undergraduate students as they used our online textbook as part of a 10-week introductory course in statistics. Students expressed negative attitudes and concerns related to R at the beginning of the course, but most developed more positive attitudes after engaging with course materials, regardless of demographic characteristics or prior programming experience. Analysis of a subgroup of students revealed that change in attitudes toward Rmay be linked to students’ patterns of engagement over time and students’ perceptions of the learning environment. Download a copy.

Student Concerns and Perceived Challengesin Introductory Statistics, How the Frequency Shifted during COVID-19, and How They Differ bySubgroups of Students

Claudia C. Sutter, Karen B. Givvin, Mary C. Tucker, Kathryn A. Givvin, Ana Leandro-Ramos & Paige L. Solomon (2022).

We explored how the frequency of concerns changed with the onset of COVID-19(N=1417) and, during COVID-19, how incoming concerns compared to later perceived challenges(n=524).Students were most concerned about R coding, understanding concepts, workload, prior knowledge,time management, and performance, with each of these concerns mentioned less frequently during thanbefore COVID-19. Concerns most directly related to the pandemic—virtual learning and inaccessibility of resources—showed an increase in frequency. The frequency of concerns differed by gender and URM status.The most frequently mentioned challenges were course workload, virtual learning, R coding, and under-standing concepts, with significant differences by URM status. Concerns about understanding concepts,lack of prior knowledge, performance, and time management declined from the beginning to the end of theterm. Workload had the highest rate of both consistency and emergence across the term. Download a copy.

Watching a Hands-on Activity Improves Students’ Understanding of Randomness

Icy (Yunyi) Zhang, Mary Tucker, James W. Stigler (2022).

Given the growing interest in using statistical programming languages like R as pedagogical tools, the findings of this study provide important and encouraging insights into the use of hands-on demonstrations to complement computer simulation in remote teaching. It validates the importance of giving students some hands-on exposure to the simulation processes prior to the compu- tational simulation we want them to understand and makes it clear that at least some of the benefits of embodied activities can be retained even if students are not performing the hands-on activities themselves. Download a copy.

Reasoning Affordances with Tables and Bar Charts

Cindy Xiong, Elsie Lee-Robbins, Icy Zhang, Aimen Gaba, Steven Franconeri (2022).

We tested whether confirmation bias exists when people reason with visualized data and whether certain visualization designs can elicit less biased reasoning strategies. We asked crowdworkers to solve reasoning problems that had the potential to evoke both poor reasoning strategies and confirmation bias. Presenting the data in a table format helped participants reason with the correct ratio strategy while showing the data as a bar table or a bar chart led participants towards incorrect heuristics. Confirmation bias was not significantly present when beliefs were primed, but it was present when beliefs were pre-existing. Additionally, the table presentation format was more likely to afford the ratio reasoning strategy, and the use of ratio strategy was more likely to lead to the correct answer. Download a copy.

Effect of Feedback with Video-based Peer Modeling on Learning and Self-efficacy

Wadi Eghterafi, Mary C. Tucker, Icy (Yunyi) Zhang, Ji Y. Son (2022).

In this study, we examined the effect of video-based feedback designed to highlight a peer engaging in effective thinking processes on self-efficacy beliefs and learning outcomes (performance on a delayed quiz). Results indicated that students who watched the mastery videos, but not the coping videos, rated their self-efficacy higher and scored higher on a class quiz taken more than a day after the feedback intervention than students who viewed a worked example. Download a copy.

Modeling First Applying Learning Science to the Teaching of Introductory Statistics

Ji Y. Son, Adam Blake, Laura Fries, James W. Stigler (2021).

In this article we describe our attempt to apply theories and findings from learning science to the design of a statistics course that aims to help students build a coherent and interconnected representation of the domain. The resulting practicing connections approach provides students with repeated opportunities to practice connections between core concepts (especially the concepts of statistical model, distribution, and randomness), key representations (R programming language and computational techniques such as simulation and bootstrapping), and real-world situations statisticians face as they explore variation, model variation, and evaluate and compare statistical models. We provide a guided tour through our curriculum implemented in an interactive online textbook (CourseKata.org) and then provide some evidence that students who complete the course are able to transfer what they have learned to the learning of new statistical techniques. Download a copy.

"Utility Value Trajectories and their Relationship with Behavioral Engagement and Performance in Introductory Statistics"

Claudia C. Sutter, Chris S. Hulleman, Karen B. Givvin, Mary Tucker (2021).

This study examined utility value trajectories overall and by gender, race, and underrepresented racial minority (URM) status within an introductory statistics course and tested the relationships between utility value, behavioral engagement, and performance. Download a copy.

"Instructed Hand Movements Affect Students’ Learning of an Abstract Concept from Video"

Icy (Yunyi) Zhang, Karen B. Givvin, Jeffrey M. Sipple, Ji Y. Son, James W. Stigler (2021).

The two studies reported here investigate the impact of instructed hand movements on students’ subsequent understanding of a concept. Students were asked to watch an instructional video—focused on the concept of statistical model—three times. These two studies found that instructed hand movement—even when presented as an unrelated, secondary task—can affect students’ learning of a complex concept. This article was featured in Scientific American. Download a copy.

"Practicing Connections:A Framework to Guide Instructional Design for Developing Understanding in Complex Domains"

Laura Fries, Ji Y. Son, Karen B. Givvin, James W. Stigler (2020).

Research suggests that expert understanding is characterized by coherent mental representations featuring a high level of connectedness. This paper advances the idea that educators can facilitate this level of understanding in students through the practicing connections framework: a practical framework to guide instructional design for developing deep understanding and transferable knowledge in complex academic domains. Download a copy.

Removing Opportunities to Calculate Improves Students’ Performance on Subsequent Word Problems

Karen B. Givvin, Veronika Moroz, William Loftus, & James W. Stigler (2019).

This paper reports our investigation on whether removing opportunities to calculate could improve students’ subsequent ability to solve similar word problems Download a copy.

Exploring the Practicing-connections Hypothesis:Using Gesture to Support Coordination of Ideas in Understanding a Complex Statistical Concept"

Ji Y. Son , Priscilla Ramos , Melissa DeWolf , William Loftus and James W. Stigler (2018).

This paper presented a framework and approach for studying how students come to understand complex concepts in rich domains. Specifically, it explores the role that a teacher’s gesture might play in supporting students’ coordination of two concepts central to understanding in the domain of statistics: mean and standard deviation. Download a copy.

Expertise and Expert Performance in Teaching

James W. Stigler and Kevin F. Miller (2018).

This chapter tries to take a broader approach to understanding the nature and development of expertise and expert performance in teaching. The paper also tries to integrate a number of ideas and findings from literatures as diverse as cross-cultural compar- isons of teaching, cognitive psychology, and systems improvement, among others. Download a copy.

DOES VAM + MET = IMPROVED TEACHING?

James W. Stigler, James Hiebert, and Karen B. Givvin (2018).

The paper discusses the logic of the more traditional approach on which many current policies for improving teaching in the United States are based and then presents an alternative research approach, in which a different theory of improvement is assumed. Download a copy.

Online Learning as a Wind Tunnel for Improving Teaching

James W. Stigler, Karen B. Givvin (2017).

The chapter proposes an approach that combines the affordances of online learning with the methodologies of systems improvement. It discusses how online learning might be a wind tunnel for the study and improvement of teaching. Download a copy.

"What Community College Developmental Mathematics Students Understand about Mathematics, Part2:The Interviews"

Karen B. Givvin, James W. Stigler, and Belinda J. Thompson (2011).

Following the prior paper, this article presents findings from one-on-one interviews with a sample of community college developmental math students. These interviews were designed to further probe students’ mathematical thinking, both correct and incorrect. Download a copy.

What Community College Developmental Mathematics Students Understand about Mathematics

James W. Stigler, Karen B. Givvin, and Belinda J. Thompson (2010).

This paper investigates what community college students actually understand about the mathematics that underlie the topics they’ve been taught and seeks evidence that students used reasoning in answering mathematical questions. Download a copy.

"Improving Social and Conceptual Connections during Remote Statistics Classes [conference paper]"

Caylor R. Davis, Karen Givvin, Jinna Hwang, and Ji Y. Son (2021).

This paper reports on the implementation of a statistics lesson in a remote classroom that incorporates an embodied group activity as a way to improve perceptions of social connection and to enrich conceptual connections during learning. Download a copy.

How Much Students Value an Introductory Statistics Course, How Value Levels Change across the Term, and How They Predict Learning [conference paper]

Claudia C. Sutter, Mary Tucker, Karen Givvin, and Chris S. Hulleman (2021).

This paper examines changes in utility value in an introductory statistics course overall, by sex and underrepresented racial minority (URM) and tested the relationship between utility value and learning. Download a copy.

Projects

Learn about our active projects.






The Better Book Project:

We are developing a new approach to education R&D in which researchers, designers / developers, and practitioners work together to build and continuously improve online instructional materials. We leverage theories and findings from the learning sciences, translational research, affordances of online learning, and the methodologies of improvement science. (Funded by Chan Zuckerberg Initiative DAF of Silicon Valley Foundation.)

- Some early thoughts on the Better Book approach are found in Stigler & Givvin, Online learning as a wind tunnel for improving teaching.

- An overview of the approach and our CourseKata web R&D platform: Stigler et al., The Better Book approach to education research and development.

We are building a web platform to support our Better Book R&D approach (CourseKata.org). The CourseKata platform takes incremental improvements in content (stored as markdown files); transcodes them into HTML pages; and renders the content via LTI in multiple learning management systems (starting with Canvas). It also supports the R&D process, including randomized experiments at the individual student level within classes. You can also know more about the project by listening to the podcast, reading this article or watching our interview video about our new approach of teaching statistics.

CourseKata Statistics for College:

Following a “learn by doing” strategy to developing the Better Book approach, our initial focus is on introductory college-level statistics. Our innovative statistics and data science textbook includes more than 1200 formative assessments and R coding exercises. The book and resulting data are designed to implement, test, and improve our practicing connections theory of transferable learning. (Funded by Chan Zuckerberg Initiative DAF of Silicon Valley Foundation.)

We are scaling CourseKata Statistics into all levels of the California higher education system: community college (starting with Pierce College); California State University (starting with the Los Angeles campus); and the University of California (starting with UCLA). (Funded by California Learning Lab.)

- We have also developed a year-long high school version of the course, with a complete set of classroom lesson plans (funded by the Chan Zuckerberg Foundation).

- Read about our practicing connections hypothesis. Fries, et al...

CourseKata Statistics and Data Science for High School:

In our newest project we are adapting our innovative statistics program for high schools. Our target audience, initially, are the “Students Previously Known as Non-STEM”: students currently taking no mathematics at all in the twelfth grade. We are aiming to build skills and understanding, while at the same time changing their self-perceptions of their own capacity for STEM-related majors and careers.( Funded by Chan Zuckerberg Initiative, and by the Schusterman Family Foundation.)





Our Amazing Funders

Who make what we do possible.

Chan Zuckerberg Initiative DAF

California Governor’s Office of Planning and Research

Schusterman Foundation



We’re deeply grateful for the support of the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation (DRL-1229004), the California Governor’s Office of Planning and Research (contract OPR18115), and the Schusterman Foundation.

Contact Us

Please let us know if you have any questions by emailing us at: ucla.tall@gmail.com