Data Police: Behavioural Science Under the Microscope

The Rise of Data Colada and the Fall of Academic Stars

Dr. Gordon Wright
g.wright@gold.ac.uk

Mon 03 Feb, 2025

Overview

  • The Replication Crisis in Behaviouural Science
  • Enter the Data Colada Team
  • Case Study 1: Francesca Gino
  • Case Study 2: Dan Ariely
  • Implications for Scientific Integrity
  • The Future of Behavioural Science

The Replication Crisis

The Problem

  • Many famous findings couldn’t be replicated
  • Small sample sizes
  • P-hacking concerns
  • Publication bias

The Impact

  • Loss of confidence in findings
  • Questions about methodology
  • Need for better oversight
  • Call for reform

Enter Data Colada

The Team

  • Uri Simonsohn (UPenn)
  • Joseph Simmons (UPenn)
  • Leif Nelson (UC Berkeley)

Legends

Their Mission

  • Investigate statistical anomalies
  • Promote research integrity
  • Challenge questionable findings
  • Foster methodological rigor

Their Methods

Statistical Analysis

  • Examination of raw data
  • Analysis of digital footprints
  • Pattern recognition
  • Collaboration with anonymous sources

Publication Strategy

  • Public blog posts
  • Detailed technical analyses
  • Open access to findings
  • Engagement with academic community

Case Study: Francesca Gino

Case Study: Francesca Gino

Background

  • Harvard Business School Professor
  • Expert in Behavioural ethics
  • Study of dishonesty
  • Rising academic star

The Investigation

  • Four papers under scrutiny
  • Excel file anomalies
  • Data manipulation evidence
  • Pattern of questionable results

Gino’s Key Papers in Question

The Studies

  1. Sign-at-top effect on honesty

  2. Cleansing products and authenticity

  3. Cheating Behaviour patterns

  4. Decision-making and morality

The Problems

The Problems

  • Data point manipulation
  • Suspicious response patterns
  • Excel timestamp issues
  • Statistical improbabilities

The Harvard Response

Actions Taken

  • Unpaid administrative leave
  • Move to revoke tenure
  • Investigation launched
  • Journal retractions requested

Gino’s Response

  • $25M lawsuit
  • Claims of discrimination
  • Procedural complaints
  • Denial of allegations

Dan Ariely Connection

Background

  • Bestselling author
  • TED Talk star
  • Behavioural economics pioneer
  • Frequent Gino collaborator

The Issues

  • Questioned data in joint work
  • Similar pattern allegations
  • Ten Commandments study concerns
  • Replication failures

Broader Implications

For Academia

  • Research integrity focus
  • Methodology scrutiny
  • Data transparency needs
  • Replication importance

For Students

  • Critical thinking emphasis
  • Methods verification
  • Ethics awareness
  • Data handling skills

The Role of Watchdogs

Benefits

  • Quality control
  • Error detection
  • Integrity promotion
  • Methodology improvement

Challenges

  • Power dynamics
  • Due process concerns
  • Career impacts
  • Field reputation

Future of Behavioural Science

Changes Needed

  • Better verification methods
  • Stronger oversight
  • Data transparency
  • Replication emphasis

Opportunities

  • Improved methods
  • Better quality control
  • More reliable findings
  • Restored confidence

Discussion Questions

  1. How can we prevent research fraud?
  2. What role should blog-based criticism play?
  3. How do we balance innovation and rigor?
  4. What are the implications for peer review?

Key Takeaways

  1. Data verification is crucial
  2. External oversight matters
  3. Methods matter as much as results
  4. Transparency is essential
  5. Replication is vital

Further Reading

  • Data Colada Blog (datacolada.org)
  • “Predictably Irrational” by Dan Ariely
  • Harvard Business School case studies
  • Academic integrity guidelines
  • Statistical methods resources

Research Methods Lecture 14 - Naughtiness