Our ability to collect and access large quantities of data over the last decade has been revolutionary for many social sciences. Suddenly, it is possible to measure human behavior, performance, and activity on an unprecedented scale, opening the door to fundamental advances in discovery and understanding. Yet such access to data has limitations that, if not sufficiently addressed and explored, can result in significant oversights. Here we discuss recent research that used data from a large global sample of high school students to demonstrate, paradoxically, that in nations with higher gender equality, fewer women pursued science, technology, engineering, and mathematics (STEM) degrees than would be expected based on aptitude in those subjects. The reasons for observed patterns is central to current debates, with frequent disagreement about the nature and magnitude of problems posed by the lack of female representation in STEM and the best ways to deal with them. In our international efforts to use big data in education research, it is necessary to critically consider its limitations and biases.