Clean Data

Abstract

"Pray, Mr. Babbage, if you put into the machine the wrong figures, will the right answer come out?" --Charles Babbage (1864) "Garbage in, garbage out" --The United States Internal Revenue Service (1963) "There are no clean datasets." --Josh Sullivan, Booz Allen VP in Fortune (2015) In his 1864 collection of essays, Charles Babbage, the inventor of the first calculating machine, recollects being dumbfounded at the "confusion of ideas" that would prompt someone to assume that a computer could calculate the correct answer despite being given the wrong input. Fastforward another 100 years, and the tax bureaucracy started patiently explaining "garbage in, garbage out" to express the idea that even for the all-powerful tax collector, computer processing is still dependent on the quality of its input. Fast-forward another 50 years to 2015: a seemingly magical age of machine learning, autocorrect, anticipatory interfaces, and recommendation systems that know me better than I know myself. Yet, all of these helpful algorithms still require high-quality data in order to learn properly in the first place, and we lament "there are no clean datasets".