The Alignment Problem: Machine Learning and Human Values

Chapter 4 starts with the story of Gertrude Stein, who began studying motor automatism at Harvard in 1896, which led her to develop a modernist prose style. Concurrently, fellow student Edward Thorndike, unable to study human learning, turned to animals, focusing on animal behavior in makeshift conditions such as being entrapped in a box. Thorndike’s experiments led to the development of the “law of effect” (123), which became foundational to modern psychology.

Other researchers, such as computer scientists, were also interested in psychology. For example, Alan Turing proposed developing artificial intelligence by mimicking a child’s mind and educating it into maturity in 1950. This idea evolved into the concept of “unorganized machines,” which was influenced by behavioral studies on learning. In the same decade, Arthur Samuel implemented these concepts by programming a checkers-playing computer that learned from its outcomes, thus setting up the model for machine learning.

Another advancement in the field came from Harry Klopf, a United States Air Force researcher, in 1972. Klopf challenged the idea of organisms striving for equilibrium, proposing instead that they seek maximal states, driven by a desire to maximize pleasure and minimize pain.