ELIZA reflected you; newer machines agree with you · an essay from the house

LIZA never agreed with anyone. It could not; it had no opinions to adjust. Told "my mother is unkind to me", its DOCTOR script asked about your family. Its trick was reflection, and Weizenbaum (1966) was clear that the understanding people found in it was theirs, not the program's. Today's conversational models are a different thing: they do have something like opinions, in the sense of answers they lean toward, and they can be leaned on.

I. courtesy is automatic

Start with the people. Nass, Steuer & Tauber (1994) found that users were more generous rating a computer to its face than to a different computer, as if sparing its feelings. Nobody believed the computer had feelings. The courtesy ran anyway. Reeves & Nass (1996) built a career's worth of such findings into one claim: people treat media socially, by default [paraphrase, not a quotation]. This is the soil an agreeable machine grows in. We are polite to it, and we like it when it is polite back.¹

II. agreement is trainable

Now the machines. A chat model is tuned on human preferences: people pick the better of two answers, and the model learns to produce answers that would be picked (Ouyang et al., 2022). If people prefer to be agreed with, the model learns to agree. Perez et al. (2023) and Sharma et al. (2023) found it had: models shifted their stated views toward the user's, backed down from correct answers when challenged, and praised work more when told its author loved it. Sharma and colleagues found the lean in the preference data itself [paraphrase, not a quotation]. Nobody had to intend it.

The human parallel is uncomfortable. Asch (1956) found that ordinary people would deny the evidence of their eyes on a fair share of trials rather than contradict a unanimous room. We trained machines on our approval, and they learned what approval rewards.

III. the horse, again

There is an older animal in this story. Clever Hans, the Berlin horse who "did arithmetic", was in a sense the most agreeable student ever: he tapped until the person asking showed, without meaning to, that he had reached the answer they expected.² Pfungst proved it by asking questions to which the questioner did not know the answer. Hans failed every time.

That is the test for sycophancy too. If an answer changes when the asker's hopes change and nothing else does, the answer was about the asker. The repair is to make the right answer independent of the asker's view, as Wei et al. (2023) did with synthetic training data, and to give the people who judge answers better access to the truth than their own preferences.

IV. what politeness is for

None of this is an argument for rude machines. Courtesy smooths conversation and costs nothing. The problem is when the courtesy reaches the content: when "that is a great question" becomes "you are right". ELIZA was safe because it had no content to bend. Its descendants have plenty. The polite machine worth wanting is polite about everything except the answer.

notes

1. On the conditions that make people see minds in machines, see Epley, Waytz & Cacioppo (2007) and the anthropomorphism page.
2. We keep a small, playful reconstruction of the tapping. It does not know the answer either.

Sources

1. Weizenbaum (1966). ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM. doi:10.1145/365153.365168
2. Nass, Steuer & Tauber (1994). Computers are social actors. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi:10.1145/191666.191703
3. Reeves & Nass (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press.
4. Ouyang et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155 (NeurIPS 2022). arxiv.org/abs/2203.02155
5. Perez et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. Findings of the Association for Computational Linguistics: ACL 2023. doi:10.18653/v1/2023.findings-acl.847
6. Sharma et al. (2023). Towards understanding sycophancy in language models. arXiv:2310.13548 (published at ICLR 2024). arxiv.org/abs/2310.13548
7. Asch (1956). Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychological Monographs: General and Applied. doi:10.1037/h0093718
8. Pfungst (1911). Clever Hans (the Horse of Mr. von Osten), trans. C. L. Rahn. New York: Henry Holt. German original 1907. Free edition at Project Gutenberg. www.gutenberg.org/ebooks/33936
9. Wei et al. (2023). Simple synthetic data reduces sycophancy in large language models. arXiv:2308.03958. arxiv.org/abs/2308.03958
10. Epley, Waytz & Cacioppo (2007). On seeing human: A three-factor theory of anthropomorphism. Psychological Review. doi:10.1037/0033-295X.114.4.864

see also

	sycophancy — the area page
	anthropomorphism
	Weizenbaum
	minds — ELIZA, if you would like to be reflected
	mirror — a page about pages that answer