Quick Review: Computer Power and Human Reason
Weizenbaum, Joseph. Computer Power and Human Reason. W. H. Freeman and Company, 1976.
Joseph Weizenbaum, the inventor and developer of ELIZA, tells us his purpose in writing this book is:
There are then, two questions that must ultimately be confronted. First, are the conceptual bases that underlie linguistic understanding entirely formalizable, even in principle, as Schank suggests and as most workers in Al believe? Second, are there ideas that, as I suggested, “no machines will ever understand because they relate to objectives that are inappropriate for machines”? page 197
Are there conceptual things computers cannot do?
According to Weizenbaum, yes. Science, and hence computers, simplify everything to a problem that can be described in formal terms.
Science can proceed only by simplifying reality. The first step in its process of simplification is abstraction. And abstraction means leaving out of account all those empirical data which do not fit the particular conceptual framework within which science at the moment happens to be working, which, in other words, are not illuminated by the light of the particular lamp under which science happens to be looking for keys. page 127
For a computer to solve a problem, then, it must be simplified to its most essential elements. In the act of making computers “act like humans,” humans are reduced to a single thing: their intelligence.
The trouble with I.Q. testing is not that it is entirely spurious, but that it is incomplete. It measures certain intellectual abilities that large, politically dominant segments of western European societies have elevated to the very stuff of human worth and hence to the sine qua non of success. It is incomplete in two ways: first, in that it fails to take into account that human creativity depends not only on intellect but also crucially on an interplay between intellect and other modalities of thought, such as intuition and wisdom; second, in that it characterizes intelligence as a linearly measurable phenomenon that exists independent of any frame of reference. page 204
Further, computers are simple number processing devices. Although Weizenbaum does not use this illustration, they are like the Chinese Room; computers process, but is is structurally impossible for them to understand.
Are there things we should not ask computers to do?
Again, according to Weizenbaum, yes. Because computers are “scientific,” and can seemingly answer complex questions with a high degree of confidence, humans tend to trust computers. In the case of ELIZA, users even started asking for “private time” with the software, building emotional attachments, and considered a violation of their privacy if anyone could see their interactions with ELIZA. The user shapes themselves to the software over time, reducing themselves to a set of problems the software knows how to solve.
The most superficial aspects of these systems’ simplicity—as reflected by their simplistic construction of their subject matters—are immediately visible. Simon, for example, sees man as “quite simple.” The “apparent” complexity of his behavior is due to the complexity of his environment. In any event, he can be simulated by a system sensitive to only “a few simple parameters,” one that consists of only a few (certainly many fewer than, say, ten thousand) “elementary information processes.” page 248
This tendency to simplify man poses an existential risk to humanity.
Germany implemented the “final solution” of its “Jewish Problem” as a textbook exercise in instrumental reasoning. Humanity briefly shuddered when it could no longer avert its gaze from what had happened, when the photographs taken by the killers themselves began to circulate, and when the pitiful survivors re-emerged into the light. But in the end, it made no difference. The same logic, the same cold and ruthless application of calculating reason, slaughtered at least as many people during the next twenty years as had fallen victim to the technicians of the thousand-year Reich. We have learned nothing. Civilization is as imperiled today as it was then. page 256
Relying on computers also cripples decision making.
Our society’s growing reliance on computer systems that were initially intended to “help” people make analyses and decisions, but which have long since both surpassed the understanding of their users and become indispensable to them, is a very serious development. page 236
Because no one knows how the machine really works, no one really knows how decisions are being made. Not only do humans trust the computer to make decisions, they do not even know how to “change the computer’s mind” when it makes a clearly unjust, unfair, or even dangerous decision.
Further, the reduction of the person to something that obeys the machine places the person in the hands of the programmers–making the programmer, effectively, omnipotent.
One would have to be astonished if Lord Acton’s observation that power corrupts were not to apply in an environment in which omnipotence is so easily achievable. It does apply. And the corruption evoked by the computer programmer*s omnipotence manifests itself in a form that is instructive in a domain far larger that the immediate environment of the computer. page 115
Final Thoughts
It is surprising how well this book has held up to the test of time. Weizenbaum’s explanation of how computers work is easy to understand, his concerns are real, and he expresses his concerns in a way modern readers will understand. This book is well worth reading if you are interested in artificial intelligence, its impact on society, and its impact on people.