Wizard Of Oz Experiment
   HOME

TheInfoList



OR:

In the field of
human–computer interaction Human–computer interaction (HCI) is the process through which people operate and engage with computer systems. Research in HCI covers the design and the use of computer technology, which focuses on the interfaces between people (users) and comp ...
, a Wizard of Oz experiment is a
research Research is creative and systematic work undertaken to increase the stock of knowledge. It involves the collection, organization, and analysis of evidence to increase understanding of a topic, characterized by a particular attentiveness to ...
experiment in which subjects interact with a
computer A computer is a machine that can be Computer programming, programmed to automatically Execution (computing), carry out sequences of arithmetic or logical operations (''computation''). Modern digital electronic computers can perform generic set ...
system that subjects believe to be autonomous, but that is actually operated or partially operated by an unseen human being.


Concept

The phrase ''Wizard of Oz'' (originally ''OZ Paradigm'') has come into common usage in the fields of
experimental psychology Experimental psychology is the work done by those who apply Experiment, experimental methods to psychological study and the underlying processes. Experimental psychologists employ Research participant, human participants and Animal testing, anim ...
, human factors,
ergonomics Ergonomics, also known as human factors or human factors engineering (HFE), is the application of Psychology, psychological and Physiology, physiological principles to the engineering and design of products, processes, and systems. Primary goa ...
,
linguistics Linguistics is the scientific study of language. The areas of linguistic analysis are syntax (rules governing the structure of sentences), semantics (meaning), Morphology (linguistics), morphology (structure of words), phonetics (speech sounds ...
, and usability engineering to describe a specific type of testing or iterative design. In such an experiment, a laboratory experimenter (the "wizard") simulates the behavior of a theoretical intelligent computer application, often by going into another room and intercepting all communications between participant and system. Sometimes this is done without the participant's prior knowledge, to manage the participant's expectations and encourage natural behaviors, while at other times the participant is aware. For example, a test participant may think that he is communicating with a computer using a speech interface, when the wizard is actually covertly entering the participant's words into the computer, enabling them to be processed as a text stream, rather than as an audio stream. The missing system functionality that the wizard provides may be implemented in later versions of the system, or it may be speculative capabilities that current-day systems do not have; the precise details are generally considered irrelevant to the study. In testing situations, the goal of such experiments may be to observe the use and effectiveness of a proposed
user interface In the industrial design field of human–computer interaction, a user interface (UI) is the space where interactions between humans and machines occur. The goal of this interaction is to allow effective operation and control of the machine fro ...
by the test participants, rather than to measure the quality of an entire system.


Origin

The name of the experiment comes from L. Frank Baum's 1900 novel '' The Wonderful Wizard of Oz'', in which an ordinary man hides behind a curtain and uses "amplifying" technology to pretend to be a powerful wizard. John F. Kelley coined the phrases "Wizard of OZ" and "OZ Paradigm" for this purpose circa 1980 to describe the method he developed during his dissertation work at
Johns Hopkins University The Johns Hopkins University (often abbreviated as Johns Hopkins, Hopkins, or JHU) is a private university, private research university in Baltimore, Maryland, United States. Founded in 1876 based on the European research institution model, J ...
. During the study, in addition to one-way mirrors and other techniques, there was a blackout curtain separating Kelley (the "Wizard") from the participant's view. The "Experimenter-in-the-Loop" technique had been pioneered at Chapatis' Communications Research Lab at Johns Hopkins as early as 1975, three years before Kelley's arrival. W. Randolph Ford used the experimenter-in-the-loop technique with his CHECKBOOK program, wherein he obtained language samples in a naturalistic setting. In Ford's method, a preliminary version of the natural language processing system would be placed in front of the user. When the user entered an unrecognized syntax, he would receive a "Could you rephrase that?" prompt from the software. After the session, the processing algorithms would be modified to address the newly obtained samples, and another session would take place. This approach led to the eventual development of his natural language processing technique, "Multi-Stage Pattern Reduction". Ford believed that Kelley coined the phrase "Wizard of Oz Paradigm" to describe a technique employed at least twice before Kelley began his work. Another team, Allen Munro and Don Norman from the University of California, San Diego, (Bobrow, et al.) used a similar technique to model a natural language understanding system at the Xerox Palo Alto Research Center circa 1975. In that employment, the wizard sat at a terminal in an adjacent room separated by a one-way mirror so the subject could be observed. Every input from the user was processed correctly by a combination of software processing and real-time wizard intervention. As the process was repeated in subsequent sessions, more and more software components were added, and the wizard's role was gradually reduced. Eventually, the machine reached a point at which it could be left unattended, enabling the wizard to validate the final system's unattended performance. In their 1985 University of Michiga
technical report
Green and Wei-Haas state the following: ''The first appearance of the "Wizard of Oz" name in print was in Jeff Kelley's thesis (Kelley, 1983a, 1983b, 1984a). It is thought the name was coined in response to a question at a graduate seminar at Hopkins (Chapanis, 1984; Kelley, 1984b). "What happens if the subject sees the experimenter ehind the "curtain" in an adjacent room acting as the computer" Kelley answered: "Well, that's just like what happened to Dorothy in the Wizard of Oz." And so the name stuck.'' There is also a passing reference to planned use of the "Wizard of Oz experiments" in
1982 proceedings paper
by Ford and Smith. Originally, Kelley also used "OZ" as an acronym for "Offline Zero", a reference to the wizard's real-time interpretation of user input during the simulation phase. Similar experimental setups had occasionally been used earlier under other names. Design researcher
Nigel Cross Nigel Cross (born 1942) is a British academic, a design researcher and educator, Emeritus Professor of Design Studies at The Open University, United Kingdom, where he was responsible for developing the first distance-learning courses in design in ...
conducted studies in the 1960s with "simulated"
computer-aided design Computer-aided design (CAD) is the use of computers (or ) to aid in the creation, modification, analysis, or optimization of a design. This software is used to increase the productivity of the designer, improve the quality of design, improve c ...
systems where the purported simulator was actually a human operator, using text and graphical communication via
CCTV Closed-circuit television (CCTV), also known as video surveillance, is the use of closed-circuit television cameras to transmit a signal to a specific place on a limited set of monitors. It differs from broadcast television in that the signa ...
. As he explained, "All that the user perceives of the system is this remote-access console, and the remainder is a black box to him. ... one may as well fill the black box with people as with machinery. Doing so provides a comparatively cheap simulator, with the remarkable advantages of the human operator's flexibility, memory, and intelligence, and which can be reprogrammed to give a wide range of computer roles merely by changing the rules of operation. It sometimes lacks the real computer's speed and accuracy, but a team of experts working simultaneously can compensate to a sufficient degree to provide an acceptable simulation." Cross later referred to this as a kind of
Reverse Turing test A reverse Turing test is a Turing test in which failure suggests that the test-taker is human, while success suggests the test-taker is automated. Conventionally, the Turing test is conceived as having a few computer AI subjects communicate wit ...
.Cross, N (2001) "Can a Machine Design?", ''Design Issues'', Vol. 17, No.41, pp. 44-50.


Significance

The Wizard of OZ method is very powerful. Originally, Kelley created a simple keyboard-input natural language recognition system that far exceeded the recognition rates of any of the far more complex systems of the day. Contemporary computer scientists and linguists thought that, in order for a computer to be able to "understand" natural language enough to be able to assist in useful tasks, the software would have to be attached to a formidable "dictionary" having a large number of categories for each word. The categories would enable a very complex parsing algorithm to unravel the ambiguities inherent in naturally produced language. The daunting task of creating such a dictionary led many to believe that computers simply would never truly "understand" language until they could be "raised" and "experience life" as humans, since humans seem to apply a life's worth of experiences to the interpretation of language. The key enabling factor for the first use of the OZ method was that the system was designed to work in a single context (calendar-keeping), which greatly constrained user input so greatly that a simple language processing model could meet the goals of the application. The processing model was a two-pass keyword/key-phrase matching approach, based loosely on the algorithms of Weizenbaum's
Eliza ELIZA is an early natural language processing computer program developed from 1964 to 1967 at MIT by Joseph Weizenbaum. Created to explore communication between humans and machines, ELIZA simulated conversation by using a pattern matching and ...
program. By inducing participants to generate language samples in the context of solving an actual task (using a computer that they believed actually understood what they were typing), the variety and complexity of the lexical structures gathered was greatly reduced and simple keyword matching algorithms could be developed to address the actual language collected. This first use of OZ was in the context of an
iterative design Iterative design is a design methodology based on a cyclic process of prototyping, testing, analyzing, and refining a product or process. Based on the results of testing the most recent iteration of a design, changes and refinements are made. T ...
approach. In the early development sessions, the experimenter simulated the system ''in toto'', performing all the database queries and composing all the responses to the participants by hand. As the process matured, the experimenter replaced human interventions, piece by piece, with newly created developed code, which was designed to process all previously received inputs. By the end of the process, the experimenter could observe and analyze the sessions in a "hands-off" mode. OZ addressed the obvious criticism that iterative methods could not be used to build a separate natural language system (dictionaries, syntax) for each new context, as such a method would require repeatedly adding new structures and algorithms to handle each new batch of inputs. OZ's empirical approach made this feasible; in its original application, dictionary and syntax growth reached asymptotic (achieving from 86% to 97% recognition rates, depending on the measurements employed) after only 16 experimental trials and the resulting program, with dictionaries, was less than 300k of code. Since its initial publication, the OZ method has been employed in a wide variety of settings, notably in the prototyping and usability testing of proposed user interface designs in advance of having actual application software in place.


See also

*
Reverse Turing test A reverse Turing test is a Turing test in which failure suggests that the test-taker is human, while success suggests the test-taker is automated. Conventionally, the Turing test is conceived as having a few computer AI subjects communicate wit ...
- A Turing test in which the objective or roles between computers and humans have been reversed * Chinese room - A thought experiment with a similar premise. * The Turk - Wizard of Oz device used as a fake chess-playing machine


References

{{DEFAULTSORT:Wizard of Oz experiment Human–computer interaction