Using the Think-Aloud Method to Gather Data on What it Takes to Comprehend a Legal Decision

• 1 Introduction

  An empirical study is set out in which both law students and legal practitioners are engaged in comprehending a legal decision while thinking aloud. This study is carried out to examine how novices, expert beginners and experts actually comprehend a legal decision. We want to find out what difficulties are experienced and what may account as causes. In the Dutch legal system decisions by judges are a major source of law. So it is important that law students read, structure and analyse legal decisions. However, reading and understanding a decision does not go without saying. Observations with first year law students reading decisions showed that they experience difficulties with seeing through the composition of the decision, with reconstructing the argument structure and with determining the legal significance of the decision¹xI obtained these observations over a series of years (2000-2013) while teaching the first year Bachelors Course Introduction to Law in which law students have to read and comprehend legal decisions. We need to gain insight in what it takes to comprehend a legal decision for designing instructions for effectively and efficiently learning this task.
  The main focus of this article is on the experimental design of the empirical study. We introduce the experimental design and our approach for analysing the protocols.

• 2 Experimental design

  We want to find out how novices, expert beginners and experts comprehend a legal decision.²xThis grouping depends on the amount of time spent on the task. We therefor ask our subjects to work on two legal decisions while thinking aloud. We start with a short introduction on the method employed followed by a description of the materials and the subjects involved.

  2.1 The Think-Aloud Method

  The method employed to explore novice, expert beginners and expert behaviour is to collect verbal data by asking novices, expert beginners and experts to work on a legal decision and instruct them to ‘think aloud’, while everything they say is recorded. These recordings (protocols) are then transcribed and analysed. This method is referred to as the Think-Aloud Method. This use of verbal data collected by introspection already has a successful history in the area of problem solving (see, for instance, Selz 1922; De Groot 1946/1965; Newell & Simon 1972; Elshout 1976; Elshout-Mohr 1976; Mettes & Pilot 1980; Snoek 1989; Muntjewerff 2001).

  There are two vital issues in gathering and analysing verbal reports. The first issue relates to the (a priori) validity of self-report data. The second issue relates to the need of an interpretation theory for analysing the data. That introspection is not without problems is related to the fact that not all cognitive processes can be reported verbally, because, for instance, they occur very rapidly or because they are difficult to verbalize. This will lead to ‘gaps’ in the verbal reports. Therefore it has been suggested that the problems that may arise with introspection should be made explicit beforehand in the form of a model or ‘theory’ about these phenomena (see, e.g., Elshout 1976; Ericsson & Simon 1980; Breuker 1982; Ericsson & Simon 1984).

  The model of Ericsson & Simon (1980) intends to explain and predict under what conditions verbal report influences task performance, under what conditions verbal report is more or less complete and under what conditions verbal report is in accordance with other data collected from the same task performance (valid).

  ‘Our purpose in presenting a specific processing model is to aid us in interpreting verbal data obtained from subjects and the relation of their verbal to other behaviour. Since the data (including the verbal data) are gathered to test theories about the human information processing system, we are engaged in something of a bootstrap operation. We need a model to interpret data that are to be used to test the model’ (Ericsson & Simon 1980, p. 222-223).

  Breuker adopts this view, though he claims that the model of self-report

  ‘(...) should be conceived as part of the model that is used to interpret (analyse) the protocols’ (Breuker 1982, p. 116).

  We in turn will adopt Breuker’s approach. The verbal data (protocols) gathered should be described or explained on the basis of a theory or interpretation model. In order to analyse the protocols it is necessary to have already some ideas about what to look for in the protocols, as protocols do not speak for themselves.

  Therefore the first step is to construct and formulate a theory to describe the categories and operations which may be relevant in performing the task. This theory is called an interpretation theory, because it is used to interpret statements in the protocols.

  Secondly a model of self-report can be described within the framework of the interpretation theory as suggested by Breuker (1982). Not all internal events can be verbalized all the time, due to the fact that not all inner events can be observed by the performer and that although an event may be observed by the performer it may be hard to describe. Using verbal data in comprehending a legal decision is comparable with the use of verbal data in other problem solving tasks (e.g., chess). These tasks are characterized by the fact that the categories and rules are a closed but not automated world. These reasoning tasks show relatively slow thinking and the processes are rather easy traceable. However, subjects have the impression that their minds are working at high speed.

  The third step consists of the translation of the protocols in terms of the interpretation theory, i.e. assigning formulas from the interpretation theory to the statements (coding). In analysing protocols many things can go wrong. It may be that the interpretation categories are inadequate, which will lead to a too superficial description of the cognitive processing, or the analysis may be too detailed.
  Protocols also contain statements that are not direct reflections of the task at hand. Irrelevant statements may not cause problems, however, incomplete statements will, because there may be statements that are multi-interpretable by providing insufficient context, or the experimenter may have been too anxious not to interfere with the self-report to ask for clarifications. There may also be real gaps in the protocols, because the report goes slower than the actual processing.
  Gaps may be caused by forgetting (time between processing and report) or by intermediate processes that are not reportable; therefore protocols may have a too rational appearance.

  We gathered our protocols by instructing the subjects to comprehend two legal decisions while expressing everything they were thinking. These verbal protocols are used as the raw data gathered about the problem solving process. However, to be able to see the implications for theories on comprehending legal decisions these protocols need substantial interpretation and analysis.

  To summarize the activities we have to carry out in the experiment:

  • gather data:

    • select materials;

    • select subjects;

  • analyse data:

    • construct or formulate an interpretation theory;

    • describe model of self-report;

    • code protocols (translate protocols in terms of interpretation theory).

  To be able to gather the data we have to select the materials, being the legal decisions to be comprehended by the subjects, and the subjects to comprehend these legal decisions.

  2.2 Materials

  Two legal decisions are presented to the subjects. Both decisions are appropriate for the subjects to solve not being too easy or too difficult. Both decisions are representative examples of problems in law.

  The first decision is Supreme Court 24 October 1978, ECLI:NL:HR:1978:AC6373 (Uitzendburo Cito (Cito Employment Agency)). The second decision is Supreme Court 23 October 1984, ECLI:NL:HR:1986:AC8567 (Bijlmer Noodweer (Bijlmer Self Defence)). Both decisions are presented to the subjects in full leaving out the conclusion of the Advocate-General and the annotation.

  2.3 Subjects

  The subjects in the experiment are five first year law students (beginners), five third year law students (expert beginners) and five legal practitioners (experts). The students all study law at the Faculty of Law of the University of Amsterdam.
  The students were asked to volunteer in the experiment. Two of the legal experts are working in the field of criminal law, two are working in the field of civil law, and one is working in the field of administrative law.

• 3 Running the experiment

  The experiment in which law students are asked to solve two legal cases while thinking aloud is carried out in three periods. The first period ran from 2 November 2011 untill 17 November 2011 and concerned expert-beginners, the second period is planned from 22 October 2015 untill 1 November 2015 and will concern novices.
  The five legal experts protocols will be gathered in the period from 1 untill 29 March 2015.

  3.1 Setting

  Each student is invited to the Faculty of Law on a specific date and time to participate in the experiment. The room used is a standard room at the department of General Legal Theory.
  The legal practitioners may either come to the room at the department of General Legal Theory, or the experimenter may come to them, dependent on their preference.
  After welcoming the subject the experimenter provides a short explanation about the purpose of the research, the course of the experiment and what is expected from the subject.
  It is also explained to the subject that the data are to be handled with strict confidentiality. The subject is provided with paper and pencil in case s/he wants to make notes.

  3.2 Instructions

  All subjects are asked to bring their lawbooks.³xIn Dutch: wetboeken. After the introduction the experimenter presents the first legal decision and asks the subject to comprehend the legal decision and to say aloud everything that comes to mind (see Fig. 1).

  Instruction by the experimenter⁴xIn Dutch: welkom eh het is vandaag 4 november 2011 10.00 eh [naam proefpersoon] de eh bedoeling is dat je een eh een rechterlijke beslissing een uitspraak die geef ik je zo eh dat je die gaat eh bekijken en probeert te begrijpen waar eh het over gaat en dat je bij dat proces eh te werk gaat zoals je gewend bent te werk te gaan als je een uitspraak eh bekijkt en dat je daarbij eh zo veel mogelijk wat je doet en wat je denkt en wat je te binnen schiet eh dat je dat hardop eh uitspreekt.
  

  3.3 Role of the experimenter

  The experimenter is present at each session. The role of the experimenter is a restrained one.
  After the introduction and the instructions the only interference allowed is prompting a subject when s/he stops talking (‘keep on talking’). In no way the experimenter is allowed to hint or correct the subject during the process of comprehending the legal decision.

  3.4 Recording

  A digital voice recorder⁵xOlympus Digital Voice Recorder VN-8700PC. is used to record every utterance of both the subject and the experimenter during the session.
  The recordings start with the experimenter introducing and explaining the experiment and ends after the experimenter has thanked the subject for his or her participation. The resulting recordings are in MP3 format and are saved with the specific date and time.

  3.5 Transcribing

  All recordings are typed out completely and verbatim. The impasses, when no one is talking, are also registered using dots between brackets (…) and an indication of the period of time (in seconds).
  We present a fragment of a verbatim expert beginner protocol in which the Uitzendburo Cito (Cito Employment Agency) decision has to be comprehended as an example (see Fig. 2).

  Fragment of a verbatim expert beginner protocol⁶xIn Dutch: ehmm nog een keer kijken wat er in het hoger beroep is gedaan waarbij in hoger beroep een vonnis van de rechtbank ehm maar daar hebben ze hem dus toen vrijgesproken van hetgeen er onder 2 en 3 (… 5 seconden) eh toen wel bewezen was aangenomen als bewezen was aangenomen (… 10 seconden) en dat ze hem dan wel voor het tenlastegelegde 1 en 2 (… 5 seconden) hebben veroordeeld nu gaat het [onverstaanbaar] voorbereidings en uitvoerings eh sorry voorbereidingshandelingen en geen uitvoeringshandelingen waardoor het dan (… 5 seconden) ehm geen poging kan zijn omdat daar zoals ik me kan herinneren een begin van uitvoering moet zijn (… 10 seconden).
  

  The transcribed protocols are the raw data that need to be analysed.

• 4 Analysing the protocols

  Analysing the data involves the construction or formulation of an interpretation theory, the description of a model of self-report and the coding of the protocols, which is a translation of the protocols in terms of the interpretation theory. Coding means assigning labels to protocol fragments following the coding scheme. The coding scheme or model therefore has to be constructed first.

  We will not perform a quantitative, statistical, analysis of the protocols on the basis of a formal coding.

  We plan a qualitative analysis using a template to explore the protocols and to compare each protocol with. We do this because we look at the protocols from two different points of view: from the point of view of order or sequence, which also refers to interdependencies and repetitions (reiterations), and from the point of view of content.

  In analysing the protocols we first explore the protocols on the basis of the template, followed by a comparison of the protocols with our template. In the exploration we will look at the order or sequence subjects use, which involves an inspection of the subtasks, and the content subjects use, which involves an inspection of the input and output data. After the exploration we compare each protocol with the template also both from the point of view of order and the point of view of content.

  We prepare the transcribed protocols for analysis by breaking up sentences in propositions and by rearranging the propositions.
  The sequence is rearranged when indicated by the subjects or when it appeared ‘logically’ necessary by the interpreter. The intended sequence is indicated by the subjects’ use of indicators or temporal relations (e.g., before, then, first, etc.).

  When protocols are obviously incomplete and it is possible to infer what was implied propositions are added. Incompleteness or gaps may relate to processes of which the subject thinks that the experimenter will know what s/he is doing. For example, after reading the legal decision, the subject may not explicitly state his or her understanding of the situation description, because this would merely be a recount of the case description. Another example may be the ‘select article’ activity. It may appear later on in the protocol that the subject has selected the article because s/he is using it. However, the activity appears difficult to report.

  There are also different kinds of verbalizations that are not directly related with comprehending the legal decision. The subject may be talking about issues that have absolutely nothing to do with comprehending the legal decision (for example ‘O dear, I must not forget to apply for the criminal law exam.’), verbalizations that refer to an evaluation of comprehending the legal decision at the meta-level (‘This is a difficult case.’), or comments on oneself (‘I am a bit nervous.’).

  4.1 Our template

  The template we use in the analysis of the protocols specifies the structured situation description (i.e. proven stated facts), the legal question, the legal rules, the reasoning structure and the conclusion for both legal decisions used in the experiment.⁷xSee Muntjewerff (2012) for a full description of the model of the task of comprehending a legal decision used as a basis of the template. See Fig. 3 for the specification of the content in the Cito Employment Agency decision.

  We explore the protocols using the template from the point of view of content. We want to find out if novices have difficulties in comprehending a legal decision and if so if we can attribute these difficulties to insufficient mastery of, or insight in, the subject matter. We therefore first have a look at what knowledge is used. The template describes which elements have to be used. There are basically two possible outcomes:

  • the protocol matches the content in the model;

  • the protocol does not match the content in the model.

  Next we explore the protocols from the point of view of order. See Table 1 below for the specification of order. There are different ways in which a subject may address the issues in the legal decision. A subject may start with selecting the facts (structured situation description 1), with the conclusion (conclusion 1) or with the legal question (legal question/point of law 1).

  1 Order of elements addressed in comprehending a legal decision

  Structured situation description	1	5	3
  Legal rules	2	4	2
  Legal question/point of law	3	3	1
  Reasoning structure	4	2	4
  Conclusion	5	1	5

  Template for the Cito Employment Agency decision
  

  4.2 Example analysis

  Here we show an example of the exploration of one of the expert beginners protocols. The subject had to read, structure and analyse the Cito Employment Agency decision. In Fig. 4 we see the results of the expert beginners protocol explored on basis of content.

  Expert beginner protocol from the point of view of content
  

  In table 2 we see the result of the expert beginners protocol explored on the basis of order. The subject started with picking out the legal question, followed by the structured situation description (the facts of the case), the reasoning structure and the conclusion. We see here that the subject did not address the legal rules.

  2 Expert beginner protocol from point of view of order

  Structured situation description	2
  Legal rules	-
  Legal question/point of law	1
  Reasoning structure	3
  Conclusion	4

  What does this teach us? The exploration of this expert beginner protocol shows us that the result is incomplete. The expert beginner does not explicitly refer to the legal rule that is at stake in this case (being article 45 of the Dutch Penal Code). However, referring to the legal rule is essential for really understanding a legal decision. We also see that the expert beginner does not (re)formulate the legal question at stake in this case. In designing instructions for learning to comprehend a legal decision support should be provided on completeness of activities and completeness of content.

• 5 Summary and future research

  Using the Think-Aloud Method is a proven method to gather data to acquire insight in the way experts, expert beginners and novices carry out a certain task. It provides data on differences in task performance, on difficulties experienced in performing a certain task and on ways of carrying out a certain task. These insights can be used to develop instructional materials for learning the specific task, for theory development and for designing software.
  At the moment we are engaged in gathering and transcribing the protocols. This will be followed by exploring and coding the protocols on the basis of our template (interpretation theory) both from the point of view of content as from the point of view of order. Then we will analyse the protocols by comparing the protocols both within groups and between groups. These analyses are carried out in order to acquire (more) knowledge and insight in the task of comprehending a legal decision and the knowledge required to be able to perform this task.

References

• Breuker 1982

  J.A. Breuker, ‘Hardop-denken. Methodologie voor het analyseren van hardopdenkprotocollen’ (Think Aloud. Methodology for Analyzing Think-Aloud Protocols), in: J.L. van de Kamp & M. van de Kamp (eds.), Methodologie van Onderwijsresearch, Lisse: Swets & Zeitlinger 1982.

• Elshout 1976

  J.J. Elshout, Karakteristieke moeilijkheden in het denken (Characteristic Difficulties in Thought). Amsterdam:University of Amsterdam 1976.

• Elshout-Mohr 1976

  M. Elshout-Mohr, Training in denkstrategieën (Training Thinking Strategies), Amsterdam: University of Amsterdam 1976.

• Ericsson & Simon 1980

  K.A. Ericsson & H.A. Simon, ‘Verbal Reports as Data’, Psychological Review (87) 1980, p. 215-251.

• Ericsson & Simon 1984

  KA. Ericsson & H.A. Simon, Protocol Analysis: Verbal Reports as Data, Cambridge, MA: MIT Press 1984.

• Ericsson & Smith 1991

  K.A. Ericsson & J. Smith (eds.), Toward a General Theory of Expertise: Prospects and Limits, Cambridge: Cambridge University Press 1991.

• Ericsson et al. 2006

  K.A. Ericsson, N. Charness, P.J. Feltovic & R.R. Hoffman (eds.), The Cambridge Handbook of Expertise and Expert Performance, New York: Cambridge University Press 2006.

• Ericsson et al. 2007

  K.A. Ericsson, M.J. Prietula & E.T. Cokely, ‘The Making of an Expert’, Harvard Business Review July 2007 (http://harvardbusinessonline.hbsp.harvard.edu/hbsp/hbr/articles).

• De Groot 1946

  A.D. Groot, Het denken van den schaker (Thought and Choice in Chess), Amsterdam: University of Amsterdam 1946.

• De Groot 1965

  A.D. Groot, Thought and Choice in Chess, The Hague: Mouton 1965.

• Mettes & Pilot 1980

  C.T.C.W. Mettes & A. Pilot, Over het leren oplossen van natuurwetenschappelijke problemen. Een methode voor ontwikkeling en evaluatie van onderwijs, toegepast op een kursus Thermodynamica (On Learning to Solve Physics’ Problems. A Method for Designing and Evaluating Instruction, Applied to a Thermodynamics Course), Enschede: Technische Hogeschool Twente 1980.

• Muntjewerff 2001

  A.J. Muntjewerff, An Instructional Environment for Learning to Solve Legal Cases: PROSA (Ph.D. thesis University of Amsterdam), Amsterdam: University of Amsterdam 2001.

• Muntjewerff 2011

  A.J. Muntjewerff, ‘CASE Learning to Structure and Analyze a Legal Decision’, Proceedings of the 10th European Conference on e-Learning – ECEL 2011, Brighton, http://academic-conferences.org/ecel/ecel2011/ecel11-proceedings.htm.

• Muntjewerff 2012

  A.J. Muntjewerff, ‘An Explicit Model for Learning to Structure and Analyze Decisions by Judges’,25th IVR World Congress: Law, Science and Technology, Frankfurt am Main, 15-20 August 2011, Paper Series 114/2012, http://publikationen.ub.uni-frankfurt.de/home.

• Newell & Simon 1972

  A. Newell & H.A. Simon, Human Problem Solving. Englewood Cliffs: Prentice Hall 1972.

• Selz 1922

  O. Selz, Zur Psychologie des productiven Denkens und des Irrtums. Eine experimentale Untersuchung. Bonn: Cohen 1922.

• Snoek 1989

  J.W. Snoek, Het denken van de neuroloog (Reasoning in Neurology). Groningen: Universiteitsdrukkerij 1989.

• Van Someren et al. 1993

  M.W. van Someren, Y. Barnard & J. Sandberg, The Think Aloud Method. London: Academic Press 1993.

Noten

• 1 I obtained these observations over a series of years (2000-2013) while teaching the first year Bachelors Course Introduction to Law in which law students have to read and comprehend legal decisions.

• 2 This grouping depends on the amount of time spent on the task.

• 3 In Dutch: wetboeken.

• 4 In Dutch: welkom eh het is vandaag 4 november 2011 10.00 eh [naam proefpersoon] de eh bedoeling is dat je een eh een rechterlijke beslissing een uitspraak die geef ik je zo eh dat je die gaat eh bekijken en probeert te begrijpen waar eh het over gaat en dat je bij dat proces eh te werk gaat zoals je gewend bent te werk te gaan als je een uitspraak eh bekijkt en dat je daarbij eh zo veel mogelijk wat je doet en wat je denkt en wat je te binnen schiet eh dat je dat hardop eh uitspreekt.

• 5 Olympus Digital Voice Recorder VN-8700PC.

• 6 In Dutch: ehmm nog een keer kijken wat er in het hoger beroep is gedaan waarbij in hoger beroep een vonnis van de rechtbank ehm maar daar hebben ze hem dus toen vrijgesproken van hetgeen er onder 2 en 3 (… 5 seconden) eh toen wel bewezen was aangenomen als bewezen was aangenomen (… 10 seconden) en dat ze hem dan wel voor het tenlastegelegde 1 en 2 (… 5 seconden) hebben veroordeeld nu gaat het [onverstaanbaar] voorbereidings en uitvoerings eh sorry voorbereidingshandelingen en geen uitvoeringshandelingen waardoor het dan (… 5 seconden) ehm geen poging kan zijn omdat daar zoals ik me kan herinneren een begin van uitvoering moet zijn (… 10 seconden).

• 7 See Muntjewerff (2012) for a full description of the model of the task of comprehending a legal decision used as a basis of the template.