MULTIPLE-RESPONDENT ANECDOTAL ASSESSMENTS: AN ANALYSIS OF INTERRATER AGREEMENT AND CORRESPONDENCE WITH ANALOGUE ASSESSMENT OUTCOMES (2024)

J Appl Behav Anal. 2012 Winter; 45(4): 779–795.

PMID: 23322932

Participants and setting

This study was conducted at a large, state-sponsored residential facility for individuals with intellectual disabilities. Assessments were administered in secluded areas of the residential apartments or quiet areas in the vocational training setting.

Residents

Twenty-seven individuals who resided at the facility participated in Experiment 1. Their ages ranged from 27 to 66 years, and all had been diagnosed with intellectual disabilities. Table 1 shows each individual's age and functioning level.

Target behaviors. All participants had a history of problem behavior of sufficient severity to necessitate the development of behavior support plans. The behavioral definitions used in Experiment 1 were developed by the individuals' unit psychologists and were part of each individual's behavior support plan. Twelve participants presented with a single target behavior, and 15 presented with two target behaviors, for a total of 27 participants with 42 target behaviors. The MAS and QABF were completed for each target behavior; thus, for residents who presented with two target behaviors, 20 assessments were completed (five MAS and five QABF for each target behavior). Due to an error in administration of the assessments, data from only four respondents are reported for Barbara. Table 1 includes a description of residents' target behaviors and definitions.

Respondents

Respondents for Experiment 1 were 113 staff members who had worked regularly with residents as direct-care, vocational, or unit management staff for a minimum of 6 months. Their educational backgrounds were unavailable, because prior to the study the facility had discontinued a longstanding hiring policy that required a high-school education or equivalent for employment. All respondents were employees of the facility at the time of the interviews. Multiple respondents (typically five) were interviewed for each resident's target behavior. This number of respondents was chosen because it seemed reasonable that it would be possible to identify five caregivers who had sufficient histories with participants to provide meaningful responses to the assessment items.

Materials

Materials used in Experiment 1 included writing utensils and two sets of each questionnaire (MAS and QABF). The general information sections of each assessment (name, residence, date, rater, target behavior, etc.) were completed by the interviewer before the assessment. Interviewers read aloud and marked the answers stated on one set of questionnaires while respondents read along with the second set.

MAS

The MAS () is a 16-question assessment with four questions that correspond to each of four subscale categories: escape, sensory, attention, and tangible. Respondents answered questions using a 7-point Likert-type scale; scores indicated the extent to which the rater observed the behavior, from 0 (never) to 6 (always). Intermediate values allowed respondents to score 1 (almost never), 2 (seldom), 3 (half of the time), 4 (usually), or 5 (almost always). The four questions that corresponded to each category were summed and ranked by point value. The category with the highest value was assumed to represent the maintaining consequence.

QABF

The QABF () is a 25-question assessment with five questions assigned to each of five subscales: attention, escape, nonsocial, physical, and tangible. The scale allowed respondents to use a 4-point Likert-type scale to score how often the client demonstrated the target behavior. Respondents chose from 0 (never), 1 (rarely), 2 (some), and 3 (often). Once completed, the assessments were scored for the number of items endorsed (i.e., if a question was answered) and for the point total assigned to each subscale category. Subscale categories were then ranked according to the score, and the category with the highest point value was assumed to represent the maintaining consequence.

Administration procedures

Twenty-three graduate and undergraduate students (interviewers) were trained to administer the QABF and MAS. Training included reading and discussion of the MAS and QABF manuals, role-playing administration procedures with a senior trainer who provided feedback, observing a senior trainer administer an assessment, and receiving feedback following initial assessment delivery. To assist respondents who may have had difficulty reading the assessment, each respondent was given a copy of the questionnaire to read along as the interviewer read each question aloud and scored answers. Interviewers confirmed at the start of each interview that the respondent had worked with the resident for 6 months or longer and then read the general information, including the definition of the target behavior, before reading assessment questions. Interviewers administered assessments in quiet, secluded areas of residential or vocational buildings to ensure that potential respondents (i.e., other staff members) were not present and could not hear respondents' answers. Interviewers read questions aloud to respondents exactly as written; no additional information or clarification was given. If respondents asked questions, they were told to “answer the best you can.” The MAS was administered first, followed by the QABF. After both instruments had been completed, the interviewer thanked the respondent and left the area. If an individual presented with two target behaviors, interviewers returned at another time to administer assessments for the second target behavior.

Respondent agreement evaluation

Two trained graduate students scored each assessment. Resulting scores were compared on a question-by-question basis. Agreement in the scoring for both the MAS and QABF was 100%.

Agreement across assessments (within respondents)

Agreement across assessments and within respondents was scored if the respondent identified the same maintaining consequence with both instruments. As noted previously, the MAS was organized according to four categories of maintaining variables (sensory, escape, attention, and tangible). The QABF was organized according to five categories of maintaining variables (nonsocial, escape, attention, tangible, and physical). Agreement was scored if a respondent ranked sensory as the maintaining variable on the MAS and either nonsocial or physical as the maintaining variable on the QABF. If a respondent scored two categories as the highest ranking (i.e., there was a tie) on one assessment, both categories were compared with the highest ranking category from the other assessment. For example, agreement was scored if a respondent scored both attention and escape as the highest ranking categories on the QABF (i.e., attention and escape received the same score, which was higher than scores for other categories) and scored attention as the highest ranking category on the MAS.

Agreement across assessments (across respondents)

Agreement across assessments (across respondents) was scored if four or five of the five respondents for each resident identified the same maintaining variable for both the MAS and QABF.

Agreement within assessments (across respondents)

Agreement within assessments (across respondents) was scored if four or five of the five respondents for each resident identified the same maintaining variable on either the MAS and QABF.

Table 2

Respondent Agreement Across and Within Assessments

Table 3

Individual Results, with Respondent Groups Listed Across Primary Maintaining Consequences

Participants, setting, and materials

All sessions were conducted in a clinic for the assessment and treatment of behavior disorders, located on the campus of the residential facility at which Experiment 1 was conducted. Sessions were conducted in one of the clinic's observation rooms. Rooms (3.7 m by 3.7 m) contained a table, two chairs, and materials appropriate for the experimental session. A one-way mirror was installed in one wall of each room for unobtrusive observation and recording of session data.

For pica assessments, materials included simulated nonfood items that could be safely consumed by participants. Simulated pica items for Greg were all-natural soap (made from edible oils and wax) placed in a soap dish; mixtures of water, white vinegar, apple cider vinegar, Simply Thick Gel, and food coloring placed in bath gel bottles; a hand sanitizer pump; and a spray bottle, to simulate bath products, cleaning chemicals, and hand sanitizer. Simulated pica items for Vern were rice paper to simulate paper, onion skins to simulate paper and leaves, dried seaweed to simulate leaves, and brown fettuccini to simulate leaves and twigs. These items were continuously available throughout all FA sessions for both participants.

Participants

Eight individuals were selected from among Study 1 participants for whom at least four of the five respondents showed agreement within the QABF and, with the exception of one case (Asa), the MAS (see Table 1 for demographic information). Participants were selected to obtain representation from all subscale categories that showed agreement among at least one group of respondents (nonsocial, attention, tangible, and escape) and based on availability and continuing need for behavioral intervention. Greg (46-year-old man) and Vern (41-year-old man) had been diagnosed with profound mental retardation. Both men engaged in pica, and their anecdotal assessments indicated that pica was maintained by nonsocial reinforcement. Jolinda was a 55-year-old woman who had been diagnosed with profound mental retardation and who exhibited SIB. Annie was a 33-year-old woman who had been diagnosed with profound mental retardation and who engaged in physical aggression to others (PAO). Anecdotal assessment results indicated that target behaviors for Jolinda and Annie were maintained by positive reinforcement in the form of tangible items. Karen was a 47-year-old woman who had been diagnosed with profound mental retardation and who engaged in verbal disruptive behavior (VDB). Asa was a 51-year-old man who had been diagnosed with profound mental retardation and who exhibited SIB. QABF assessment results for Karen and Asa indicated that their problem behaviors were maintained by negative reinforcement in the form of escape from task demands; Asa's MAS outcomes were inconclusive. Chad was a 27-year-old man who had been diagnosed with profound mental retardation, and Peter was a 29-year-old man who had been diagnosed with mild mental retardation. Peter's FA was conducted prior to the current study, and those data were initially presented by Dracobly and Smith (2012). Both men engaged in SIB that, according to QABF results, was maintained by attention. Chad's MAS results indicated that his problem behavior was maintained by tangible reinforcement, and Peter's MAS results indicated that his problem behavior was maintained by automatic reinforcement. Chad sustained a fall, unrelated to his participation in the study, after participating in only four sessions of his FA. Based on injuries related to the fall and concerns about the severity of his SIB, he was not eligible for further participation in the study and, therefore, it was not possible to evaluate correspondence between his FA and anecdotal assessments.

Target behaviors

The operational definitions used for the FA were based on residents' target behavior definitions found in their formal behavior support plans. Definitions were refined for the FA when necessary based on direct observations of problem behavior and a review of the client's records, including the daily reports of problem behavior across all settings. Greg and Vern's target behavior was pica, defined as the insertion of simulated pica items into the mouth. Jolinda's target behavior was SIB, defined as biting her fingers or hands, head banging, or slapping herself. Annie's target behavior was PAO, defined as scratching, hitting, pinching, biting, pushing, or grabbing another person. Karen's target behavior was VDB, defined as yelling, screaming, or crying. Asa's target behavior was SIB, defined as banging his head on hard surfaces or biting his hand orfinger. Chad's target behavior was SIB, defined as his head striking any object (including people). Peter's target behavior was SIB, defined as hitting himself with his hand on any part of his body (e.g., head, face, chin, forehead, leg, etc.) or hitting his head or hand against an object. For Peter, head up, defined as no part of his chin or neck touching his chest or shoulders, also was scored.

Observation procedures

Trained observers used handheld computers to record target behaviors. Frequency measures were used for pica (Greg and Vern); head banging (Asa, Chad, and Peter); hitting body parts against objects (Peter); hitting others (Annie); biting (Annie); pinching (Annie); hitting self (Peter); head-up (Peter); and scratching, pushing, or grabbing others (Annie). Duration measures were used for yelling, screaming, and crying (Karen); head banging (Jolinda); slapping self (Jolinda); and biting hands or fingers (Asa and Jolinda).

Interobserver agreement

A second observer independently and simultaneously scored 85% of Greg's sessions, 65% of Vern's sessions, 80% of Jolinda's sessions, 60% of Annie's sessions, 64% of Karen's sessions, 63% of Asa's sessions, and 38% of Peter's sessions. Interobserver agreement was calculated by dividing each session into 1-s intervals, summing the number of intervals in which the primary and secondary observers agreed on the occurrence or nonoccurrence of the target behavior, dividing the result by the total number of intervals in the session, and converting the outcome to a percentage. Agreement was calculated slightly differently for Peter due to difficulty in determining the exact second of the onset of head up. Therefore, agreement for Peter's sessions was calculated as above but with a moving 2-s window (e.g., if the primary observer recorded an event at time x, agreement was scored if the secondary observer recorded the same event at time x – 1 s, time x, or time x + 1 s), dividing the result by the total number of intervals in the session, and converting the result to a percentage. Mean interobserver agreement was 99% (range, 85% to 100%) for Peter, 99% (range, 99% to 100%) for Greg, 99% (range, 97% to 100%) for Vern, 99% (range, 96% to 100%) for Jolinda, 96% (range, 80% to 100%) for Annie, 98% (range, 89% to 100%) for Karen, and 98% (range, 91% to 100%) for Asa.

General procedures

Procedures were similar to those described by Iwata et al. (1982/1994; contact the authors for a detailed description of procedures for each condition). All eight participants were exposed to three test conditions (alone or no interaction, attention, and demand) and a control condition (play), presented in a multielement format. Six of the participants (excluding Greg and Vern) also were exposed to a tangible condition. Each session lasted 10 min. One to six sessions were conducted per day in the following order: alone or no interaction, attention, play, tangible (if relevant), and demand. Sessions were conducted at the same time each day, 3 to 5 days a week, and the number of sessions conducted each day was arranged so as to start with a different session on successive days (i.e., no day ended with a complete cycle through conditions) so that sequencing patterns were unlikely to develop. Graduate students who were trained in facility protocols for management of aggression, protection of human subjects, and cardiopulmonary resuscitation served as therapists. Because of the high intensity of his SIB, an FA of precursor behavior was conducted for Peter (). Experimental contingencies were in effect for his precursor behavior (head up) during the analysis, and no consequences were provided for SIB. The operant function of his SIB was inferred from outcomes of the precursor assessment.

Table 4

Correspondence Between Anecdotal Assessments and the Functional Analysis

Annie

Annie's FA results are shown in Figure 2. Following one cycle of conditions during which no responding was observed, PAO occurred at consistently high frequencies in tangible sessions (M = 32.5 responses per session) and at lower frequencies in the no-interaction (M = 5.5 responses per session), attention (M = 4 responses per session), play (M = 2.2 responses per session), and demand (M = 2 responses per session) conditions. These outcomes indicated that PAO was maintained by social positive reinforcement in the form of access to tangible items. Outcomes of both the MAS (five of five respondents) and QABF (four of five respondents) corresponded to the FA results.

Greg

Greg's results are shown in Figure 2. Pica occurred exclusively in the alone condition and remained at zero in the attention, play, and demand conditions, suggesting an automatic reinforcement function for his pica. Four of five respondents to the QABF identified a nonsocial function, and five of five respondents to the MAS identified a sensory function, demonstrating perfect correspondence with the results of the FA.

Vern

Vern's FA results are displayed in Figure 2. Pica occurred across all four conditions, but differentiation can be seen from Sessions 12 through 20, with highest levels of pica in the alone condition (M = 8 responses per session), followed by attention (M = 5.1 responses per session), play (M = 3.1 responses per session), and demand (M = 1 response per session) conditions. For both the MAS and QABF, five of five respondents indicated a nonsensory or automatic reinforcement function of Vern's pica; thus, anecdotal assessment results corresponded perfectly with results from the FA.

Jolinda

Jolinda's FA results are displayed in Figure 2. She engaged in more SIB during the alone condition (M = 3.4 responses per session) than during other conditions; however, she also engaged in SIB during other test conditions (tangible M = 0.72 responses per session; demand M = 0.36 responses per session; attention M = 0.18 responses per session; control M = 0 responses per session). No SIB occurred during the final three cycles of assessment. Based on these inconsistent and largely undifferentiated outcomes, the FA does not provide differential support for any account of Jolinda's SIB. These outcomes show no correspondence with those from both anecdotal assessments, in which four of five respondents identified social positive reinforcement in the form of tangible items as the likely maintaining contingency for Jolinda's SIB.

Karen

Karen's FA results are shown in Figure 3. Verbal disruptive behavior occurred exclusively in the demand condition (M = 115 s per session), strongly indicating that her problem behavior was maintained by negative reinforcement in the form of escape from task demands. Results from both the MAS (four of five respondents) and QABF (four of five respondents) corresponded to the FA results.

Asa

Asa's FA results are shown in Figure 3. SIB consistently occurred during a high percentage of intervals in the escape condition (M = 18.7% of intervals) relative to other test conditions. Lower levels of problem behavior were observed in the play (M = 7.2% of intervals), alone (M = 5.5% of intervals), attention (M = 2.8% of intervals), and tangible (M = 1.2% of intervals) conditions. These outcomes suggest that his SIB was maintained by negative reinforcement in the form of escape from task demands. Five of five raters on the QABF identified escape as the primary maintaining contingency; however, the MAS did not produce agreement among four of five raters for Asa's SIB. Therefore, only the results of Asa's QABF corresponded with FA results.

Peter

Results of Peter's precursor FA are presented in Figure 3. Precursor behavior consistently occurred at higher rates in the attention condition (M = 1.7 responses per minute) than in other test conditions. Lower levels of problem behavior were observed in play (M = 1.1 responses per minute), tangible (M = 0.69 responses per minute), no-interaction (M = 0.13 responses per minute), and demand (M = 0.06 responses per minute) conditions. SIB occurred only once during the precursor FA, in the first presentation of the attention condition. Taken together, these results suggest that his SIB was maintained by positive reinforcement in the form of caregiver attention. QABF results showed that four of five raters identified attention as the primary maintaining consequence; however, MAS results showed four of five raters identified a sensory function for Peter's SIB. Therefore, only the results of Peter's QABF corresponded with FA results.