boss observation manual
LINK 1 ENTER SITE >>> Download PDF
LINK 2 ENTER SITE >>> Download PDF
File Name:boss observation manual.pdf
Size: 2999 KB
Type: PDF, ePub, eBook
Category: Book
Uploaded: 8 May 2019, 16:14 PM
Rating: 4.6/5 from 600 votes.
Status: AVAILABLE
Last checked: 8 Minutes ago!
In order to read or download boss observation manual ebook, you need to create a FREE account.
eBook includes PDF, ePub and Kindle version
✔ Register a free 1 month Trial Account.
✔ Download as many books as you like (Personal use)
✔ Cancel the membership at any time if not satisfied.
✔ Join Over 80000 Happy Readers
boss observation manualHowever, they pose challenges for training and attaining high levels of interrater reliability between observers. Reliable use of the BOSS for clinical or research purposes requires training to reach reliable standards (kappa ? 0.80). The current study conducted training observations in one suburban and one urban elementary school in the Greater Boston area. To enhance interrater reliability and reduce training time, supplemental guidelines, including 30 additional rules to follow, were developed over two consecutive school years. The complete protocol was then used for training in the third school year. To reach sufficient interrater reliability (kappa ? 0.80) during training, 45 training observations were required in the first year while, in the third year, only 17 observations were required. High interrater reliability was sustained after training across all three school years, accumulating a total of 1,001 post-training observations. It is estimated that clinicians or researchers following this proposed protocol, who are naive to the BOSS, will require approximately 30 training observations to reach proficient reliability. We believe this protocol will make the BOSS more accessible for clinical and research usage, and the procedures used to obtain high interrater reliability using the BOSS are broadly applicable to a variety of observational measures. Download full-text PDF Reliable use of the BOSS for clinical or research purposes requires training to reach reliable standards (kappa ? 0.80). The current study conducted training observations in on e suburban and one urban elementary school in the Greater Boston area. T o enhance interrater reliability and reduce training time, supplemental guidelines, in cluding 30 additional rules to f ollow, were developed over two consecutive school years. T o reach sufficient interrater reliability (kappa ? 0.http://domarcas.com/img/userfiles/boss-rv-3-manual.xml
- Tags:
- boss observation manual, boss observation manual pdf, boss observation manual download, boss observation manual 2017, boss observation manual free.
80) during training, 45 training observations were required in the first year while, in the third year, only 17 observations were required. High interrater reliability was sustained after training across all three school years, accumulating a total of 1,001 post- training observations. It is estimated that clinicians or researchers following this proposed protocol, who are naive to the BOSS, will require approximately 30 training observations to reach profi- cient reliability. W e believe this prot ocol will make the BOSS more accessible for clinical and research usage, and the procedures used to obtain high interrater reliability using the BOSS ar e broadly applicable to a variety of observational measures.Behavioral Observation of Student s in Schools (BOSS). Classroom observation. Behavior observation 1 Introduction Students with academic or behavioral concerns in school are assessed using a wide range of instruments such as behavior rating scales, semi-structured interview s, and direct observations. Direct observation methods are demanding in that they require trained field researchers to carry them out. Training methods are not always well documented, and techniques for obtaining high levels of inte rrater reliability, a critical characteristic o f any observational techn ique, may be elusive. Thus, we demonstrate methods for training field researchers to administer an observational tool, such as the BOSS, working toward a high level of interrater agreement using the robus t kappa statistic. Our methods include close study of formal definitions, coding of practice video and in-class observations, rapid quantitative analysis of observation data through a programmed Interrater Agreement Calculator, and immediate post-observation discussion.http://fatihbozkaya.com/media/boss-rv-5-manual.xml In general, direct observations are used for a number of assessment tasks such as screening for emotional and behavioral problems and monitoring interventions, thus serving as an objective measure for both educators and researchers (Riley-Tilman et al. 2005 ). Direct observation o f children in their classroom is one of the most common assessment strategie s used by school psychologists (Shapiro an d Heick 2004; Wilson and Reschly 1996 ) as it allows for measurement of a behavior as it occurs naturally, without bias of a parent or teacher. Direct observation assessments can be separated into two categories: naturalistic observ ations with anecdotal de- scriptions or systematic observations with interval coding. Naturalistic observations do not allow for standardization or psychometric testing of the assessment tool (Hintze and Matthews 2004; Hintze et al. 2008 ). In contrast, systemat ic observations use an interval recording format and provide quantitative data, which can help reduce observer bias. Furthermore, the interval recording form at is useful for systematic training of observers using a detailed protocol and checking for reliability between observers. This article exclusively addresses the training of systematic direct observations of children in their classroom. Although systematic observ ations use an interval record- ing format, observer bias can still occur and affect the accuracy and objectivity of a direct observation. Observer bias refers to the tendency of an observer to consistently view and record observed behaviors in a particular way (either negatively or Educ Asse Eval Acc A bias in student observations from a clinician could lead to faulty recommendations. For instance, if an observ er perceives a child to be very distracted and freq uently off-task, that child could be recommended for unneeded preferred seating. A biased researcher could lead to systematic error s and faulty research conclusions.http://www.raumboerse-luzern.ch/mieten/bosch-she33m05uc-manual The lifetime prevalence of ADHD for children and adolescents is estimated at 9 (Merikangas et al. 2010 ). High rates of off-task, disruptive, noisy, and rule-violating behavior displayed by children with ADHD often occur in school settings. A study by Vile-Junod et al. ( 2006 ) found that children with ADHD from ages 6 to 10 years exhibited signifi- cantly decreased rates of academic engagement than their compariso n peers when directly observed. More specifically, children with ADHD are observed to display higher rates of: physical and verbal aggression, seeking attention, out-of-chair time, and noncompliance (Vile-Junod et al. 2006 ). The serious nature of ADHD symptoms in the classroom and their negative effects on academic progress (Mautone et al. 2005 ) indicates the need of a preci se assess- ment of student behavior to allow for the development of accurate treatment plans. The accepted battery of assessment tools for diagnostic or follow-up evaluations of ADHD includes behavior rating scales, semi-structured interviews, and direct obser- vations (Atkins and Pelham 1991; DuPaul et al. 1992; DuPaul and Stoner 1994; Montague et al. 1994 ). Assessments from a subjective party, such as a parent or teacher, may be biased in terms of the types of behaviors they report. A study by Hartman et al. ( 2007 ) found that parents can be more biased than teachers when rating ADHD symptoms. Teachers generally have access to many normal students with whom they can compare a target student. On the other hand, parents commonly have only their own child(ren) with whom to compare the target child. Furthermore, ADHD symptoms may be more appare nt to teachers as the school environme nt requires more sustained attention, grea ter independent completion of tasks, and higher activity and impulse control than the home environment (Hartman et al. 2007 ).http://drbillbaker.com/images/boss-observation-manual.pdf The BOSS is one example of an objective observation system for coding class- room behavior (Shapiro 2011 ) and has been found reliable between observers (Volpe et al. 2005 ). In addition, the BOSS has also been shown to differentiate between children with ADHD and their typically developing peers (i.e., children who were reported by their teachers to be average in terms of classroom behavior and academic achievement). Thus, the BOSS is well adapted to measure ADHD symptoms in the classroom setting. However, even when using objective assessments, errors can still occur. Merrell ( 1999 ) reported six categories of coding errors that can threaten the validit y of observation data: Educ Asse Eval Acc In other words, an observer ’ s expectation of a specific behavior to occur may result in a biased observation. We believe coding errors in these six categories can be reduced through a precise training protocol. Such a protocol is also imperative to track and reach high levels of interrater reliability during training. Requiring high levels of inte rrater reliability assures that the data collected from a direct observation does not depend on who conducted the observation but on the predefined standardized coding scheme. Interrater reliability can be calculated as the agreement between two observers using two calculation methods: percent agreement and kappa score. In general, a percent agreement of 80 represents reasonable reliability (Miles and Huberman 1994 ). For example, in a study done by Mautone et al. ( 2005 ), a video tape of a classroom was used to train observers on the BOSS until they reached a percent agreement of 80 in each domain. After the trainin g, percent agreement was calculated for 19.5 of the observation sessions with a maintained mean percent agreement of 94.8 and a range between 86.7 and 99.0 . While percent agreement is an accepted calcul ation for interrater reliability, a more robust calculation is the kappa score (Hintze 2005 ).https://stellabakingcompany.com/wp-content/plugins/formcraft/file-upload/server/content/files/1626f5426b660e---bosch-wff2000-washing-machine-manual.pdf The kappa score is considered to be more robust because it also considers and corrects for the random chance that two observers would code the same behavior by accident. For instance, two observers could have a percent agreement score of 98.5 but a kappa score of 0.00 because a behavior happened during one interval of the 15-min observ ation and they did not agree on that one observation interval. Therefore, it is more difficult to reach a high kappa score than it is to have a high percent agreement. Hintze ( 2005 ) states a kappa score is calculated as: Educ Asse Eval Acc A kappa of 1.00 signifies perfect agreement, whereas a kappa of 0.00 signifies agreement equiv- alent to chance, and a kappa less than 0.00, which happens very rare ly, signifies agreement less than expected by chance. In general, a kappa from 0.21 to 0.40 is considered fair, 0.41 to 0.60 moderate, 0. 60 to 0.80 substantial, and finally 0.81 and above as almost perfect (Gelfand and Hartmann 1975; Landis and Koch 1977 ). The BOSS has been found to yield high kappa scores during training as compared with other direct observation measures. The BOSS developer (Shapiro 2011 ) reports training for proficiency takes 10 to 15 h, which is shorter when compared with other measures suc h as the Classroom Observation Code, which takes 50 h of training to develop profi- ciency (V olpe et al. 2005 ). Although a short manual has been published, a systematic stepwise training protocol is not currently available. The BOSS was used in the current study to observe st udents with ADHD in the context of a school-based intervention; therefore, the observations were in the same context as that of many school psychologists evaluating children with ADHD. However, it was found that training by using the BOSS Manual (Shapiro 2011 )w i t h video and in-class practice observations alone did not enable the research team to reach high interrater reliability on training during the first school year.www.cocuknorologu.com/image/files/compaq-f700-service-manual.pdf Supplemental specifications in additio n to the BOSS Manual behavior criteria were needed in order to avoid coding errors associated with inad equate training and to achieve high reliability indices. T herefore, Supplemental Guidelines for training were created at the beginning of the first and second school year, followed by the use of this completed BOSS training protocol for training in the beginning of the third (last) school year. This paper describes an in-depth, standardized, and user-friendly BOSS training protocol that has been developed by this research team to serve as a training guide, complementing the published BOSS Manual (Shapiro 2011 ), for clinicians and educational researchers. For the current research group, the use of t his protocol led to high interrater reliability scores (percent agreement and kappa scores; see “ Results ” section) and successful standardization for the continued use of the BOSS. Educ Asse Eval Acc T raining incl uded two steps: a practice video and furt her practice in a classroom setting. Dur ing training Round On e, two school psyc hology gradua te students who had already rece ived formal trai ning on the BOSS throug h review of the BOSS Manu al (Shapiro 20 11 ) and cond ucting prac tice video obse rvations in a co urse instru cted by Robert V olpe became obse rvers for our study. A thir d observer, chosen to be the lead observer for lo gistical reas ons, complet ed an in-depth lit erature review of th e BOSS and its training in Roun d One, and became the BOS S trainer. Through discus sions with other observers ab out repeated co ding discre pancies due to ambi guous guidelines, the lead observer co mpiled the Sup plemental Gui delines and tr ained furthe r observer s for Round Tw o and Round Three. All of the subseq uent traine e observers we re naive to the BOSS method; howeve r, all obser vers had prior exp erience working wit h children.https://www.superioreagle.com/wp-content/plugins/formcraft/file-upload/server/content/files/1626f5434ca5bd---bosch-wff1800-manual-english.pdf Some observers had pa st experience wo rking with childr en with ADHD, while othe rs did not. Practice tr aining observations were co nducted in rand omly selected el ementary school classr ooms in one suburb an (Round One) and one ur ban (Round T wo and Round Three) school in th e Greater Boston area. O bservers were wel l-trained in ef forts to avoid bias. They random ly selected targe t students to observe; this means they were not informed wh ether childr en they were obse rving had been diagnosed with ADHD or any other diso rder. Furthermor e, they were inst ructed to avoi d selecting st udents from any particul ar demograp hic group (e.g., race or ethnicity). The de velopment of th is enhanced, mo re accessib le, and clarif ied BOSS train ing protocol was carrie d out over 3 years for a lar ger study. 2.2 Materials This enhanced BOSS training protocol simplifies the flow of the training and includes the use of the following materials: 2.2.1 Manual for the Behavioral Observation of Students in Schools (BOSS) The BOSS Manual (Shapiro 2011 ) offers a solid foundation for BOSS training. It provides the rationale for using the observational tool and a descripti on of the five behavioral categorie s for observation (Table 1 ) and the teacher and peer cod ing (Table 2 ). The BOSS defines classroom engagement as the desired behavior, repre- sented by on-task active (AET) and on-task passive (PET) behaviors. Impulsivity and hyperactivity are measured by off-task motor (OFT-M) and verbal (OFT -V) behav- iors, while inattention is quantified by the frequency of off-task passive (OFT-P) behaviors (Table 1 ). In addition, the BOSS Manual provides instruction in the completion of the BOSS Observation Form (Shapiro 2011 ) and the development of a coding interval audiotape. The BOSS Observation Form assesses the number of on- task and off-task behaviors a child exhibits over a 15-min observation period.http://www.stockholmswingallstars.com/wp-content/plugins/formcraft/file-upload/server/content/files/1626f54401fe52---bosch-wfg-2060-service-manual.pdf The observations can be done for up to 30 min; however, most published papers report the Educ Asse Eval Acc However, there are data supporting the ability of the BOSS to discriminate between children with ADHD and typically developing children (DuPaul et al. 2004 ). 2.2.2 BOSS Observation Form The BOSS Observation Form (Shapiro 2011 )i si n c l u d e di nt h eB O S SM a n u a l (Shapiro 2011 ) and is used to record the occurrence of the five behavior categories Table 1 Description of BOSS behavior categories and classroom settings categories (Shapiro and Heick 2004 ) Category Description On-task (momentary time sampling) Active engaged time (AET) Student is actively attending to assigned work, e.g., writing, reading aloud, or raising hand. Passive engaged time (PET) Student is passively attending to assigned work, e.g., listening to a lecture, looking at an academic worksheet, or silently reading assigned material. Off-task (partial-interval method) Off-task motor (OFT-M) Any instance of motor activity that are not directly associated with the assigned academic task, e.g., engaging in any out-of-seat behavior (defined as buttocks not in contact with the seat), aimlessly flipping the pages of a book, or manipulating objects not related to the academic task (e.g., playing with a paper clip, throwing paper, twirling a pencil, folding paper). Off-task passive (OFT-P) Times when a student is passively not attending to an assigned academic activity for a period of at least three consecutive seconds. Includes when a student quietly waits after the completion of an assigned task, but is not engaged in an activity authorized by the teacher, e.g., sitting quietly in an unassigned activity, looking around the room, or staring out the window. The teacher is either doing work individually at a desk or rotating around the classroom. The teacher is working with children in a small group (eight children or fewer).cnsilos.com/d/files/concrete-and-clay-tile-installation-manual-for-moderate-climate-regions.pdf Small group: Teacher present (SmGp:TPsnt) Target child is doing work in a small group (eight children or fewer). The teacher or an assistant teacher may be working with the group or just be present in the classroom. Large group: Teacher present (LgGp:TPsnt) Target child is doing work in a large group (i.e., whole classroom or a group with nine or more children). The teacher is present in the room instructing the class. Educ Asse Eval Acc There are 60 intervals over the 15-min observation, and each interval contains five boxes, corresponding to the five behavior categories, to record on-task and off-task behaviors. Once the observation begins, when an on-task or off-task behavior is observed, a check is placed in the designated box for that interval. Both engaged behaviors (AET and PET) are scored at the beginning of each 15-s interval; this is referred to as momentary time sampling. During the remainder of each interval, off-task behaviors are recorded if the event occurs within that interval; this is referred to as partial-interval method. At every fifth interval, teacher-directed instruction and classroom peer behaviors are coded instead of the target child ’ s behaviors (Table 2 ). Amendments were made to this BOSS Obse rvation Form to include additional helpful information (see “ Procedure ” ). 2.2.3 Practice video Robert V olpe provided a DVD with a 15-min training video of seven students and one teacher in an urban classroom setting. The video was used to train observers concerning the development of: familiarity with the five behavior categories, fluency on the BOSS Observation Form, and familiarity with the Supplem ental Guidelines. The video shows one teacher providing direct instruction for the entire class and time for independent seatwork. 2.2.4 Coding interval audio Robert V olpe provid ed the coding in terval audi o, developed fo llowing the directions in the BOSS Manual ( Shapiro 2011 ), whi ch is an mp3 file that prompts ob servers every 15 s through an ear piec e by calling out the nu mber of the observat ion ( “ Observ ation 1, ” “ Observat ion 2, ” etc.), so th at observers do no t to have to simultan eously look at a watch. This al lows observ ers the abilit y to focus on clas sroom activi ties and codi ng. The audio is also requ ired to synchron ize coding for a se ssion with multip le observer s. Table 2 BOSS coding instructions (Shapiro and Heick 2004 ) Aspect of the BOSS Description Observation time Either 15 (60 intervals) or 30 (120 intervals)-min periods. Peer coding Code for a peer in the classroom every fifth interval (e.g., interval 5, 10, 15) using the behavior coding used for the target child (see Table 1 ). An observer can either pick three peers to alternate between ahead of time or go in order around the classroom observing peers. Teacher-directed instruction (TDI) Code for TDI every fifth interval (same interval classroom peer is coded). Coded for if the teacher, or an assistant teacher, is giving academic instruction. This can be in a large group setting, small group setting, or one- on-one. Educ Asse Eval Acc They inc rease the standardization of the BOSS training and therefore increase inte rrater reliability and decrease length of training. These guidelines are broken down into general informa- tion guidelines and guidelines for each category of the BOSS. 2.2.6 Interrater Agreement Calculator We developed the Interrater Agreement Calculator in Microsoft Excel to use in the training and fidelity procedure. After completing an observation, observe rs enter data from the BOSS Observation Forms into the Excel file, which takes less than 1 min. This serves two purposes. First, a formula in the Excel file highlights coding boxes that differ between observers, allowing observers to easily identify and discuss discrepancies. This is an essential step in reducing repeated coding errors and improving scores on the next observation. The completed materials were used in training Round Three. The following material s are explained below: 1. Amended BOSS Observation Form: The scoring section at the bott om of the observation form was removed, and its ’ function was replaced by the Interrater Agreement Calculator. Four sections that were found helpful when conducting an observation were then added. Furthermore, this space can be used to record changes in the academic subject or classroom setting (see Table 2 ) and at which interval this took place. Educ Asse Eval Acc A review of BOSS Observation Forms was conducted to analyze where repeated errors were the most frequent. This review showed that off-task behaviors were the least specific and the hardest to code accurately. Based on these findings, a list of 30 additional guidelines pertaining to general information as well as the five behavioral categories was compiled (Table 3 ) to remedy these recurrent unclear areas. These additional guidelines decreased training time in Round Two, where some further clarifications were also added. In Round Three, however, no further changes were required. 3. Interrater Ag reement Calcul ator: The Interrater Agreem ent Calculato r was created in Microsoft Ex cel during Round On e to compare observat ions between observers and to calcul ate percent ag reement and ka ppa scores. It ta kes approxim ately 1 min to enter each obse rvation. (See se ction on “ Interr ater Agreement Calc ulator ” ). 2.3.2 Observations and post-observation discussions During training, two or more observers pa rticipate simultaneously for all train- ing observations (double observations), where the lead observer is always present (for seating arrangements, see “ In-class practice observations ” ). Reli- ability scores are calculated between the lead observer and each of the trainee observers. Double observations are used throughout research and clinical ob- servations to ensure continued reliability. Each tra ining observation includes the 15-min observation along with a 15-min p ost-observation discussion, which is just as important for training as the observation itself. Educ Asse Eval Acc This procedure decreases the total numbe r of in- class training observations needed, which can be a burden to the school system, classrooms, and teachers. Before beginning the practice video observations, observers review the BOSS Manual (Shapiro 2011 ) and the Supplemental Guidelines. They then learn how to record information on the BOSS Observation Form and how to code the five behavioral categories. Obs ervers select a specific target student in the video to observe and select the peer students to be observed every fifth interval (Table 2 ). For each subsequent practice video observation, the same video is used, yet a different target student and different peers are observed each time. While watching the video and listening to the coding inte rval audio, observers fill out the BOSS Observation Form. As stated above, observers shoul d separately enter the BOSS Observation Form into the Interr ater Agreement Calculator and r eview which observat ions they have coded di fferently. It is possible to review discrepant intervals of the video during the post- observation discussion, which can be helpful in the beginning. Particular attention to these discussions, which take an average of 15 min, will ultima tely lead to an increased understanding of the coding categories and to higher kappa scores on subsequent observations. This process continues until a kappa score of around 0.70 is reached on all three kappa calculations, when competency and fluency in filling out the BOSS Observation Form is achieved. Observers then proceed with practice in-class observations. 2. Practice in-class observations: The goals of the practice in-class observatio ns are to precisely record behaviors in a variety of classroom activities and to obtain a kappa score of 0.80 or higher. After observers reached kappa scores of 0.70 on the practice video observations, a nearby school was contacted and asked if observers could complete practice observations in their classroom. Since observ - ing in schools may be seen as a burden to some teachers or school officials, if a specific teacher is willing to have multiple observations in his or her classroom, the observers should opt to train in a classroom that is amenable to their presence and select a different target student for each observation. In this case, the diversity of settings and grades for practice observations comes seconda ry to the observations going smoothly, as discussing the coding discrepancies between observers is the most beneficial piece of training, regardless of the classroom setting. These in-class observations give observers the opportunit y to apply the knowledge and experience they gained through coding and discussing the prac- tice video to a live classroom setting, also giving observers the opportunity to refine their skills and become familiar with what a real classroom environment is like. Upon entering the classroom, observers select a student at random for their target child and pick the best vantage point possible, which is an extremely important step. It is ideal for observers to select a position that allows a full view for all observers of the child ’ s face, arms, legs, and eyes and is not blocked by the teacher moving around. Additionally, a position not directly in front of the target child is recommended, so that the student is not aware of being observed, even Educ Asse Eval Acc Observers then fill out the i dentifying information on the BOSS Observation Form, recording the child ’ s gender and color of clothing in place of child ID. Peer selection is important in order to ensure that the observers code for the same peer child for the peer comparison even if the children get up and move during the observation. Thus, obse rvers typically choose three easily visible peers and rotate between them in the same order for every fifth observ a- tion. Observers then start the observation, listening to the coding interval audio on an mp3 device with an earpiece in only one ear, enabling observers to listen to both the audio and to what is happening in the classroom. During observations with two observers, each observer listens to one of the two earpieces to ensure that they are both coding in synch. An adapter that allows two sets of headphones to be plugged into the same device can be used for three observers. As part of the amended BOSS Observation Form des cribed above, if the classroom setting changes within the 15-min period, the observation number at which the setting changed, and the setting the classroom was changed to is recorded (e.g., large group teacher present to independent seatwork teache r present at observation 15). Similarly, if the academic subject changes, the observation interval at which it changed, and the academic subject the classroom changed to is recorded. Fur- thermore, if there is a long transition time or if the target student leaves the classroom, the audio is paused, and observers record this in the notes section of the BOSS Observation Form.