Harris, F.C. and Lahey, B.B. A method to combine event and non-attendance assessments. Journal of Applied Behavior Analysis 1978,11, 523-527. Maxwell, A. E., and Pilliner, A.E. G. Reliability coefficients and agreement for ratings. British Journal of Mathematical and Statistical Psychology 1968,21, 105-116. Mitchell, S.K. Interobserver Accord, reliability and generalization of data collected in observational studies. Psychological Bulletin 1979,86, 376-390.
IOA interval by interval. In short, the interval interval method assesses the proportion of intervals in which both observers agreed to determine whether the target reaction occurred. Note that this implies agreement on attendance and lack of response. This is calculated by adding the total number of agreed intervals to the sum of agreed intervals and divided at regular intervals. Not surprisingly, this approach often leads to high convergence statistics. As Cooper et al. (2007) reports, this is especially true when partial interval recordings are used. In the examples in Figure 2, observers disagree on the first and seventh intervals, resulting in an interval agreement value of 71.4% (5/7). Total duration of IOA. Like the total number of IAOs for event-based data, the total duration of the IOA provides a relatively insensitive measure of observer agreement. The total duration of the IOA brings together all calendars in a cumulative duration for each observer, and is calculated by dissecting the smaller duration by the longer duration.
So the more timings there are, the more possibilities there are for the discrepant data to be masked by this metric. As shown in Figure 3, the recorded durations of the two observers for the second, third and fourth occurrences of the response are essentially discrete. However, each observer`s sums correspond to the others, giving a total duration of 100%. Test s.i.A. IOA. Savvy readers will find that IOA algorithms based on the above events are adapted to free-operator responses, responses that can occur at any time and are not anchored in events, but these measures do not explicitly take into account the experience-based reaction, which measures binary results (e.g. B presence/non-presence, yes/no, on-task/task). Thus, the experimental IOA measures the number of trials with consent divided by the total number of trials. This metric is as strict as the exact approach to the agreement. House, A.E., House, B.J. – Campbell, M.B.
Measures of interobserver agreement: Calculation formulas and effect distributions. Journal of Behavioral Assessment 3, 37-57 (1981). doi.org/10.1007/BF01321350 average duration per deposit. If the number of calendars is high, it is important to limit data aggregation in order to identify possible variations in the permanent data of two observers. The average duration IOA algorithm per deposit achieves this by determining an IOA score for each timing, and then by deifing them by the total number of timings in which the two observers collected data. Note that this approach is similar to the approach described above of partial agreement at regular intervals. In the example of Figure 3, there were 99.7, 2.3, 69.2 and 92.7% approval levels for intervals 1 to 4, respectively. The average of these four levels of the agreement results in an average of 66% per event agreement – a much more conservative estimate than that of the statistics of the total duration of the IOA. The IOA points interval. An approach to improve the accuracy of the agreement between two observers for interval recording is simply to limit agreement analyses to cases where at least one observer has recorded a target response in the meantime.