Pilot Phase

print this section

To address the goals defined during the conceptualization phase, a pilot study was conducted. Items for extended and novel content, as well as improved alignment to the DSM Symptom Criteria, were evaluated in the pilot study samples. The pilot study was administered digitally, either through local or remote administration. Parent/guardian and teacher raters completed the test from an emailed link. Youth raters completed the test digitally, in person, and with supervision from a trained administrator to provide support for the youth if needed. The parent/guardian, teacher, and youth (who were 18 years old) raters provided informed consent to participate in this data collection effort. For youth raters who were 17 years of age or younger, informed consent was obtained from their parent/guardian, and assent was obtained from the youths themselves. All participants received small monetary compensation for their participation. The items and scales were then subjected to empirical evaluation.

Samples

The pilot version of the Conners 4 was administered to the parents and teachers of youth (and to the youth themselves) where the youth did not have a mental health diagnosis (described herein as the General Population samples). The pilot version was also administered to raters of youth (and youth themselves) where the youth did have a mental health diagnosis (described herein as the Clinical samples).

The General Population samples consisted of roughly equal representation by age and by gender of the youth being rated, and representation of broad categories of race/ethnicity, geographic region, and parental educational level (PEL) that closely matched U.S. census proportions at the time of recruitment (within approximately 2% of the proportions derived from the American Community Survey 2016; U.S. Census Bureau, 2017). A description of the rated youth in the pilot samples is presented in Table 6.1 and a description of the raters is provided in Table 6.2. These samples (N = 518 for Parent, N = 490 for Teacher, N = 388 for Self-Report) were drawn from the general population and individuals were eligible to participate if the youth being rated had never been diagnosed with a psychological or psychiatric disorder. For the Parent sample, 56.2% reported being the child’s biological mother, 35.7% reported being the child’s biological father, and the remaining 8.1% identified as non-biological parents, legal guardians, and other relatives. For the Teacher sample, all of the raters reported that they knew the student for longer than one month, 97.0% interacted with them weekly or more often, and all teachers knew the student well (either moderately or very well). Teachers reported providing instruction for a wide range of classes, including math, English, history, and music.

The Clinical samples (N = 349 for Parent, N = 242 for Teacher, N = 245 for Self-Report) gathered ratings of youth with a confirmed mental health disorder (see Table 6.3 for a detailed description of the youth being rated, and Table 6.4 for information about the raters). Clinical cases were recruited directly through clinicians, who provided details of the diagnoses to confirm their status. In particular, the sample predominantly featured youth with a diagnosis of one of the three presentations of ADHD (45.9% for Parent, 44.7% for Teacher, and 40.7% for Self-Report). Other clinical disorders, such as Oppositional Defiant Disorder, Conduct Disorder, Generalized Anxiety Disorder, and Major Depressive Disorder, were also represented in this sample. Many youth in the Clinical sample reported having more than one mental health diagnosis. Youth in the Clinical sample varied with respect to whether they were currently taking medication to treat their disorders. For those who reported taking medication, they were not asked to stop taking their medication. For the Parent sample, 81.9% were biological mothers, 8.0% were biological fathers, and the remaining 9.8% included non-biological parents, legal guardians, and other relatives. For the Teacher sample, all raters reported that they knew the student for longer than one month, 97.9% interacted with them weekly or more, and 96.3% knew the student well (either moderately or very well). Teachers reported providing instruction for a wide range of classes, including math, English, history, and music, as well as special education and resource room (i.e., a special education program in a separate setting or a smaller designated room).


Click to expand

Table 6.1. Demographic Characteristics of Rated Youth: Conners 4 Pilot Study General Population Sample

Demographic

Parent

Teacher

Self-Report

N

%

N

%

N

%

Age (in years)

6 to 7

85

16.4

79

16.1

8 to 9

85

16.4

81

16.5

74

19.1

10 to 11

91

17.6

82

16.7

76

19.6

12 to 13

85

16.4

83

16.9

79

20.4

14 to 15

90

17.4

83

16.9

77

19.8

16 to 18

82

15.8

82

16.7

82

21.1

Gender

Male

261

50.4

243

49.6

197

50.8

Female

257

49.6

247

50.4

191

49.2

U.S. Race/Ethnicity

Hispanic

116

22.4

117

23.9

95

24.5

Asian

24

4.6

26

5.3

21

5.4

Black

69

13.3

63

12.9

53

13.7

White

283

54.6

258

52.7

197

50.8

Other

26

5.0

26

5.3

22

5.7

U.S. Region

Northeast

90

17.4

77

15.7

56

14.4

Midwest

119

23.0

108

22.0

86

22.2

South

180

34.7

168

34.3

142

36.6

West

129

24.9

137

28.0

104

26.8

Parental Education Level

High school diploma or lower

167

32.2

153

39.4

Some college or associate’s degree

174

33.6

120

30.9

Bachelor’s degree or higher

177

34.2

115

29.6

Total

 

518

100.0

490

100.0

388

100.0


Click to expand

Table 6.2. Demographic Characteristics of Raters: Conners 4 Pilot Study General Population Sample

Rater Demographic

Parent

Teacher

N

%

N

%

Gender

Male 

206

39.8

133

27.1

Female 

312

60.2

357

72.9

U.S. Race/Ethnicity 

Hispanic 

115

22.2

42

8.6

Asian 

28

5.4

23

4.7

Black 

70

13.5

21

4.3

White 

289

55.8

389

79.4

Other 

16

3.1

15

3.1

Relation to Youth Being Rated 

Biological mother 

291

56.2

Biological father 

185

35.7

Non-biological mother 

11

2.1

Non-biological father 

11

2.1

Other relative 

17

3.3

Other guardian

3

0.6

Length of Relationship 

1–5 months

181

36.9

6–11 months

96

19.6

More than 12 months

213

43.5

How well does the teacher know the student being rated? 

Moderately well 

223

45.5

Very well 

267

54.5

How often does the teacher interact with the student being rated? 

Less than weekly

15

3.1

Weekly 

88

18.0

Almost daily 

387

79.0

Age in years M (SD)

41.9 (9.3)

48.7 (13.1)

Total

518

100.0

490

100.0


Click to expand

Table 6.3. Demographic Characteristics of Rated Youth: Conners 4 Pilot Study Clinical Sample

Demographic

Parent

Teacher

Self-Report

N

%

N

%

N

%

Age (in years)

6 to 7

39

11.2

15

6.2

8 to 9

61

17.5

38

15.7

41

16.7

10 to 11

69

19.8

40

16.5

45

18.4

12 to 13

57

16.3

38

15.7

39

15.9

14 to 15

58

16.6

45

18.6

49

20.0

16 to 18

65

18.6

66

27.3

71

29.0

Gender

Male

188

53.9

129

53.3

114

46.5

Female

161

46.1

113

46.7

131

53.5

U.S. Race/Ethnicity

Hispanic

43

12.3

13

5.4

22

9.0

Asian

5

1.4

2

0.8

2

0.8

Black

34

9.7

35

14.5

32

13.1

White

254

72.8

181

74.8

174

71.0

Other

13

3.7

11

4.5

15

6.1

U.S. Region

Northeast

49

14.0

77

31.8

35

14.3

Midwest

132

37.8

94

38.8

113

46.1

South

139

39.8

61

25.2

96

39.2

West

7

2.0

7

2.9

0

0.0

Parental Education Level

High school diploma or lower

86

24.6

89

36.3

Some college or associate’s degree

92

26.4

49

20.0

Bachelor’s degree or higher

170

48.7

106

43.3

Diagnosis

ADHD Inattentive

60

17.2

46

19.0

50

20.4

ADHD Hyperactive/Impulsive

11

3.2

4

1.7

7

2.9

ADHD Combined

89

25.5

58

24.0

43

17.6

Disruptive Disorders

28

8.0

22

9.1

24

9.8

Generalized Anxiety Disorder

68

19.5

40

16.5

58

23.7

Major Depressive Disorder

41

11.7

24

9.9

38

15.5

Other Diagnoses

52

14.9

48

19.8

25

10.2

Total

349

100.0

242

100.0

245

100.0

Note. There were 21 omitted responses for U.S. Region and 1 omitted response for parental educational level (Parent form). There were 3 omitted responses for U.S. Region (Teacher form). Disruptive Disorders include Oppositional Defiant Disorder and Conduct Disorder.


Click to expand

Table 6.4. Demographic Characteristics of Raters: Conners 4 Pilot Study Clinical Sample

Rater Demographic

Parent

Teacher

N

%

N

%

Gender 

Male 

31

8.9

67

27.7

Female 

318

91.1

175

72.3

U.S. Race/Ethnicity 

Hispanic 

40

11.5

11

4.5

Asian 

4

1.1

1

0.4

Black 

35

10.0

24

9.9

White 

248

71.1

202

83.5

Other 

0

0.0

1

0.4

Canadian Race/Ethnicity 

Not visible minority 

21

6.0

3

1.2

Visible minority 

1

0.3

0

0.0

Relation to Youth Being Rated 

Biological mother 

286

81.9

Biological father 

28

8.0

Non-biological mother 

15

4.3

Non-biological father 

1

0.3

Other relative 

11

3.2

Other guardian

7

2.0

Length of Relationship 

1–5 months

43

17.8

6–11 months

118

48.8

More than 12 months

81

33.5

How well does the teacher know the student being rated? 

Not very well

9

3.7

Moderately well 

113

46.7

Very well 

120

49.6

How often does the teacher interact with the student being rated? 

Less than weekly

4

1.7

Weekly 

32

13.2

Almost daily 

205

84.7

Age in years M (SD)

42.0 (8.0)

40.3 (9.3)

Total

349

100.0

242

100.0

Note. There was 1 omitted response for "relation to youth being rated" (Parent form) and 1 omitted response for "how often does the teacher interact with the student being rated" (Teacher form).

Analyses & Results

The items from the Content Scales, Impairment & Functional Outcome Scales, and DSM Symptom Scales were evaluated using both classical test theory (CTT) and item-response theory (IRT) frameworks. Items were first reviewed via (a) response frequencies by clinical and demographic group, to ensure all response options were endorsed and to understand the distribution of responses; (b) inter-item correlations to ensure relatedness of content; and (c) group differences to ensure the items appear to behave in hypothesized ways (e.g., individuals diagnosed with ADHD Combined Presentation were expected to endorse the Hyperactivity and Impulsivity items more so than individuals with ADHD Inattentive Presentation).

Items were then subjected to exploratory factor analyses (EFA) to understand the latent structure of the item pool across all three rater forms (i.e., Parent, Teacher, and Self-Report) and for purposes of dimension and item reduction. The tentative factor structure was incorporated into further analyses for the inspection of internal consistency, inter-scale correlations, and item-to-scale correlations.

The results, including the structure of the scales and item-level analyses, were weighed alongside clinical relevance to determine the set of best performing items that would be carried through to the next phase of development. Items were flagged for removal if

  • non-negligible demographic differences were observed;
  • items did not relate well to their intended factor;
  • expected differences between clinical groups were not notable, or did not adhere to hypotheses (e.g., if an item written to measure inattentive behavior was rated similarly both by individuals with ADHD and those with no clinical diagnosis); or
  • either item-to-scale-total correlations or item communalities were low (i.e., an indicator of the unique contribution of an individual item to its scale; low values defined as < .20; Kline, 1986).

Additionally, IRT analyses were used to ensure items were kept that provided sufficient precision of measurement across the latent trait continuum with emphasis and preference given to items that provided measurement precision at higher trait levels. Poorly performing items were either removed or revised in instances where a small change could be applied. In addition, the Critical & Indicator Items, along with items from the different validity metrics (i.e., positive impression, negative impression, and ADHD SVT) were all retained without changes for further investigation at the next stage of development. At the completion of the pilot phase, there were 269 items on the Conners 4 Parent, 256 items on the Conners 4 Teacher, and 290 items on the Conners 4 Self-Report.


<< Conceptualization, Initial Planning, and Item Development Standardization Phase >>