Post on 29-May-2018
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 1/39
The Case of De La Salle-Santiago
Zobel School (DLSZ)
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 2/39
Shift from Classical Test Theoryapproach to Item Response Theory
(IRT)
IRT was introduced to DLSZ in School
Year 2008-2009 by
Dr. Carlo Magno.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 3/39
Curriculum-based Achievement tests
Grade/Year Levels
Senior Prep
Grades 1 to Grade 7
First year to Fourth year high school
Subject Areas
English Math
Filipino Science
Christian Living Social Studies
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 4/39
Two (2) forms of the achievementtests were created
Experts in the subject areas content
validated and externally reviewed
the test items and the curriculum
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 5/39
Subject Area Teachers¶ workshop wereconducted
Teachers /Subject Area Coordinators
revised some items
Consultants reviewed revisions via email
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 6/39
Test item pool and TOS to IPA
Orientation of Teachers for the pilot testing
Pilot testing for all levels
Gathering of test data from the pilot tests
Statistical analysis of test items
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 7/39
` Descriptive statistics
` Mean and standard deviations were obtained to
determine the central tendency and variability of
the scores for each form of the tests.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 8/39
Two ways:
` Internal consistency ± Cronbach¶s alpha value
of .70
` Parallel forms of the test
-Pearson r was used to correlate the
scores of each examinee for both forms.
-All correlation coefficients obtainedwere significant (p<.05)
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 9/39
` Content validity
The items were reviewed by experts in the
field based on its intended skills and content.
` Concurrent validity
Each form of the test was correlated with
each other. Majority of the correlation
coefficients were significant.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 10/39
` The one-parameter Rasch model was used todetermine the difficulty and fit of the items.
Item difficulty
` Positive logit values indicate that the
item is difficult and negative logit values
indicate that the item is easy.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 11/39
SY µ08-¶09 SY ¶09-¶10
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 12/39
Item Fit
Means Square Values (MNSQ) within 0.8 to 1.2 are
said to have good fit.
Bad items are determined if the ability of the persons
do not match difficulty of the items.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 13/39
Legend: - -
- - 1
ITEM FIT RESULTS
ENIOR PREP
FormNo. of
Items
No. of Good Fit
Item/s
No. of Bad Fit
Item/s
N MEAN TDEV PROPORTION
CORRECT
P 3 1 33 1 .16 1 .4 4.67 4. .64 .65
FORM SY2008-2009 taken by the Upper Level (PILOT TESTING)
SY2008-2009 SY2009-2010
Seni r r ep 7, , , 7, , 19, 18, 18,
FORM No. of Items
SY2008-2009 SY2009-2010
No. of Difficult
Items
No. of Easy
Items
No. of Difficult
Items
No. of Easy
Items
SP 30 16 14 16 14
FORM DIFFICULT to EASY items EASY to DIFFICULT items
SP 23, 28 2, 5
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 14/39
Legend: - SY 2008-2009
- SY 2009-2010
ENGLISH
Form
No.
of
Items
No. of
Good Fit
Item/s
No. of
Bad Fit
Item/s
N MEAN STDEVPROPORTION
CORRECT
Engl 1 45 42 45 3 - 215 241 31.51 26.60 6.52 7.45 0.70 0.59
Engl 2 44 43 43 1 1 210 291 29.07 26.53 6.66 6.66 0.66 0.60
Engl 3 60 56 58 4 2 215 328 32.53 31.44 9.55 8.91 0.54 0.52
Engl 4 70 65 69 5 1 265 313 37.48 40.23 11.18 8.02 0.54 0.57
Engl 5 70 62 68 8 2 198 290 39.73 37.77 11.22 10.14 0.57 0.54
Engl 6 75 75 74 - 1 269 285 47.19 42.73 9.34 10.07 0.63 0.57
Engl 7 80 79 80 1 - 292 316 51.42 45.82 10.72 10.66 0.64 0.57
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 15/39
FORM
SY2008-2009 taken by the
Upper Level
(PILOT TESTING)
SY2009-2010
Engl 1 19, 27, 31 NO REVISIONS NEEDED
Engl 2 46 23
Engl 3 9, 10, 21, 40 39, 50
Engl 4 20, 22, 34, 35, 53 34
Engl 5 28, 32, 33, 37, 44, 46, 50, 53 12, 44
Engl 6 NO REVISIONS NEEDED 31
Engl 7 78 NO REVISIONS NEEDED
FORM DIFFICULT to EASY items EASY to DIFFICULT items
Engl 1 19 40, 41
Engl 2 NONE NONE
Engl 3 9, 10, 18, 33, 42, 55 5, 16, 40, 43, 47, 52Engl 4 20, 25,50, 61, 70 27, 52, 53, 60
Engl 5 24, 32, 37, 39, 40, 48, 59, 65, 68 28, 46, 53, 54
Engl 6 29, 42, 53 21, 34, 35, 57, 64
Engl 7 3, 8, 11, 20 37, 55, 74
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 16/39
Form
No. of
Items
SY2008-2009 SY2009-2010
No. of Difficult Items
No. of EasyItems
No. of
Difficult Items
No. of Easy Items
Engl 1 45 21 24 22 23
Engl 2 44 21 23 23 21
Engl 3 60 32 28 32 28
Engl 4 70 36 34 35 35
Engl 5 70 36 34 31 39
Engl 6 75 37 38 39 36
Engl 7 80 39 41 39 41
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 17/39
Legend: - SY 2008-2009
- SY 2009-2010
r
.
f Ite
. f
Good itIte /
. f
d itIte /
S
I
L 1 30 28 29 2 1 198 241 25.69 25.00 2.82 3.17 0.86 0.83
L 2 30 27 29 3 1 179 289 21.16 20.73 4.12 3.84 0.71 0.69
L 3 40 39 39 1 1 255 323 25.58 24.20 5.28 5.24 0.64 0.61
L 4 50 50 49 - 1 283 318 27.25 27.66 6.57 6.31 0.55 0.55
L 5 50 50 48 - 2 282 301 36.15 35.04 5.86 7.29 0.72 0.70
L 6 60 60 59 - 1 285 310 39.56 37.76 6.68 7.76 0.66 0.63
L 7 60 60 58 - 2 233 320 42.14 43.06 6.29 8.05 0.70 0.72
H IS I LI I G
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 18/39
FORMSY2008-2009 taken by the
Upper Level (PILOT TESTING)SY2009-2010
CL 1 22, 23 10
CL 2 2, 13, 25 30
CL 3 12 34
CL 4 NO REVISIONS NEEDED 40
CL 5 NO REVISIONS NEEDED 33, 44
CL 6 NO REVISIONS NEEDED 7
CL 7 NO REVISIONS NEEDED 2, 60
FORM DIFFICULT to EASY items EASY to DIFFICULT items
CL 1 5 28, 13, 14,
CL 2 14, 24 13, 26
CL 3 8, 21, 35, 36 1, 13, 22, 23, 33, 40CL 4 6, 15, 17, 27, 38, 43, 47, 50 9, 12, 14, 44, 46
CL 5 5, 9, 21, 23, 35 14, 17, 24, 33
CL 6 30, 37 4, 25, 40, 58
CL 7 3, 6, 9, 29, 35, 51 4, 5, 14, 16, 47, 57
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 19/39
Form No. of Items
SY2008-2009 SY2009-2010
No. of
Difficult Items
No. of Easy
Items
No. of Difficult
Items
No. of Easy
Items
CL 1 30 14 16 16 14
CL 2 30 16 14 16 14
CL 3 40 18 22 21 19CL 4 50 26 24 22 28
CL 5 50 26 24 27 23
CL 6 60 28 32 31 29
CL 7 60 28 32 28 32
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 20/39
Legend: - SY 2008-2009
- SY 2009-2010
FORMNo. of
Items
No. of
Good Fit
Item/s
No. of
Bad Fit
Item/s
N MEAN STDEV
PROPORTIO
N
CORRECT
Math 1 30 27 28 3 2 190 239 24.08 19.81 4.42 4.81 0.80 0.66
Math 2 40 35 36 5 4 212 292 26.68 21.69 6.62 5.35 0.67 0.54
Math 3 29 23 27 6 2 244 318 21.81 19.43 4.79 4.39 0.75 0.65
Math 4 49 47 49 3 - 216 312 30.25 32.24 7.72 8.49 0.62 0.66
Math 5 50 48 48 2 2 273 287 30.05 23.83 7.80 7.48 0.60 0.48
Math 6 58 59 55 1 3 260 281 31.92 28.59 8.16 9.01 0.55 0.48
Math 7 60 59 59 1 1 251 332 30.68 29.41 9.00 8.10 0.51 0.49
MATH
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 21/39
FORM
SY2008-2009 taken by the
Upper Level (PILOT
TESTING)
SY2009-2010
Math 1 9, 16, 20 16, 25
Math 2 5, 19, 20, 24, 32 23, 26, 31, 32
Math 3 2, 3, 7, 18, 21, 25 18, 25
Math 4 31, 36, 39 NO REVISIONS NEEDED
Math 5 25, 41 25, 41
Math 6 14 8, 14, 50
Math 7 6 50
FORM DIFFICULT to EASY items EASY to DIFFICULT items
Math 1 5 12, 26, 20
Math 2 38, 25, 5 20, 32, 23, 18, 31, 19
Math 3 21, 27, 24, 14 7
Math 4 42, 36, 38 6, 27, 1
Math 5 39, 46, 44 8, 28, 4
Math 6 44, 37, 20, 42, 15, 34, 26, 18, 31, 28 45, 6, 25
Math 7 30, 31, 49 57, 52, 47, 46, 17, 36, 18, 11
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 22/39
Form No. of Items
SY2008-2009 SY2009-2010
No. of Difficult
Items
No. of Easy
Items
No. of Difficult
Items
No. of Easy
Items
Math 1 30 16 14 18 12
Math 2 40 16 24 19 21
Math 3 29 15 14 12 17
Math 4 49 25 24 25 24
Math 5 50 27 23 27 23
Math 6 58 32 26 26 32
Math 7 60 26 34 32 28
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 24/39
FORMSY2008-2009 taken by the
Upper Level (PILOT ESTING)SY2009-2010
Sci 1 10, 27 12
Sci 2 10, 22, 24, 25, 27 NO REVISIONS NEEDED
Sci 3 17 15, 32
Sci 4 30 NO REVISIONS NEEDED
Sci 5 6, 7, 8, 10, 11, 22, 44 10
Sci 6 NO REVISIONS NEEDED NO REVISIONS NEEDED
Sci 7 27 NO REVISIONS NEEDED
FORM DIFFICULT to EASY items EASY to DIFFICULT items
Sci 1 6, 7, 26 4, 1, 10
Sci 2 11, 26, 28 22, 25
Sci 3 24, 27 2, 17
Sci 4 11, 12, 23, 26,45 2, 10, 15, 20, 25, 27, 33, 37, 46
Sci 5 2, 13, 15, 16, 33, 36, 43, 47, 49 8, 11, 22, 24
Sci 6 11, 15, 18, 21, 23, 26, 33, 37, 48, 53, 54 8, 31, 32, 35, 41, 42, 56
Sci 7 31, 32, 36, 39, 51, 52, 54, 58 1, 3, 7, 12, 13, 20, 26, 28, 30, 41
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 25/39
Form No. of
Items
SY2008-2009 SY2009-2010
No. of Difficult
Items
No. of Easy
Items
No. of Difficult
Items
No. of Easy
Items
Sci 1 30 13 17 13 17
Sci 2 30 14 16 13 17
Sci 3 40 22 18 22 18
Sci 4 50 22 28 26 24
Sci 5 50 26 24 21 29
Sci 6 60 31 29 27 33
Sci 7 60 29 31 31 29
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 26/39
Legend: - SY 2008-2009
- SY 2009-2010
FORM
No. of
Items
No. of
Good FitItem/s
No. of
Bad FitItem/s N MEAN STDEV
PROPORTION
CORRECT
Fil 1 30 27 27 3 3 229 239 21.40 16.85 5.22 5.25 0.71 0.56
Fil 2 35 32 34 3 1 174 292 23.92 21.40 6.15 5.75 0.68 0.61
Fil 3 40 37 40 3 - 204 328 24.53 22.42 8.64 7.40 0.61 0.56
Fil 4 50 45 49 5 1 258 318 27.55 26.42 9.03 8.38 0.55 0.53
Fil 5 50 54 47 6 3 266 287 27.81 26.52 8.90 7.95 0.56 0.53
Fil 6 60 55 55 5 5 276 305 39.24 32.42 9.43 10.25 0.65 0.54
Fil 7 60 56 59 4 1 240 319 37.46 31.83 9.58 8.80 0.62 0.53
FILIPINO
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 27/39
FORMSY2008-2009 taken by the
Upper Level (PILOT TESTING)SY2009-2010
Fil 1 4, 9, 26 9, 20, 26
Fil 2 24, 28, 29 8
Fil 3 2, 5, 24, 27 NO REVISIONS NEEDED
Fil 4 15, 19, 21, 39, 43 38
Fil 5 5, 16, 22, 26, 36, 38 5, 25, 26
Fil 6 5, 15, 20, 37, 54 5, 10, 11, 19, 20
Fil 7 4, 30, 39, 48 9
FORM DIFFICULT to EASY items EASY to DIFFICULT items
Fil 1
Fil 2 33 21
Fil 3 4, 9, 14, 29, 38 10, 16, 30
Fil 4 2, 11, 40 16, 35, 49
Fil 5 31, 40, 42 3, 5, 9, 30, 34, 36
Fil 6 1, 31, 40, 44, 49, 56
Fil 7 6, 8, 12, 22, 23, 42, 43, 45, 57 2, 13, 27, 32, 34
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 28/39
Form No. of Items
SY2008-2009 SY2009-2010
No. of Difficult
Items
No. of Easy
Items
No. of Difficult
Items
No. of Easy
Items
Fil 1 30 13 17 14 16
Fil 2 35 17 18 17 18
Fil 3 40 22 18 20 20 Fil 4 50 27 23 27 23
Fil 5 50 22 28 25 25
Fil 6 60 28 32 33 27
Fil 7 60 29 31 25 35
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 29/39
Which of the tests obtained the sameresults when administered to 2 different
groups for«
Item Difficulty?English 2 and Filipino 1 only
Item Fit?
Science 6 only (no bad fit items)
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 30/39
Batch 1 ± IPA Personnel and Coordinators
Batch 2 ± Math and Science Teachers
What did you like in the activity?
- workshop is very useful and
very relevant.
- Everything about the latest
technology that we can use for
Item Analysis
My suggestions to improve the activity are:
- more workshops, it was an interesting
topic but then we need more time so we
can absorb all the important details
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 31/39
Batch 3 ± Social Studies & CL teachers
Batch 4 ± English & Filipino teachers
What i li i the activit
- It is helpful in determining items difficulty and
good fitting items in tests.It will further help usteachers in creating reliable and valid
assessments.
- The topic is interesting because it's ''new''
- New insights.
- Actual/hands on.
- It was just too fast.
What did you dislike in the activity?
My suggestions to improve the activity are:
- Study further its advantage before bringing
this down to us, teachers.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 32/39
Batch 4 ± English & Filipino teachers
What did you like in the activity?
What did you dislike in the activity?
My suggestions to improve the activity are:
- the IRT is ok but would need more time
to familiarize with it.
- The topic is very applicable in test construction and testing.
- it's a different way to analyze items (basically multiple choice) although theconcern about why do it tediously needs to be addressed (aside from the fact
that the teacher gets to use the improvements needed in the test)
- I really like the lecture part. the speaker explained it very well in very
interesting manner. congratulations!
- Longer time for practice/hands on.
- use of varied data so we can practice using the program more.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 33/39
SUMMARY
Best Practices in the conduct of IRT:
1. The school sought the assistance of consultants from
De La Salle University , Manila, Dr. Alicia Bustos-Orosa,
to take the lead in the test development project and Dr.
Carlo Magno, for the statistical treatment and analysis of test data.
2. The Instructional and Performance Assessment Office
(IP A) in collaboration with the Principals and subject
area coordinators, worked closely with the consultants inrelation to planning the in-service programs for the
faculty.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 34/39
3. Two of the IPA personnel were trained to
prepare the test data needed by Dr. Magno, which
was followed by consultative meetings on the
technical analysis of test data.
4. The administration provided the financial
support needed for the conduct of the IRT by
allotting a budget for the purchase of a scanner ,
software for making answer sheets, software for
IRT, training the faculty and hiring additional
personnel to perform the tasks.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 35/39
5. The IPA technical support staff did the IRT for the
achievement tests administered in February and
March 2009 under the supervision of Dr. Magno.
6. The training for the IPA personnel, principals, andsubject area coordinators was held on November 7,
2009. This was followed by training for Science and
Math teachers on December 5, 2009. The CL and
Social Studies teachers had their turn on June 26, 2010.The last group, the English and Filipino teachers, had
their training on July 24, 2010.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 36/39
7. The items in the first end termexams (September 15-17) will be
analyzed using the IRT approach and
this will be done by the teachers withthe assistance of the IPA Office.
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 37/39
1.As a whole, the academic community is receptive
and open to trying out new approaches to improve
their craft. It is therefore easier for DLSZ to
introduce innovations.
2.The IRT results can be further validated by looking
into other factors that have influenced the results.
IMPLICATIONS
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 38/39
3. Since there are still revisions required to improve
the test items, seminar- workshops specific for the
subject areas identified are recommended.
4. To further practice using the IRT, opportunities
should be provided to the teachers, not just
limited to end-term examinations.
IMPLICATIONS
8/9/2019 Diesta Irt Presentation
http://slidepdf.com/reader/full/diesta-irt-presentation 39/39
³Item writing is an art. It requires an uncommoncombination of special abilities. It is mastered only
through extensive and critically supervised practice. It
demands and tends to develop high standards of quality
and a sense of pride in craftsmanship.´
Robert Edel
Thank you. God bless.