MEASURING AND COMPARING HIGHER MEASURING AND COMPARING HIGHER EDUCATION QUALITY EDUCATION QUALITY

BETWEEN COUNTRIES WORLDWIDEBETWEEN COUNTRIES WORLDWIDE

AnatoliAnatoli A. Maslak, A. Maslak, TatiTatijjanaana S. AnisimovaS. AnisimovaSlavianskSlaviansk--onon--KubanKuban State Pedagogical InstituteState Pedagogical Institute

This study was supported by a Fulbright Scholar Grant 68427622, This study was supported by a Fulbright Scholar Grant 68427622, which was awarded to the first author. Special thanks to the Colwhich was awarded to the first author. Special thanks to the College lege of Education, University of Illinoisof Education, University of Illinois--Chicago, for providing the Chicago, for providing the resources necessary for conducting this research.resources necessary for conducting this research.

PURPOSE OF INVESTIGATIONPURPOSE OF INVESTIGATION

TheThe purposepurpose ofof thisthis investigationinvestigation isis totoestablishestablish a a unidimensionalunidimensional intervalinterval scalescale forformeasuringmeasuring eacheach countrycountry onon thethe qualityquality ofofhigherhigher educationeducation, , basedbased onon indicatorsindicators((itemsitems) ) characterizingcharacterizing variousvarious aspectsaspects ofof a a countrycountry’’ss qualityquality. . TheThe datadata fromfrom thesetheseindicatorsindicators areare publiclypublicly availableavailable throughthrough thetheUnitedUnited NationsNations EducationalEducational, , ScientificScientific, , andandCulturalCultural OrganizationOrganization (UNESCO), (UNESCO), forfor allallcountriescountries worldwideworldwide. .

ADVANTAGES OF MEASUREMENTADVANTAGES OF MEASUREMENT

A single A single measuremeasure ofof qualityquality cancan facilitatefacilitate::-- a a countrycountry’’ss educationeducation qualityquality toto bebemonitoredmonitored overover timetime;;-- enableenable a a comparisoncomparison ofof educationeducation qualityqualitybetweenbetween differentdifferent countriescountries;;-- decisiondecision makingmaking relativerelative toto improvingimproving a a countrycountry’’ss qualityquality ofof educationeducation. .

CURRENT STATE OF ESTIMATIONCURRENT STATE OF ESTIMATION

CurrentlyCurrently, a , a countrycountry’’ss qualityquality ofof higherhighereducationeducation isis summarizedsummarized byby simplesimpledescriptivedescriptive statisticsstatistics ofof thesethese manymanyindicatorsindicators.. These indicators capture These indicators capture various aspects of a countryvarious aspects of a country’’s education s education quality,quality, andand therethere seemsseems toto bebe a a needneed totocombinecombine thesethese resultsresults intointo a single a single measuremeasure ofof qualityquality..

METHODMETHODThere is created a There is created a unidimensionalunidimensional measurement scale measurement scale of higher education quality with the of higher education quality with the RaschRasch partial partial credit model. The fitness of the indicators to that credit model. The fitness of the indicators to that RaschRasch model is evaluated in order to determine model is evaluated in order to determine whether each indicators contribute to a whether each indicators contribute to a unidimensionalunidimensional scale. Provided that all items do fit, scale. Provided that all items do fit, then it is possible to represent each country by a then it is possible to represent each country by a single measure of higher education quality. Given the single measure of higher education quality. Given the constructed constructed unidimensionalunidimensional scale, the different scale, the different countries are then compared on education quality.countries are then compared on education quality.

INDICATORS OF HIGHER EDUCATION QUALITYINDICATORS OF HIGHER EDUCATION QUALITY

For the purposes of evaluating the quality of For the purposes of evaluating the quality of higher education in each country, there were higher education in each country, there were used thirteen indicators (items) based on used thirteen indicators (items) based on information from UNESCOinformation from UNESCO’’s databases s databases pertaining to year 1994 (pertaining to year 1994 (www.unesco.orgwww.unesco.org). ).

These indicators, labeled These indicators, labeled XX11, , XX22,, XX33,,……,,XX1313, , are each described as follows, along with their are each described as follows, along with their response scale.response scale.

ХХ11: : CountryCountry’’s level of economic s level of economic development in 1994development in 1994

Response scaleResponse scale: : 0 = a least developed country; 0 = a least developed country; 1 = a less developed country; 1 = a less developed country; 2 = a developed country; 2 = a developed country; 3 = the most developed country.3 = the most developed country.

ХХ22: Amount of manpower resources : Amount of manpower resources that were engaged in higher that were engaged in higher educationeducation

Response scaleResponse scale: : [([(NNStudentsStudents + + NNFacultyFaculty ++ NNStaffStaff)/ Total Country )/ Total Country

Population] * 10000.Population] * 10000.(where say, (where say, NNstudentsstudents refer to the number of refer to the number of students in a country in 1994)students in a country in 1994)

ХХ33: Country: Country’’s financing of education s financing of education

Response scaleResponse scale: : Percent of Gross Domestic Product devoted to Percent of Gross Domestic Product devoted to education as a whole in 1994 (not just higher education as a whole in 1994 (not just higher education).education).

ХХ44: Country: Country’’s financing of higher s financing of higher educationeducation

Response scaleResponse scale: : Percent of Gross Domestic Product devoted to Percent of Gross Domestic Product devoted to higher education in 1994.higher education in 1994.

ХХ55: Country: Country’’s Student s Student –– teacher teacher ratio ratio

Response scaleResponse scale: : NNStudentsStudents / / NNFacultyFaculty in 1994.in 1994.

ХХ66:: CountryCountry’’s s unweightedunweighted number of number of graduates relative to manpower graduates relative to manpower resourcesresources

Response scaleResponse scale: : ((ww11))NNBachelorsBachelors + (+ (ww22))NNMastersMasters + (+ (ww33))NNDoctorateDoctorate) / ) / XX22

(where for example, (where for example, NNBachelorsBachelors refers to the refers to the number of students in a country who obtained number of students in a country who obtained a Bachelora Bachelor’’s degree in 1994, and weights s degree in 1994, and weights ww11 = = ww22 = = ww33 = 1).= 1).

ХХ77: : CountryCountry’’s weighted number of s weighted number of graduates relative to manpower graduates relative to manpower resourcesresources

Response scaleResponse scale: : ((ww11))NNBachelorsBachelors + (+ (ww22))NNMastersMasters + (+ (ww33))NNDoctorateDoctorate) / ) / XX22

(Same as (Same as XX66, setting , setting ww11 = 1, = 1, ww22 = 2, = 2, ww44 = 4.= 4.Here the weights place more importance to Here the weights place more importance to higher degrees).higher degrees).

ХХ88:: Proportion of students who Proportion of students who obtained any kind of higherobtained any kind of higher--education degree in the humanities, education degree in the humanities, relative to the total number of relative to the total number of graduatesgraduatesResponse scaleResponse scale: : NNHumanitiesHumanities / (/ (NNBachelorBachelor + + NNMastersMasters + + NNDoctorateDoctorate).).

ХХ99:: Proportion of students who Proportion of students who obtained any kind of higherobtained any kind of higher--education education degree in the natural sciences, degree in the natural sciences, relative to the total number of relative to the total number of graduatesgraduates

Response scaleResponse scale: : NNNaturalSciencesNaturalSciences / (/ (NNBachelorsBachelors + + NNMastersMasters + + NNDoctorateDoctorate).).

ХХ1010:: Proportion of students who Proportion of students who obtained any kind of higherobtained any kind of higher--education education degree either in mathematics or degree either in mathematics or computer science, relative to the total computer science, relative to the total number of graduatesnumber of graduatesResponse scaleResponse scale: :

NNMathCompMathComp / (/ (NNBachelorsBachelors + + NNMastersMasters + + NNDoctorateDoctorate).).

XX1111: In the country, proportion of : In the country, proportion of students in 1994 who obtained any students in 1994 who obtained any kind of higherkind of higher--education education degree in degree in medicine, relative to the total number medicine, relative to the total number of graduatesof graduatesResponse scaleResponse scale: : NNMedicineMedicine / (/ (NNBachelorsBachelors + + NNMastersMasters + + NNDoctorateDoctorate).).

XX1212: Proportion of students who : Proportion of students who obtained any kind of obtained any kind of higherhigher--educationeducationdegreedegree inin engineeringengineering, , relativerelative toto thethetotaltotal numbernumber ofof graduatesgraduates

Response scaleResponse scale: : NNEngineeringEngineering / (/ (NNBachelorBachelor + + NNMastersMasters + + NNDoctorateDoctorate))

ХХ1313: Proportion of students who : Proportion of students who obtained any kind of higherobtained any kind of higher--education education degree in either business or degree in either business or commerce, relative to the total commerce, relative to the total number of graduatesnumber of graduatesResponse scaleResponse scale: : NNBusinessCommerceBusinessCommerce / (/ (NNBachelorBachelor + + NNMastersMasters + + NNDoctorateDoctorate).).

RAW DATA TRANSFORMINGRAW DATA TRANSFORMING

The The RaschRasch model assumes item responses to be model assumes item responses to be ordinal and discrete. So for each of the indicators ordinal and discrete. So for each of the indicators XX22,,……,,XX1313, the continuous response scale was initially , the continuous response scale was initially discretizeddiscretized into a rating scale: 0 = Lowest, 1 = Low, 2 into a rating scale: 0 = Lowest, 1 = Low, 2 = Medium, 3 = High, 4 = Highest. In particular, for = Medium, 3 = High, 4 = Highest. In particular, for each of these indicators, we constructed this rating each of these indicators, we constructed this rating scale by taking the range of the responses observed scale by taking the range of the responses observed over all countries in the data, and dividing that range over all countries in the data, and dividing that range into five equal parts. into five equal parts.

COUNTRIES CATEGORIZATIONCOUNTRIES CATEGORIZATION

In total, thirtyIn total, thirty--nine countries were analyzed in this study. With nine countries were analyzed in this study. With respect to indicator respect to indicator XX11, Great Britain, Germany, Italy, Canada, , Great Britain, Germany, Italy, Canada, U.S.A., France, Japan, and Russia were each categorized as a U.S.A., France, Japan, and Russia were each categorized as a ““3 = most developed country,3 = most developed country,”” Australia, Belgium, Bulgaria, Australia, Belgium, Bulgaria, Hungary, Spain, Republic Korea, New Zealand, Netherlands, Hungary, Spain, Republic Korea, New Zealand, Netherlands, Norway, Portugal, Ireland, Sweden, Poland, and Austria were Norway, Portugal, Ireland, Sweden, Poland, and Austria were each rated as a each rated as a ““2 = developing country,2 = developing country,”” each of Algeria, each of Algeria, Brazil, Honduras, China, Congo, Mexico, Tunis, and Ethiopia Brazil, Honduras, China, Congo, Mexico, Tunis, and Ethiopia was a was a ““1 = less developed country,1 = less developed country,”” and finally, Benin, and finally, Benin, Burundi, Lesotho, Malaysia, Nigeria, Burundi, Lesotho, Malaysia, Nigeria, RuandaRuanda, Sudan, Uganda, , Sudan, Uganda, and Namibia were each categorized as a and Namibia were each categorized as a ““00 = a least developed = a least developed country.country.””

MEASUREMENT MODELMEASUREMENT MODEL

{ }[ ]( )

( )∑ ∑

∑

= =

=

−

−=∈==

jm

t

t

kjkn

x

kjkn

jknjnjnjk mkxp

0 0

0

exp

exp,,...,1Pr

δθ

δθδθ

where θn to represent the higher-education quality of country n,

parameter δjk - the difficulty of k category of indicator j,

xnj – value of indicator j for country n,

pnjk – probability of reaching by country n the category k from category k−1 of indicator j.

SOFTWARESOFTWARE

RUMM 2010. RUMM 2010. RaschRaschUnidimensionalUnidimensional Measurement Measurement Models Models -- Pert: RUMM Laboratory Pert: RUMM Laboratory Ltd, 2001. Ltd, 2001. -- 87p. 87p. (www.(www.rummlab.comrummlab.com))

Item characteristic curve the non fitting indicator Item characteristic curve the non fitting indicator XX55”” CountryCountry’’s Student s Student –– teacher ratioteacher ratio””

““HARD” INDICATORSHARD” INDICATORS

Indicators EstimatedDifficulty

Chi Square p-value

Correlation with Country Measures

X9: proportion of graduates in natural sciences

.965 .121 .10

X11: proportion of graduates in medicine

.720 .392 .17

X2: manpower resources engaged in higher education

.570 .020 .69

X8: proportion of graduates in humanities

.551 .144 .22

X13: proportion of graduates in business

.130 .088 .02

X12: proportion of graduates in engineering

-.026 .577 .38.246

.263

.277

.280

.277

Standard Error.295

““EASY” INDICATORSEASY” INDICATORS

Indicators EstimatedDifficulty

Chi Square p-value

Correlation with Country Measures

X6: unweighted number of graduates relative to manpower resources

-.279 .732 .18

X4: financing of higher education -.289 .873 .08

X10: proportion of graduates in mathematics or computer science

-.309 .362 .35

X7: weighted number of graduates relative to manpower resources

-.351 .655 .39

X1: level of economic development -.573 .147 .81

X3: financing of education -1.110 .775 .31

.270

.257

Standard Error

.282

.195

.254

.245

Item characteristic curve for the best fitting indicator Item characteristic curve for the best fitting indicator XX44““CountryCountry’’s financing of higher educations financing of higher education““

Item characteristic curve for the least fitting indicator Item characteristic curve for the least fitting indicator XX22““manpower resources engaged in higher educationmanpower resources engaged in higher education””

Item characteristic curve for the easiest indicator XItem characteristic curve for the easiest indicator X33““CountryCountry’’s financing of educations financing of education””

Item characteristic curve for the hardest indicator XItem characteristic curve for the hardest indicator X99““Proportion of students in the natural sciencesProportion of students in the natural sciences””

CountryCountry--Item Location DistributionItem Location Distribution

COUNTRIES WITH HIGHEST MEASURESCOUNTRIES WITH HIGHEST MEASURES

Rank of country Country

Level of economic

developmentEducation

quality (logits)Standard error

(logits)1 The USA 3 1.039 .471 France 3 1.039 .472 Great Britain 3 .829 .452 Australia 2 .829 .452 Spain 2 .829 .453 Italy 3 .631 .444 Germany 3 .442 .435 Belgium 2 .257 .435 Republic Korea 2 .257 .435 Ireland 2 .257 .436 Japan 3 .075 .427 New Zealand 2 -.105 .437 Russia 3 -.105 .43

COUNTRIES WITH MEDIUM MEASURESCOUNTRIES WITH MEDIUM MEASURES

Rank of country Country

Level of economic

developmentEducation

quality (logits)Standard error

(logits)8 Bulgaria 2 -.287 .438 Norway 2 -.287 .438 Sweden 2 -.287 .439 Hungary 2 -.472 .439 Netherlands 2 -.472 .4310 Canada 3 -.663 .4410 Algeria 1 -.663 .4410 Burundi 0 -.663 .4411 Brazil 1 -.861 .4511 China 1 -.861 .4511 Tunis 1 -.861 .4511 Poland 2 -.861 .4511 Austria 2 -.861 .45

COUNTRIES WITH LOWEST MEASURESCOUNTRIES WITH LOWEST MEASURES

Rank of country Country

Level of economic

developmentEducation

quality (logits)Standard error

(logits)12 Honduras 1 -1.071 .4712 Mexico 1 -1.071 .4713 Portugal 2 -1.299 .4913 Ethiopia 1 -1.299 .4913 Ruanda 0 -1.299 .4913 Uganda 0 -1.299 .4914 Congo 1 -1.553 .5214 Malaysia 0 -1.553 .5214 Sudan 0 -1.553 .5214 Namibia 0 -1.553 .5215 Benin 0 -2.206 .6416 Lesotho 0 -2.688 .7616 Nigeria 0 -2.688 .76

HIGHER EDUCATION QUALITY OF THE COUNTRIES, HIGHER EDUCATION QUALITY OF THE COUNTRIES,

DEPENDING ON THE LEVEL OF ECONOMIC DEVELOPMENTDEPENDING ON THE LEVEL OF ECONOMIC DEVELOPMENT

Level of economic development

Number of countries

Median value of Education Quality

Average value of education quality

0 9 -1,39 -1.52 1 8 -.80 -.95 2 14 -.30 -.21 3 7 .47 .38

Education quality in Russia equals –0,105

Thank YouThank You!!

anatoliy_maslak@mail.ruanatoliy_maslak@mail.ru8(86146)430438(86146)43043

ViceVice--Chancellor for Research,Chancellor for Research,SlavianskSlaviansk--onon--Kuban State Pedagogical InstituteState Pedagogical Institute

ДополнениеДополнение к Рекомендациямк Рекомендациям

13. Разработать наборы 13. Разработать наборы индикаторов качества высшего индикаторов качества высшего (инженерного) образования и провести (инженерного) образования и провести их апробацию.их апробацию.

1414. Измерить качество высшего . Измерить качество высшего образования в технических вузах на образования в технических вузах на основе аттестационных показателей основе аттестационных показателей МинобрнаукиМинобрнауки..

УважаемыеУважаемые коллегиколлеги!!ПриглашаемПриглашаем ВасВас принятьпринять участиеучастие вв работеработе 88--ойой всероссийскойвсероссийской

((сс международныммеждународным участиемучастием)) научнонаучно -- практическойпрактической конференцииконференции««ТеорияТеория ии практикапрактика измеренияизмерения латентныхлатентных переменныхпеременных»»,,

ии 22--ойой летнейлетней школышколы««ТеорияТеория ии практикапрактика измеренияизмерения латентныхлатентных переменныхпеременных»»

КонференцияКонференция состоитсясостоится 23 23 июняиюня 2006 2006 годагода вв СлавянскомСлавянском--нана--КубаниКубани государственномгосударственном педагогическомпедагогическом институтеинституте попо аадресдресуу: : 353560 353560 гг. . СлавянскСлавянск--нана--КубаниКубани КраснодарскогоКраснодарского краякрая, , улул. . КубанскаяКубанская, , 200.200.

ЛетняяЛетняя школашкола проводитсяпроводится cc 24 24 июняиюня попо 2 2 июляиюля 20062006гг. . ПодробнаяПодробная информацияинформация нана сайтесайте wwwwww..sgpisgpi..ruru