INEE Curso UIMP 2016 - Evaluación educativa: Luka Boeskens

Perspectivas Actuales Nacionales e Internacionales en Evaluación EducativaUniversidad Internacional Menéndez Pelayo, Santander, 7 July 2016Presentation byLuka BoeskensDirectorate for Education and Skills

Policies to Improve the Effectiveness of Resource Use in Schools[Insights from the OECD School Resources Review]

Purpose and scope• Motivation• Main issues for analysis• Conceptual framework

Methodology• Country participation• Analytical phase• Review phase• Synthesis phase (outlook)

Policy challenges• Sources of inefficiency• Policy approaches

OECD School Resources Review

0 20 000 40 000 60 000 80 000 100 000 120 000 140 000 160 000 180 000 200 000300

350

400

450

500

550

600

650

511.338208

385.595556

368.102547

426.737491420.512968

409.291568

447.984415

376.488601

387.824630

413.281467409.626613

391.459889

438.738260

422.632355

471.131461478.823277

490.571021

477.044455

612.675536

481.644744

498.957882520.545522

466.481430

517.501097

553.766659

487.063181

499.749903

518.070400513.525056

484.319298

494.984674

485.321181

573.468314

518.750335

536.406918

501.127422501.497460492.795697

522.971758

478.260636

514.745239

UK

504.150766500.026757

481.366786

505.540743

489.373070

530.931004

489.845098R² = 0.369063315519053R² = 0.00587924272458274

Average spending per student from the age of 6 to 15 (USD, PPPs)

Mat

hem

atic

s pe

rform

ance

(sco

re p

oint

s)

Cumulative expenditure per student less than USD 50 000

Cumulative expenditure per student USD 50 000 or more

PISA 2012,Fig IV.1.8

Purpose: Provide analysis and policy advice on how to govern, distribute, utilise and manage resources so that they contribute to achieving countries’ educational objectives to the fullest.Levels of education system covered

o Pre-primary (ECEC), primary and secondary education levels, including vocational and pre-vocational education (VET) at secondary level

Administrational levels of analysiso System level o Sub-system level (regional and local)o School level

Purpose and Scope of the Review

“What policies best ensure that school resources are effectively used to improve student outcomes?”

Types of resource inputso Financial resources (e.g. public funding of individual schools)o Human resources (e.g. teachers, administrators and their use of time)o Physical/material resources (e.g. school network, buildings, equipment)o Learning time

Types of outputso Access, participation and completiono Learning outcomes (e.g. cognitive, non-cognitive, key competencies)o Labour market and social outcomes (e.g. social mobility, work

productivity)

Purpose and Scope of the Review

Looking at both equity and efficiency of resource use…

Effectiveness o Educational effectiveness refers to the ability of a school or school system

to adequately accomplish stated education objectives.o Studies of educational effectiveness analyse whether specific resource

inputs have positive effects on outputs, and if so, how large these effects are.

Efficiencyo Educational efficiency, on the other hand, refers to the achievement of

stated education objectives at the lowest possible cost.o That is, efficiency is effectiveness plus the additional requirement that

this is achieved in the least expensive manner

Purpose and Scope: Key concepts

-0.500.511.5300

350

400

450

500

550

600

650

700R² = 0

Equity in resource allocation (index points)

Mat

hem

atic

s pe

rform

ance

(sco

re p

oint

s)Countries with better performance in mathematics tend to allocate educational resources more equitably

Greater equity

Less equity

Adjusted by per capita GDP

Fig IV.1.11

Resource Governance Resource Distribution

Resource Utilisation Resource Management

How to distribute resources acrosslevels, sectors and student groups?

How to utilise resources for different programmes and priorities at the point

of use?

How to manage, monitor andfollow up on resource use?

How to govern, plan andimplement resource use?

Main issues for analysis

1 Governance

- For education

- For schooling

Resources available

Planning resource

useSources of

revenue

Implementingpolicies to improve

effectiveness

Linking to school system

governance

- Defining prioritiesand targets- Distribution of responsibilities- Forecasting and Simulation- Instruments andMethods

- Clarity of objectives- Consensus building- Communication & consultation- Policy experi-ments& pilots

2 Resource Distribution

Between - Admin levels- Education levels / sectors

Financial transfers

Targeted funding for

specific groups

School funding Funding

formulae

Human resources

Physical resources

- Targeted programmes - Funding for SEN- Financial aid- Free school meals- After-school support - Ensuring adequate

supply- Attracting &retaining staff- Recruitment & career systems

3 Resource Utilisation

- T&L environments

- Using facilities & materials

Organising schools

School staff

- Organising school

leadership - Allocating

teacher resources to

students

Organising student learning

time

Matching resources to

students’needs

- Use of teachers’ time- Tasks beyond teaching- Support staff- Expert teachers- Class size

- Differentiated/ personalised instruction- Mentoring- Out-of-school support- Grouping of students

4 Resource Management

Competency frameworks

Building capacity to

manage resources

Monitoring resource

useTransparency

IncentivesReporting

- Evaluatingeffectiveness of resource use- School accountability for resource use- Feeding evaluation results into nextbudgetary cycle - Outcome-based

planning- Intrinsic motivation- Rewards and sanctions

ContextEconomic

Demographic

Political

Cultural

Governance

Optimal outputsAccess, participation,

completionLearning outcomes

Labour market & social outcomes

Resource Distribution

Resource Utilisation

Resource Managemen

t

Education System Goals

Conceptual framework

Conceptual framework

2 METHODOLOGY

The Review is organised in three phases:o Analytical Phase: bring together evidence-based policy lessons from

existing data and research: literature reviews and country background reports (CBRs).

o Review Phase: provide tailored policy advice to individual countries based on in-depth country review visits.

o Synthesis phase: generate overall policy conclusions based on evidence from the analytic and review phases.

Methodology: Three phases

Work is guided by Group of National Experts on School Resources (GNE) and supported by collaborations:o Within the OECD: INES, PISA, TALIS, CERI work (e.g. Governing

Complex Education Systems – participation in Review of Uruguay), ECEC policy work, work by the Public Governance Directorate.

o With social partners: BIAC, TUAC (Permanent Observers to GNE on School Resources)

o With other international organisations: e.g. Eurydice, UNESCO, Organising Bureau of European School Student Unions (OBESSU), IADB, European Agency for Special Needs and Inclusive Education

o Formal partnership: European Commission: Partially covering participation costs of countries which are part of the EU’s Erasmus+ Programme and providing some resources for background work.

o Formal collaboration with the World Bank for the Review of Kazakhstan, which extends to participation in GNE.

Methodology: Collaboration

Methodology: Country participation

The following levels of participation are possible for each country:o GNE meetings: Participation in meetings of the Group of National

Experts (all OECD members countries are part of the GNE)

o Country Background Report: Preparation of a Country Background Report, whose information will be reflected in final comparative analysis

o Country Review: Preparation by the OECD Secretariat of a Country Review report providing in-depth analysis and policy recommendations

• Austria • Belgium (Flemish Community)• Chile• Colombia• Czech Republic• Denmark• Estonia• Kazakhstan• Lithuania• Slovak Republic• Uruguay

Country Reviews (11)

• Belgium (French Community)• Iceland• Luxembourg• Spain• Slovenia• Sweden

Country Background Reports (6)

Methodology: Country participation

o 2-3 additional countries expected to join soono Countries can still join to create Country Background Report or to opt for a

full Country Review

The Secretariat undertakes background work through the preparation of literature reviews:

Published:– School Size Policies (Macarena Ares)– Student Learning Time (Anna Gromada and Claire Shewbridge)– Learning Support Staff (Francesc Masdeu)– Budgeting and Accounting in Education Systems (Tala Fakharzadeh)

To be published:– Targeted School Funding– Funding of Vocational Education and Training– Funding of Special Needs Education – Public Funding of Private Schools

Methodology: Analytical background work

The Secretariat gathers data on countries’ policies and practices through Country Background Reports (CBRs):CBRs are prepared by countries, based on the Review’s common framework.

9 CBRs published:– Iceland, Kazakhstan, Slovak Republic, Austria, Belgium (Flemish

Community), Denmark, Uruguay, Lithuania and Estonia

8 CBRs in preparation: – Czech Republic, Sweden, Belgium (French Community), Chile,

Luxembourg, Slovenia, Colombia and Spain

Methodology: Analytical background work

In-depth country reviews

Drawing on the CBRs, OECD-led Review Teams conduct an in-depth review and prepare Country Reviews with specific advice to individual countries.

6 Country Reviews have been published: Kazakhstan, Belgium (Flemish Community), Slovak Republic, Lithuania, Estonia and Austria.4 more Country Reviews to be published this year: • Czech Republic (July)• Denmark (September)• Uruguay (September/October)• Chile (October/November)

Methodology: Review phase (1)

Review visits:• Preliminary 2-day visit to plan the country review visits• Involving diverse set of external experts

Focus areas for country reviews:Countries select from among the thematic modules proposed by the Review.The most prevalent themes addressed in country reviews thus far include:• Review of School Funding

– Funding formulas – Funding for special needs education– Funding for VET

• Review of the School Network organisation, including– Addressing small schools / school consolidation– Provision of school places

• Review of the Human Resources management, particularly of the teaching workforce• Review of Equity of Resource Use

Methodology: Review phase (2)

3 Thematic Reports:Next phase: Bring together the evidence and materials produced through the

OECD Country Reviews and present the key findings and policy messages. Proposed foci:

• Funding of School Education (early 2017)

• Organisation of the School Network• Management of Human Resources

Reports will provide:

• Overview of trends, practices and policy developments

• Reviews of the literature

• Examples of successful policy initiatives

• Policy options grounded in analysis

Methodology: Synthesis phase (1)

Principal sources of evidence for thematic report:

Country BackgroundReports

Thematic Report on the Fundingof School Education

Analysis in CountryReview reports

Research-BasedEvidence

(e.g. Literature Reviews)

Analysis of international data (e.g. Education at a Glance, PISA, TALIS,

Eurydice, World Bank)

Collection of qualitative data on country approaches to school funding

(via questionnaires)

Methodology: Synthesis phase (2)

Thematic Report on Funding aims to…

• …provide an international comparative analysis of funding policies in school education;

• …provide a stock-take of current school funding policies and practices in countries;

• …draw attention to effective school funding policy initiatives;

• …develop a comprehensive framework for the development of school funding policies;

• …propose evidence-based policy options for the development of school funding policies; and

• …identify priorities for follow-up work at national, regional and international levels.

Thematic Report: Purpose

3 POLICY CHALLENGES

27

School Resource Use: Major objectives and concerns

Efficient use of resources:• Matching supply of and demand for education services

- Potential concern: Dropouts; inadequate transitions; poor labour market outcomes

• Balancing quality education with cost of provision- Potential concern: Small schools and small classes

• Making an effective use of human resources- Potential concern: Poor use of teacher resources

• Making an effective use of school infrastructure

Equitable use of resources:• Distributing resources equally (same resources for similar type of

provision -- horizontal equity)- Potential concern: Level of resources differ across similar students

• Distributing resources to provide same educational opportunity (resources allocated according to needs -- vertical equity)

- Potential concern: Poor outcomes of disadvantaged student groups

Sources of inefficiency (1): Dropouts, inadequate education transitions, poor labour market outcomes

Causes:• Supply of education services does not meet needs of all students

- Little diversity of offers- VET offers not sufficiently connected to the labour market (e.g. not

including workplace-based component)

• Early student tracking – students moving downstream to less demanding programmes

• Little individual / differentiated support – for an increasingly heterogeneous student population

• Grade repetition (see next slide)

Japa

n

Nor

way

Est

onia

Icel

and

Isra

el

Uni

ted

Kin

gdom

Slo

veni

a

Pol

and

Cze

ch R

epub

lic

Kor

ea

Sw

eden

Finl

and

Den

mar

k

New

Zea

land

Slo

vak

Rep

ublic

Aus

tralia

Can

ada

Irela

nd

Uni

ted

Sta

tes

Aus

tria

Italy

Por

tuga

l

Ger

man

y

Spa

in

Fran

ce

Net

herla

nds

Bel

gium

0

10000

20000

30000

40000

50000

60000

0

2

4

6

8

10

12

14

Total cost per repeater (one grade year)Total annual cost, relative to total expenditure on primary and secondary education (%)

USD

, PPP

s

%

Grade repetition is an expensive policy

Sources of inefficiency (1): Dropouts, inadequate education transitions, poor labour market outcomes

Policy approaches:• Supply of education services to align with student needs

- Greater diversity of school offerings- VET offerings more relevant to the labour-market (worked-based component) –

links to employers- Greater partnership between general and vocational schools- A good degree of school choice – to stimulate a more diverse and innovative

school offer

• Informing and facilitating student choice- Strengthening career guidance- Pathways across educational tracks/options

• Supporting students with learning difficulties- Greater individualised support- Avoid early tracking- Targeted funding for students at risk (e.g. scholarships)

Sources of inefficiency (2): Small schools and small classes

Causes• Demographic changes

- Fall in birth rates- Population emigration versus immigration- Greater demand in urban areas and declining population in rural, more

isolated areas

• Fragmentation of supply across small providers- E.g. small municipalities

• Rigid boundaries across education sub-systems and levels of administration – lack of co-ordination of provision

• Fragmentation of supply in secondary education – specialised curriculum and teachers

• Inadequate regulations on opening and closing schools- Private providers with smaller classes


1990 1995 2000 2005 2010 2015 202050

60

70

80

90

100

110

120

0 to 4 years 5 to 9 years10 to 14 years 15 to 19 years

Belgium

1990 1995 2000 2005 2010 2015 202050

60

70

80

90

100

110

120

0 to 4 years 5 to 9 years10 to 14 years 15 to 19 years

Lithuania


Causes• Demographic changes

- Fall in birth rates- Population emigration versus immigration- Greater demand in urban areas and declining population in rural, more

isolated areas

• Fragmentation of supply across small providers- E.g. small municipalities

• Rigid boundaries across education sub-systems and levels of administration – lack of co-ordination of provision

• Fragmentation of supply in secondary education – specialised curriculum and teachers

• Inadequate regulations on opening and closing schools- Private providers with smaller classes


Policy options• School consolidation

- Closing / downsizing services of individual schools- Sharing of resources among schools- Clustering of schools

• Provide local services to younger children (in a given distance)• Co-ordination and planning

- Co-ordination across levels of school administration- Regional planning of education services- Co-operation between local providers (e.g. municipalities)- Modular approach to provision

• Rationalise education offerings in secondary education- Reducing diversity in cognitive outcomes in early years instead

• Regulated entry into the school network- Assessment of need and minimum school and class size

School Resource Use: School consolidation

Approaches to consolidation• Establishing criteria for consolidation

- Cost of provision- Distance to alternative school options for children- School’s role in the local community- Quality of the education provided by the school

• Establishing financial incentives for consolidation- Minimum school size and class size to obtain public funding- Little funding compensation for small size

• Providing central financial support for the transition period- Transfer of students to new school- Defining new use for old school building- Providing financial support for redundant teachers


Policy options• School consolidation

- Closing / downsizing services of individual schools- Sharing of resources among schools- Clustering of schools

• Provide local services to younger children (within a given distance)• Co-ordination and planning

- Co-ordination across levels of school administration- Regional planning of education services- Co-operation between local providers (e.g. municipalities)

• Rationalise education offerings in secondary education- Reducing diversity in cognitive outcomes in early years instead

• Regulated entry into the school network- Assessment of need and minimum school and class size

Mathematical formula with monetary values attached to variablesPotential to reach efficiency and equity objectives but these are by no means guaranteed – but clearly improves transparency of funding distribution.

Characteristics:(i) Level of formula: national formula for individual schools / or to

distribute funding to local authorities; formulas developed by local authorities.

(ii) Costs covered by formula: typically a formula for personnel costs and a formula for operating costs (e.g. heating, electricity, maintenance).

(iii) Typical groups of variables: student numbers and grade level; student characteristics; school characteristics; curriculum and educational programmes.

Funding formulas: Definition and characteristics

Choices need to be made – there is no single best practice:• Comprehensiveness of the formula: proportion of total funding distributed

through formula.

• Ear-marked or not: possible areas for ear-marked money.

• Adjustment for small schools: e.g. compensation coefficient for certain student brackets.

• Variables to be used: mostly inputs (outputs such as degree completions might have undesired effects); student groups with different weights.

• Per-student or per-class formula: funding on the number of students or the number of classes

Funding formulas: Implementation challenges

A number of challenges exist in implementing formulas:• Estimating cost coefficients is difficult: cannot estimate cost of providing

education of a given quality.

• Actual costs might differ across localities and schools: price of certain inputs might be different; how to account for differences in teacher composition in terms of experience and qualifications.

• “Discontinuities” introduced by school size and class size categories: may reduce incentives for competition (effect of “marginal” student).

• Simplicity vs sensitivity to local conditions: simple formula vs need to capture differences across local conditions.

• Need for local discretion vs inefficiencies: local adjustment of formula amounts desirable but could reallocate resources to more inefficient schools.

• Requires good quality data: e.g. student and teacher-level data.

Funding Formulas: Implementation challenges

42

Thank you for your attention

All publications of the School Resources Review can be found at: www.oecd.org/edu/school/schoolresourcesreview.htm

For further information:[email protected]

INEE Curso UIMP 2016 - Evaluación educativa: Luka Boeskens

Education

Transcript of INEE Curso UIMP 2016 - Evaluación educativa: Luka Boeskens