Session 5A: Field work and quality assurance

Beatriz Godoy, Mario Navarrete

Sistemas Integrales Delhi, march 20th 2013

What are the most frequent problems you have encountered in fieldwork?

A. Inadequate selection of staff

B. Poor training C. Weak supervision D. Bad or inconsistent data E. High non response rate F. Fieldwork takes longer

than expected G. Delayed payments H. Excessive paperwork

2 A. B. C. D. E. F.

21%23%

5%

13%

26%

13%

A horror story Survey Month

Households reporting illnesses, accidents (Q407)

Households reporting chronic

deseases (Q401)

Households reporting

agricultural activities (Q901)

Number of crops

reported (Q911)

Households reporting livestock

activities (Q918)

Households reporting fishing activities (Q924)

Total Nb of durables

(Section 12)

Nb of HH having credit (Section 13)

HH size mean

Number of lines with food consumption

1 1,645 705 572 1,352 563 17 9,538 1,017 7.8 95,687

2 1,352 624 503 1,299 530 13 8,853 937 7.5 99,491

3 996 577 507 1,421 502 12 9,818 910 7.6 96,028

4 898 642 486 1,469 504 25 9,301 880 7.9 96,139

5 816 545 436 1,243 464 3 9,180 811 7.1 97,094

6 691 513 477 1,442 465 23 9,667 841 7.5 97,315

7 625 529 465 1,338 494 26 9,621 738 7.5 108,432

8 658 498 439 1,264 433 17 9,437 707 7.3 100,888

9 769 552 433 1,244 428 11 9,584 740 7.2 98,893

10 858 534 468 1,227 436 25 9,224 719 7.4 98,415

11 743 517 399 1,165 411 10 9,294 737 7.2 97,138

12 693 464 356 1,133 440 25 9,722 705 7.1 96,052

Survey month

No. of households reporting illnesses

1 1,645

2 1,352

3 996

4 898

5 816

6 691

7 625

8 658

9 769

10 868

11 743

12 693

The factors of fieldwork quality

Field staff selection and

training

The fieldwork organization

Data management

Supervision and control

4

Field staff selection and training

5

Field staff selection and

training

The fieldwork organization

Data management

Supervision and control

Call for applicants

Pre-selection of candidates

based on profiles

Training

Staff selection based on

performance and integrity

Selection approval by hiring entity

Final formalization of hiring with the selected

personnel

6

Field staff selection and training

The fieldwork organization

7

Field staff selection and

training

The fieldwork organization

Data management

Supervision and control

The fieldwork organization

Fieldwork Organization

Organization of field workers

The fieldwork

plan

Survey Plan

8

Organization of field workers

• Based on the notion of independent teams

• Specialized personnel within each team, or even specialized teams

• Computer-based quality controls integrated into the field work

9

Côte d’Ivoire 1984

Peru 1985

Ghana 1987

Jamaica 1988

Bolivia 1989

Paraguay 1997

Argentina 1997

Bangladesh 1995, 2001

Timor-Leste 2001, 2007

Niger 2008

Namibia 2006

Mongolia 2008

Iraq 2006-2012

India (UP) 1998-2000

Pakistan 1991 - 2002

Vietnam 1996

Honduras 1998

Papua New Guinea 2008

Polynesia 2000

Guatemala 2000

Nicaragua 1989, 2000, 2005

Panama 1998, 2003

Nepal 1994 – 2002 - 2010

Ethiopia 2011

Uganda 2010

Integrating computer quality controls to fieldwork in LSMS Survey’s

(around the world in three decades)

Data entry in fixed locations Data entry in the field CAPI

Organization of field workers

• An interview team 1 team supervisor 2-4 interviewers 1-2 specialized people in specific measurements

(anthropometry, cognitive development testing, hemoglobin, etc.)

1 data entry operator 1 driver

11

Team supervisor Interviewers Specialist Data entry operator

The fieldwork plan at a sampling point

Things to consider: – Number of interviews per sampling point – Geographical scattering of households within each sampling point – Conformation of teams – Household listing operation, if applicable – Quality controls – Biometric and cognitive test, measurements, etc., if applicable – Interview duration, number of visits and correction visits – Other questionnaires, for example, at the community level – Rest time – Approximate travel time to the next sampling point

12

Optimistic scenario Pessimistic scenario $

$

What is the team supposed to do in each sampling point, and how long does it take

Example of a work plan at a sampling point

Día Operador de datos

H1 H2 H3 H4 H5 H6 H7 H8 H9 H1 H2 H3 H4 H5 H6 H7 H8 H92 V1 V1 V1 V1 V1 V1 V1 V1 V1 Ingreso Q. H1 a H93 VC1 VC1 VC1 V2 V2 V2 V2 V2 V2 V2 V2 V2 Ingreso Q. H1 a H94 VC1 VC1 VC1 V3 V3 V3 V3 V3 V3 V3 V3 V3 Ingreso Q. H1 a H95 VC1 VC1 VC1 V4 V4 V4 V4 V4 V4 V4 V4 V4 Ingreso Q. H1 a H96 VC2 VC2 VC2 V5 V5 V5 V5 V5 V5 V5 V5 V5 Ingreso Q. H1 a H97 VC2 VC2 VC2 V6 V6 V6 V6 V6 V6 V6 V6 V6 Ingreso Q. H1 a H98 VC2 VC2 VC2 V7 V7 V7 V7 V7 V7 V7 V7 V7 Ingreso Q. H1 a H99 VC3 VC3 VC3 VC3 V8 V8 V8 V8 V8 V8 V8 V8 V8 Ingreso Q. H1 a H910 VC3 VC3 VC3 VC3 VC3 V9 V9 V9 V9 V9 V9 V9 V9 V9 Fin ingreso H1 a H91112

Envío de datos a Niamey

DescansoDesplazamiento

1Consolidación y

envio de datos ola

Jefe de equipo Encuestador 1 Encuestador 2 Encuestador 3Listado de hogares

Selección de hogares a encuestar

Survey plan (Who visits each sample point and when)

A bad survey plan

Summer

Spring Autumn

Winter

A better survey plan

Team 5

Team 1

Team 4

Team 3

Team 2

Even better (if you can)

Teams 1, 2 and 3

Teams 4 and 5

Inter-penetrating samples (Mahalanobis,1953)

Survey Plan • Which team will visit each sampling point and when • The survey plan should consider:

– Time spent by a team at a sampling point (work plan) – Number of sample points – Total time available for fieldwork – Spatial and temporal distribution of the sample – Number of field teams – External constraints

• Weather • Security • The calendar of the evaluated project

18

Example of a survey plan

19

Piso ecológico Departamento Municipio Provincia UPM Viviendas

UPM Población

UPM Equipo Fecha de inicio Fecha de fin

1 LA PAZ Achacachi Omasuyos 20065 308 1002 1 1-Nov-11 10-Nov-11

1 LA PAZ Achacachi Omasuyos 20073 341 1304 1 11-Nov-11 21-Nov-11

1 LA PAZ Achacachi Omasuyos 20088 167 440 1 22-Nov-11 2-Dec-11

1 LA PAZ Achacachi Omasuyos 20097 360 1211 1 3-Dec-11 13-Dec-11

1 LA PAZ Caquiaviri Pacajes 20179 213 416 2 14-Dec-11 24-Dec-11

1 LA PAZ Caquiaviri Pacajes 20188 256 760 2 25-Dec-11 4-Jan-12

1 LA PAZ Puerto Acosta Camacho 20245 204 511 2 5-Jan-12 15-Jan-12

1 LA PAZ Puerto Acosta Camacho 20253 297 521 2 16-Jan-12 26-Jan-12

1 LA PAZ Puerto Carabuco Camacho 20232 266 709 3 27-Jan-12 6-Feb-12

1 LA PAZ Chuma Muñecas 20328 310 1260 3 7-Feb-12 17-Feb-12

1 LA PAZ Chuma Muñecas 20336 332 1100 3 18-Feb-12 28-Feb-12

1 LA PAZ Viacha Ingavi 20983 249 551 3 29-Feb-12 10-Mar-12

1 LA PAZ Jesus de Machaca Ingavi 21002 298 473 4 11-Mar-12 21-Mar-12

1 LA PAZ Pucarani Los Andes 20638 204 498 4 22-Mar-12 1-Apr-12

1 LA PAZ Pucarani Los Andes 20646 276 721 4 2-Apr-12 12-Apr-12

1 LA PAZ Pucarani Los Andes 20658 223 814 4 13-Apr-12 23-Apr-12

Good organization is key to:

• Minimize data collection errors – Correction of errors directly in the household – Increase in response rate – Decrease problems due to protocol adherence

• Timeliness

20

Data management

21

Field staff selection and

training

The fieldwork organization

Data management

Supervision and control

Interview

Data entry

Quality report

Supervisor review

Re-visit

Data Entry in the Field

INTERNET

Consolidated data base

Data transfer

22

FieldworkMonitoring

Production of a data base: • Must be timely • Must be well documented • Exported to statistical software Partially, to monitor field work Finally, for analytical use

23

Data Management

Characteristics of the data entry program

• Development platform available: CsPro; LSD; Blaise; etc.

• Variables dictionary • Validations

– Range checks – Checks against reference tables (anthropometrics,

food composition, etc) – Skip controls – Consistency checks

• Data consolidation and export • All programing must be ready before fieldwork

24

What happens without integration?

• The long and frustrating “data cleaning” process becomes unavoidable

The data loose their policy relevance

• Data quality is not guaranteed

“Data cleaning” converges, at best, to a database that is

only internally consistent, but does not necessarily represent reality

• “Data cleaning” entails a myriad of decisions, generally

undocumented

Users mistrust the data

25

Benefits of integration

• Reliable and timely databases • Permanent monitoring of fieldworker performance, allowing

early warning of inadequate behaviors • Guarantee that all fieldworkers apply uniform criteria

throughout the full period of data collection • Inconsistencies solved by direct observation of reality, rather

than by office guesswork • Foster the total quality culture

26

SUPERVISION AND CONTROL

27

Field staff selection and

training

The fieldwork organization

Data management

Supervision and control

Quality Control

Human Supervision

• HH selection • Check up-visit • Observation of

interviews

Computer -Assisted

• Scrutiny of questionnaires

Computer Assisted

• Monitoring of data quality indicators

28

In the field

In the central Office

Quality assurance during the field phase

Fiel

d Te

am

Interviewer controls questionnaire Supervisor verifies quality of interviewers Supervisor controls questionnaires with the aid of PED messages Return to household to correct errors, verification of inconsistencies and complete any missing information

Cent

ral S

uper

visio

n

Supervising Missions Verification of standard compliance and work schedule Verification of errors or remaining inconsistencies Statistical observations by interviewers and teams Telephone check ups

Inve

stig

atin

g Te

am

Supervising Missions Verification of errors or remaining inconsistencies remanentes Statistical observations by interviewers and teams Verification of BD conformation

29

Quality assurance during the field phase

30

INTERVIEWERS

SUPEVISION OF IMPLEMENTING FIRM

SUPERVISION OF INVESTIGATORS

SUPERVSION OF FIELD TEAM

Enquêteur Valeurs 0 et

5 Autres valeurs Total

112 70.0% 30.0% 100.0% 113 23.8% 76.2% 100.0% 114 38.9% 61.1% 100.0% 122 45.5% 54.5% 100.0% 123 38.1% 61.9% 100.0% 124 33.3% 66.7% 100.0% 132 12.5% 87.5% 100.0% 133 13.0% 87.0% 100.0% 134 22.2% 77.8% 100.0% 142 13.3% 86.7% 100.0% 143 12.9% 87.1% 100.0% 144 42.9% 57.1% 100.0% 152 47.1% 52.9% 100.0% 153 30.0% 70.0% 100.0% 154 20.0% 80.0% 100.0% 162 84.6% 15.4% 100.0% 163 35.0% 65.0% 100.0% 164 41.7% 58.3% 100.0% 171 66.7% 33.3% 100.0% 172 44.4% 55.6% 100.0% 173 33.3% 66.7% 100.0% 174 77.3% 22.7% 100.0%

Follow-up of quality indicators

Last digit distribution of de anthropometric measurements

If the measurements have

been made correctly, it would be expected that the last digit

will be equally distributed between 0 to 9, that is, 10%

each.

31

Unknown Total

0 1 0 Team 0 100.0% 100.0%

1 84.8% 15.2% 100.0% 2 51.5% 48.5% 100.0% 3 100.0% 100.0% 4 53.8% 46.2% 100.0% 5 53.3% 46.7% 100.0% 6 97.8% 2.2% 100.0% 7 41.4% 58.6% 100.0% 9 86.8% 13.2% 100.0% 11 78.9% 21.1% 100.0% 12 100.0% 100.0% 13 83.7% 16.3% 100.0%

Total 77.4% 22.6% 100.0%

Follow-up of quality indicators

32

Percentage of « unknown » answers by

team

33

Example of development of quality indicators

Example of a follow-up EQUIPO REGION ola 1 ola2 ola 3 ola4 ola5 ola 6 ola 7 ola 8 ola 9 ola 10 ola 11 ola 12 ola 13 ola 14 ola 15 ola 16

11 Agadez 2,9 2,8 1,5 1,0 1,3 0,8 1,2 0,5 0,6 0,5 0,2 1,4 1,5 0,2 0,3 0,212 Diffa 1,0 0,9 2,0 0,1 2,2 3,5 0,7 1,1 0,8 1,1 1,0 1,2 1,4 0,8 0,0 0,013 Maradi 16,1 11,5 10,8 1,2 1,5 0,9 1,2 0,5 1,7 0,7 0,5 0,2 0,2 0,1 0,2 0,414 Maradi 28,8 11,8 9,3 4,1 6,0 2,6 3,0 2,4 2,4 0,9 0,3 0,1 0,4 0,5 0,8 7,215 Tahoua 12,6 3,4 5,4 1,6 1,5 2,7 1,0 0,6 0,5 0,1 0,2 0,6 0,4 0,4 0,5 0,116 Tahoua 5,1 3,7 5,0 1,5 3,0 0,8 0,7 0,9 0,7 0,8 0,8 0,4 0,2 0,2 0,2 0,117 Tillaberi 10,6 12,3 6,0 1,0 0,7 0,7 0,2 0,1 0,2 0,1 0,1 0,1 0,4 0,2 0,2 0,518 Tillaberi 3,5 2,3 1,6 1,4 1,2 1,9 1,8 1,2 0,5 0,6 1,7 0,8 2,7 1,3 0,8 2,119 Zinder 5,7 9,3 19,6 1,8 2,0 1,9 1,8 0,5 0,1 0,3 0,2 0,4 0,3 0,3 0,2 0,620 Zinder 12,6 5,5 6,2 4,1 4,6 3,2 2,7 0,8 0,5 0,8 0,8 0,2 0,4 0,1 0,2 0,921 Zinder 71,9 9,9 8,2 3,2 2,5 1,3 4,2 1,7 1,0 2,9 2,9 2,5 2,5 1,6 1,0 0,922 Dosso 12,3 4,9 2,3 2,3 1,9 1,4 4,5 2,9 1,5 1,7 2,4 2,6 1,0 1,0 1,1 1,623 Dosso 62,1 4,1 1,9 1,0 0,3 0,2 0,1 0,0 0,0 0,2 0,1 0,3 0,1 0,1 0,3 0,124 Niamey 4,0 9,5 1,2 2,3 1,4 0,5 0,9 0,9 1,4 1,4 1,3 0,8 1,5 1,5 1,6 1,425 Niamey 6,5 4,7 1,7 1,9 1,9 0,7 0,7 0,3 0,6 0,3 0,3 1,6 1,5 0,8 0,5 0,9

Después de re entrenamientoAntes de re entrenamiento Re Entr.

34

• Requirements: – Age, gender, language, education, etc.. – Credentials – depending on the regulations of each

country- to carry out specific duties, such as: • Biometric testing • anthropometry • Cognitive testing

• Time availability and disposition to work in the field

• Selection of more candidates than needed

35

Preselection of candidates

• Strategy – Centralized/ Decentralized – Duration in weeks (2 weeks or longer) – Formation of groups(groups of no more than 30-40

people) – Specific training for each area (questionnaires, biometric

and cognitive tests, anthropometric measurements, data entry etc.)

• Stages of Training – Preparation of training material and agenda – Training of master trainers – Training of field staff

36

Training

• Detailed agenda of training – Lectures – In-classroom practice – Field practice (households, children, measurements…)

• Materials and logistics – Adequate infrastructure – Standardized slide presentations – Vignettes / case studies – Quizzes / final evaluation – Sufficient questionnaires, manuals and equipment – Refreshments – very important for morale – Transport to and from training site and per diem allowance

37

Training