Address for Correspondence Hoek van Hollandlaan 13 2554 EA The Hague The Tel:
+ e-mail:
efamro@hetnet.nl
EFAMRO
- Quality Standards for Access Panels - QSAP
printer friendly version
Download: The relevance of new Quality standards for international access panels by Dr A.J. Olivier
Access
panels are being used more and more often in market research surveys, including
Internet surveys as well as those conducted by more traditional means. The advantages of this method of sampling are
such that the use of Access panels is set to increase considerably in the years
ahead. However, the rapid growth of this
approach to sampling brings with it a number of potential dangers. At present the
market is largely unregulated and it is often difficult to find out exactly
what quality standards are being applied in recruiting panel members and how
they are maintained. For this reason
EFAMRO set itself the objective of developing a quality norm for working with
access panels.
The Quality
Standards are, to a substantial part derived from the Esomar guidelines with
respect to Online surveys and are to be treated as an appendix to, as well as
an integral part of the Efamro ISO/TCC 225 after formal approval by the EFAMRO
Board have been given.
For the
purposes of these standards an ‘Active Access Panel Member’ is defined as:-
A natural person, on a voluntary basis, returns to the panel manager a minimum set of background data, completely filled in, and via a double-active-opt-in approach unequivocally indicates that he/she wishes to become a member of the access panel concerned, on the clear understanding that the panel is solely used for purposes of market research and participates in panel activities for at least once a year.
Accepting
this definition means that the following discussion points are dealt with
- The extend to which panel members can be
involved in online direct marketing activities.
- The clear and trustworthy definition of
Active Panel members
- The minimum set of background variables,
which must be recorded
In this
document we will define the following items for each relevant quality
dimension.
- Minimum Quality specification of minimum
quality if applicable
- Transparency. The minimum degree of openness and/or the
specification of a quality level for a dimension with no specified minimum
quality.
- ISO/TCC 225 certification activities as a tool for
the implementation of guidelines.
This document was made on request of
the Efamro Board by:
Lex Olivier - Research International
Folker Michaelsen - GFK
Andrei Postoaca – Ipsos
Bryan Bates - Efamro
1
Definition of a Panel Member
Chapter 1
|
Quality Dimension |
Double active Opt-in/Confirmed
Opt-out |
|
Question |
Is the
respondent aware that he/she is a member of an access panel for market
research and has he/she voluntarily and actively applied for this? |
|
Minimum Quality |
Panel
members have actively indicated that they want to be a panel-member and have
actively agreed to the “terms and conditions” before joining.
Each
individual panel member must be given a permanent opportunity to opt-out. An email
could be sent to the panelmember |
|
Transparency |
Institute
must describe opt-in and opt-out procedures. |
|
ISO/TCC
225 Implementation |
Check if
“Double - active – opt – in” procedure is in place. Check if
“opt – out” procedures are in place. |
Chapter 2
|
Quality Dimension |
Definition of Panelsize |
|
Question |
When a
survey is conducted |
|
Minimum Quality |
No minimum requirements as long as panel sizes are honestly
stated and data about panel members is easily available in accordance with at
least one of the following two different definitions: 1 Panel members/Household members (Individuals who
joined the panel as members of recruited households) - Unique e-mail address is known - Background - Agreed to participate 2 Registered Individuals - Background - Agreed to participate |
|
Transparency |
Full and
open consistent |
|
ISO/TCC
225 Implementation |
Check for
correct and consistent handling of definitions in proposals and marketing
material |
Chapter3
|
Quality Dimension |
Unique persons/ households |
|
Question |
Is
duplication avoided before the persons concerned are added to the sampling
frame as panel members? |
|
Minimum Quality |
Each
panel member must have a unique e-mail address that can apply to a full
household. Survey
participation on the basis of individual invitation only. Password
procedure must be in place at an individual level. |
|
Transparency |
Agency
must indicate what procedures are in place. |
|
ISO/TCC
225 Implementation
|
Check if
“remove duplicates” procedures are in place. Check if password
handling procedures are in place. |
Chapter 4
|
Quality Dimension |
Available background |
|
Question |
Is the
same set of data recorded for each panel member? What is the minimum number
of existing characteristics? |
|
Minimum Quality |
·
Full Name ·
Gender ·
region ·
age (day/year of birth) ·
Postal code + house number (for validity checks and removal of duplicates) ·
Household size Information
must be actual, so a consistent update procedure is necessary |
|
Transparency |
Overview
of availability of minimal background given
/easily available. Additional
background
|
|
ISO/TCC
225 Implementation |
Check for
compliance. Minimum
background data must be 100% complete. Check if the
data are, systematically and/or periodically, updated. |
2. Panel Structure – Panel Management
Chapter 5
|
Quality Dimension |
Minimal participation level |
|
Question |
Are
requirements made concerning the response of a panel member within a certain
period, in order for him/her to be regarded as an ‘active’ panel member? |
|
Minimum Quality |
A
panelmember must participate at least once a year in a survey activity or a
panel activity in order to stay on the panel. If this is not the case, than a
panelmember must be removed. |
|
Transparency |
Agency
must communicate the number panel members and/or the number of registered
individuals. |
|
ISO/TCC
225 Implementation |
A panel
activity monitoring program must be in place Inactive
panel members must be removed from the panel. |
Chapter6
|
Quality Dimension |
Research burden |
|
Question |
How
frequently are panel members approached for research in a certain timeframe? |
|
Minimum Quality |
Maximum
participation burden in terms of number
of completed surveys must be specified. |
|
Transparency |
Transparency
through describing maximum burden |
|
ISO/TCC
225 Implementation |
Check if
a system is in place to monitor the compliance with targeted maximum research
burden and |
Chapter7
|
Quality Dimension |
Recruitment |
|
Question |
In what
way does recruitment take place; what source(s) is/are used? |
|
Minimum Quality |
Potential
Panelmembers must be invited. Unsolicited joining will be impossible. A Panel
recruitment policy must be in place. Recruitment
campaign effectiveness must be evaluated. |
|
Transparency |
Agency
must reveal its recruitment policies in general terms. There is
no obligation to reveal the individual identity of recruitment
subcontractors. |
|
ISO/TCC
225 Implementation |
Check for
existence of: Evaluations
of recruitment effectiveness. Existence
of accreditation programs for recruitment subcontractors. |
Chapter 8
|
Quality Dimension |
Exclusively for market research |
|
Question |
Does the
access panel serve exclusively as a sampling frame for market research or are
panel members also deployed for other activities |
|
Minimum Quality |
A
panelmember should not be approached for any DM activity by the research
company |
|
Transparency |
Clear
statement |
|
ISO/TCC
225 Implementation |
Check if
agency uses panels for DM activities and if so if what tools are in place to
assure compliance with minimum quality criteria. |
Chapter 9
|
Quality Dimension |
Panel relations |
|
Question |
If panel
members are rewarded for their participation, how does the panel organization
implement this and how is the relation with the panelmembers maintained. |
|
Minimum Quality |
A
question from a consumer must be answered within 24 hours (weekdays) If this
is not possible, an acknowledgement of receipt must be given. This will
include an ultimate reaction time period |
|
Transparency |
Agency
must make public |
|
ISO/TCC
225 Implementation |
Check if
a loyalty/reward system is in place Check if
a loyalty/reward system is in line with promise Check if
employees are available in order to answer questions. |
3. Project management
Chapter 10
|
Quality Dimension |
Response control |
|
Question |
How does
an agency measure and monitor non-response rates and completion rates. |
|
Minimum Quality |
The
agency must report response rates and completion rates per project and take
action when results are not in line with targets. For the
following topics Ø Number of mails send to members Ø Panelmembers who opened the questionnaire Ø How many panel members screened
out Ø How many panel members Dropped out Ø How many panel members Completed
the survey Over the
year the performance on these indicators must be measured and evaluated |
|
Transparency |
The
agency must define clearly quantified targets in terms of response rates and
completion rates. |
|
ISO/TCC
225 Implementation |
Check if
response rates and completion rates are measured. Check if
response rates and completion rates are reported to client. Check if
actions are undertaken in case of structural non-compliance. |
Chapter 11
|
Quality Dimension |
Survey Setup |
|
Question |
Each
panelmember must have the same chance to participate in the study. How does
the agency setup the survey? |
|
Minimum Quality |
All panel
members eligible for participation in a study must have an equal chance to
participate in the study. Invitation
to participate must be based on a predefined stratification. |
|
Transparency |
Information
about the sample algorithms and relevant panel management rules must be available |
|
ISO/TCC
225 Implementation |
Check if
a system is in place to monitor the compliance with targeted maximum survey
burdens of panel members. Check if
sampling algorithms are used. |
Chapter 12
|
Quality Dimension |
Survey Bias Control |
|
Question |
The
answer of respondents on questions may not be influenced by the
participations in earlier surveys. |
|
Minimum Quality |
Exclusions
on the basis of individual survey participation history must be possible. The
agency my must have procedures in place to prevent a respondent, filling in
the questionnaire twice. |
|
Transparency |
The
agency must clearly state how survey histories are tracked and how
respondents are excluded from participation if relevant. |
|
ISO/TCC
225 Implementation |
Check if
a participation history per respondent is available and whether it is
actually used. Agency my
must guarantee one interview per respondent and have visible procedures in
place |
4 Legal Issues
Chapter 13
|
Quality Dimension |
Legal Compliance |
|
Question |
Does the panel adhere to all demands of the
applicable legal obligations of governmental and industry bodies?
|
|
Minimum Quality |
Must
adhere |