Address for Correspondence

Hoek van Hollandlaan 13

2554 EA The Hague

The Netherlands

Tel:  +31-10-2894 505

e-mail:    efamro@hetnet.nl

EFAMRO - Quality Standards for Access Panels - QSAP

Access panels are being used more and more often in market research surveys, including Internet surveys as well as those conducted by more traditional means.  The advantages of this method of sampling are such that the use of Access panels is set to increase considerably in the years ahead.  However, the rapid growth of this approach to sampling brings with it a number of potential dangers. At present the market is largely unregulated and it is often difficult to find out exactly what quality standards are being applied in recruiting panel members and how they are maintained.  For this reason EFAMRO set itself the objective of developing a quality norm for working with access panels.

The Quality Standards are, to a substantial part derived from the Esomar guidelines with respect to Online surveys and are to be treated as an appendix to, as well as an integral part of the Efamro ISO/TCC 225 after formal approval by the EFAMRO Board have been given.

For the purposes of these standards an ‘Active Access Panel Member’ is defined as:-

A natural person, on a voluntary basis, returns to the panel manager a minimum set of background data, completely filled in, and via a double-active-opt-in approach unequivocally indicates that he/she wishes to become a member of the access panel concerned, on the clear understanding that the panel is solely used for purposes of market research and participates in panel activities for at least once a year.

Accepting this definition means that the following discussion points are dealt with

• The extend to which panel members can be involved in online direct marketing activities.
• The clear and trustworthy definition of Active Panel members
• The minimum set of background variables, which must be recorded

In this document we will define the following items for each relevant quality dimension.

• Minimum Quality specification of minimum quality if applicable
• Transparency. The minimum degree of openness and/or the specification of a quality level for a dimension with no specified minimum quality.
• ISO/TCC 225 certification activities as a tool for the implementation of guidelines.

This document was made on request of the Efamro Board by:
Lex Olivier - Research International

Folker Michaelsen - GFK

Martin Oxley – TNS/Sofres

Andrei Postoaca – Ipsos

Bryan Bates - Efamro

1 Definition of a Panel Member

Chapter 1

Quality Dimension	Double active Opt-in/Confirmed Opt-out
Question	Is the respondent aware that he/she is a member of an access panel for market research and has he/she voluntarily and actively applied for this?
Minimum Quality	Panel members have actively indicated that they want to be a panel-member and have actively agreed to the “terms and conditions” before joining. Panelmembers have confirmed the existence of their e-mail address. Each individual panel member must be given a permanent opportunity to opt-out. An email could be sent to the panelmember informing them that their background information will be deleted within 30 days.
Transparency	Institute must describe opt-in and opt-out procedures.
ISO/TCC 225 Implementation	Check if “Double - active – opt – in” procedure is in place. Check if “opt – out” procedures are in place.

Chapter 2

Quality Dimension	Definition of Panelsize
Question	When a survey is conducted information about the actual panelsize is  needed. This in order to judge if a panel can cope with the estimated incidence rate or with a scheduled tracking setup.
Minimum Quality	No minimum requirements as long as panel sizes are honestly stated and data about panel members is easily available in accordance with at least one of the following two different definitions: 1  Panel members/Household members (Individuals who joined the panel as members of recruited households)      - Unique e-mail address is known      - Background information available      - Agreed to participate 2  Registered Individuals     - Background information available (Name-Gender-Age)     - Agreed to participate
Transparency	Full and open consistent information must be given
ISO/TCC 225 Implementation	Check for correct and consistent handling of definitions in proposals and marketing material

Chapter3

Quality Dimension	Unique persons/ households
Question	Is duplication avoided before the persons concerned are added to the sampling frame as panel members?
Minimum Quality	Each panel member must have a unique e-mail address that can apply to a full household. Survey participation on the basis of individual invitation only. Password procedure must be in place at an individual level.
Transparency	Agency must indicate what procedures are in place.
ISO/TCC 225 Implementation	Check if “remove duplicates” procedures are in place. Check if password handling procedures are in place.

Chapter 4

Quality Dimension	Available background information
Question	Is the same set of data recorded for each panel member? What is the minimum number of existing characteristics?
Minimum Quality	· Full Name · Gender           · region · age (day/year of birth) · Postal code + house number (for validity checks and removal of duplicates) · Household size Information must be actual, so a consistent update procedure is necessary
Transparency	Overview of availability of minimal background information must be given /easily available. Additional background information can be added to the list if this is done in combination with the following : The number of panelmember for which specific additional background                information is available, must be stated.  An indication about the recency of the data collection must be given.
ISO/TCC 225 Implementation	Check for compliance. Minimum background data must be 100% complete. Check if the data are, systematically and/or periodically, updated.

2. Panel Structure – Panel Management

Chapter 5

Quality Dimension	Minimal participation level
Question	Are requirements made concerning the response of a panel member within a certain period, in order for him/her to be regarded as an ‘active’ panel member?
Minimum Quality	A panelmember must participate at least once a year in a survey activity or a panel activity in order to stay on the panel. If this is not the case, than a panelmember must be removed.
Transparency	Agency must communicate the number panel members and/or the number of registered individuals.
ISO/TCC 225 Implementation	A panel activity monitoring program must be in place Inactive panel members must be removed from the panel.

Chapter6

Quality Dimension	Research burden
Question	How frequently are panel members approached for research in a certain timeframe?
Minimum Quality	Maximum participation burden in terms of number of completed surveys must be specified.
Transparency	Transparency through describing maximum burden
ISO/TCC 225 Implementation	Check if a system is in place to monitor the compliance with targeted maximum research burden and

Chapter7

Quality Dimension	Recruitment
Question	In what way does recruitment take place; what source(s) is/are used?
Minimum Quality	Potential Panelmembers must be invited. Unsolicited joining will be impossible. A Panel recruitment policy must be in place. Recruitment campaign effectiveness must be evaluated.
Transparency	Agency must reveal its recruitment policies in general terms. There is no obligation to reveal the individual identity of recruitment subcontractors.
ISO/TCC 225 Implementation	Check for existence of: Evaluations of recruitment effectiveness. Existence of accreditation programs for recruitment subcontractors.

Chapter 8

Quality Dimension	Exclusively for market research
Question	Does the access panel serve exclusively as a sampling frame for market research or are panel members also deployed for other activities
Minimum Quality	A panelmember should not be approached for any DM activity by the research company
Transparency	Clear statement
ISO/TCC 225 Implementation	Check if agency uses panels for DM activities and if so if what tools are in place to assure compliance with minimum quality criteria.

Chapter 9

Quality Dimension	Panel relations
Question	If panel members are rewarded for their participation, how does the panel organization implement this and how is the relation with the panelmembers maintained.
Minimum Quality	A question from a consumer must be answered within 24 hours (weekdays) If this is not possible, an acknowledgement of receipt must be given. This will include an ultimate reaction time period
Transparency	Agency must make public
ISO/TCC 225 Implementation	Check if a loyalty/reward system is in place Check if a loyalty/reward system is in line with promise Check if employees are available in order to answer questions.

3. Project management

Chapter 10

Quality Dimension	Response control
Question	How does an agency measure and monitor non-response rates and completion rates.
Minimum Quality	The agency must report response rates and completion rates per project and take action when results are not in line with targets. For the following topics information must be available on a project-by-project level and for the panel in general. Ø Number of mails send to members Ø Panelmembers who opened the  questionnaire Ø How many panel members screened out Ø How many panel members Dropped out Ø How many panel members Completed the survey Over the year the performance on these indicators must be measured and evaluated
Transparency	The agency must define clearly quantified targets in terms of response rates and completion rates.
ISO/TCC 225 Implementation	Check if response rates and completion rates are measured. Check if response rates and completion rates are reported to client. Check if actions are undertaken in case of structural non-compliance.

Chapter 11

Quality Dimension	Survey Setup
Question	Each panelmember must have the same chance to participate in the study. How does the agency setup the survey?
Minimum Quality	All panel members eligible for participation in a study must have an equal chance to participate in the study. Invitation to participate must be based on a predefined stratification.
Transparency	Information about the sample algorithms and relevant panel management rules must be available
ISO/TCC 225 Implementation	Check if a system is in place to monitor the compliance with targeted maximum survey burdens of panel members. Check if sampling algorithms are used.

Chapter 12

Quality Dimension	Survey Bias Control
Question	The answer of respondents on questions may not be influenced by the participations in earlier surveys.
Minimum Quality	Exclusions on the basis of individual survey participation history must be possible. The agency my must have procedures in place to prevent a respondent, filling in the questionnaire twice.
Transparency	The agency must clearly state how survey histories are tracked and how respondents are excluded from participation if relevant.
ISO/TCC 225 Implementation	Check if a participation history per respondent is available and whether it is actually used. Agency my must guarantee one interview per respondent and have visible procedures in place

4 Legal Issues

Chapter 13

Quality Dimension	Legal Compliance
Question	Does the panel adhere to all demands of the applicable legal obligations of governmental and industry bodies? Privacy Legislation Tax legislation Lottery legislation ESOMAR guidelines
Minimum Quality	Must adhere
Transparency	Must publicly announce
ISO/TCC 225 Implementation	Legal Check

	Expected Future Quality Dimensions
Question	The Committee discussed the quality issues which will need more attention in the near future
Shirking	The self completion character of online surveys means that research outcomes may be influenced by shirking behaviour. Definition of shirking and the development of tools which can measure and benchmark shirking behaviour. Questionnaire which do not match minimum criteria should be excluded from the data file
Stratification internet usage	Sample stratification is currently done based on gender/age/region. It should be investigated whether other criteria like heavy/light internet usage and family cycle should replace the old criteria
Home vs Work	Respondents taking interviews from the workplace could face legal actions from employers. A self regulation from the research industry could be needed to avoid legal claims against research agencies
Downtime	If a server goes down during an interview as a result of technical breakdown or as a result of surfing overload this may influence results. Has the agency procedures in place to deal with this issue
Leading invitations	Invitation letters should not auto generate participation. Non existence has to be guaranteed.