ActiSKU - Tag - Online Research

Does the internet provide reliable information?

Bennet Holmes — Wed, 25 Feb 2009 16:29:00 +0000

The EXPLOSION of online research !

At a time when the internet was still a “trendy” tool used by the younger generation, mostly written in C++, accessible mainly in cities due to lack of adequate networks in rural areas, and subject to ADSL connections (still rare 10 years ago)… online research was given a bad label.

The main problem was “non-representativeness”; a “ball-and-chain” that made it inadequate since it did not provide proper sampling.

In other words, for the internet to become a viable research tool, the surveyed cross-section should be more representative of the overall French population. This has not been the case and a disparity still exists even now. Experts respond to this by questioning whether representation is truly necessary.

Upon further reflection…

According to the figures published in the study referenced below, the profile of the French population versus that of internet surfers is as follows: 42% men as opposed to 53% amongst internet surfers; with respect to the CSP+ (upper socio-professionals): 20% men in the French population as opposed to 34.4% amongst internet users. To be even more precise: 18.5% in France as opposed to 22% on the Ile de France.

In spite of a significant internet penetration rate, there are still some very significant disparities in terms of age: 72% for 12-17 year olds and only 40% for 60-69 year olds (even though the growth in the number of internet surfers for the latter population was very strong from 2007 to 2008).

The question one might ask is whether or not marketing research groups systematically demand national representation for their research? Apart from surveys on general topics, political surveys and a few other research fields, the answer is NO.

Research companies often prefer criteria that are more representative for a given population: women, professionals, children, managers, etc.

In other words, if the methods are employed properly (to avoid surveying targets that don’t use the internet very much) and in proper context, the internet appears to be an excellent research tool, mainly because of its speed of implementation and low cost.

Sensitive data: sampling methods

Throughout history, promoters of “probabilistic” sampling methods were always opposed to “pragmatists” (non-probabilistic). They have always believed that research results cannot be correctly interpreted without calculating error margins. For them, only the scientific method can produce reliable results.

On the other hand, pragmatists, who favour quota methods and defend a more “open” approach, such as the one that uses the internet, do not dismiss results obtained using non-probabilistic methods. Today these people are among those who accept online methods consider the results obtained thereby to be sufficiently convincing.

In another, recently-published bulletin that addresses the benefits of using online research for market evaluation, we state that online sampling cannot be probabilistic, for at least one good reason: it is impossible to send thousands of emails to internet surfers since anti-spam laws forbid this in some countries (France allows this, as long as it is for research purposes).

Standard methods of research definitely have the advantage of being probabilistic but, in spite of this, they have quite a few drawbacks, as even the authors of the research cited below will concede. It forces them to admit that true probabilistic sampling does not exist! Surveys are often restricted to large cities, due to cost reasons, and some areas are never sampled (suburbs and other areas), etc.

Proving its worth

To counter these online “problems”, survey and research companies have quickly realised that it was useful to create completely self-contained online Access Panels and some companies have made it their speciality (e.g. Panel on the web). It is also important to diversify the panellist recruitment methods as much as possible by mixing off-line and online as a means of ensuring reliability and quality.

At least two pitfalls still need to be avoided: the results have to be adjusted to compensate for the differences between the sampling structure and that of the targeted population and, above all, to avoid “self-selection” that allows internet surfers to declare themselves as participants in a studies that only collect names, by simply clicking on a pop-up or some other internet media.

Even with these precautions, online studies must be set up perfectly in order to produce reliable results. To ensure this, some organisations perform “counter-studies” in order to validate their online results.

It is due to this “professionalisation” and the use of multiple filters that, today, close to 25% of research is performed “online”.

This reflection and this article were taken from a very interesting document published in September, 2008 by Syntec Etudes Marketing et Opinion /afm: http://www.revue-ram.org/documents/entreprises/terradot.pdf

The market Research

Pascal Bianchi — Wed, 25 Feb 2009 16:00:00 +0000

The world market research represents a sales turnover of 28.2 billion dollars in 2007 according to the annual Esomar report.

The European market has captured 43% of the world market with a sales turnover of 10.6 billion dollars. The North American market is second with 36%, then the Asia-Pacific rim (14%), Latin America (5%) and the MEA region (2%). All this in a world-wide market that increased by 6.8% in 2006 (an increase of 14.6 billion dollars).

The research market has greatly expanded in the last few years. On one hand, the major, well-known, world research institutes such as ACNielsen, Taylor Nelson Sofres, IMS and GfK are continuing their ascent by launching into other growing markets and consolidating their international presence. The top ten world-wide research groups reported a sales turnover of more than 11 billion Euros in 2004 (a growth of 19.5% over 2003).

On the other hand, smaller companies are maintaining their market share by providing research analysis software, strategic marketing advice and online methodologies. Internet research is thus enabling small companies to compete with the larger ones by offering simplified, “low-cost” studies that are custom-designed for specific needs.

According to Esomar, online quantitative studies represented 25% of all the research studies in 2007. According to Synthec Etudes de Marché et D’opinion, online studies (qualitative and quantitative) represented 22% of the research market in France in 2007.

Comparison of Study Techniques (source: Synthec Etudes, AFM):

Benefits of online studies versus in-field studies

Accessibility to numerous and varied targets. Ability to recruit rare profiles with low incidence rates collection speed. Background information and pre-qualification of respondents Sincerity and richness of responses. Sometimes lower cost.

Quality problems with online results

There are problems due to the very nature of online sampling but these will become less significant over time as the Internet penetration rate increases. Non-representation o Non-probabilistic sampling Risk of “professionalisation” of respondents Collection method: Can an Internet surfer represent a non-surfer? Projectability of the results (external validity)

Quality problems with “in-field” results

In-field testing, which is often considered to be more “qualitative” and more costly, is, in fact, rarely probabilistic:

o Cost-wise: in order to minimize travelling expenses, surveys are often limited to large cities o Street surveys are often limited to the busy main streets o Digicodes are making many areas and random streets more and more difficult to access o Some difficult suburbs are never surveyed ( this is also true for very rich areas ) o As well as certain entire regions that are rarely visited by pollsters (the “ the center of France and most rural areas).

Comparison of the quality of in-field and online surveys results

Variations in one particular study, conducted in 2007 with 4,427 volunteers randomly selected from an online XL panel (Kristof De Wulf and Sam Berteloot), were analysed according to the recruiting t methods used (phone, face to face, emails, web-posted forms, ad campaigns on the internet etc.) while the socio-demographic characteristic of the sampled population was filtered out so that the differences were only due to variations in the sourcing method.

In this analysis, the following factors were evaluated:

The participation rate. The time to complete the questionnaire. The consistency of responses. The depth of responses to open questions (number of words). The response style (monotone or varied). The level of knowledge. The consumption profile

The results show that:

There is no difference in the consumption behaviour (food, drink, equipment, leisure, political involvement etc.). There is no difference in the quality of responses (consistency, depth), except in responses obtained by email (fewer responses), and by Internet (higher click rate). There is no difference in motivation to participate in the study

Motivation of volunteers in online panels

One online, XL panel study of 2,084 panellists used random drawings and donations to charitable institutions as a means of motivation (Annelies Verhaghe 2007):

I want to express my opinion: 32%. I want to help the researcher by participating in the study: 27%. I want to have a chance to win a prize: 20%. I enjoy answering questionnaires: 10%. This is a way of supporting charity: 8%. I like learning things: 2%. Other: 1%

Impact of the duplication of panellists in several panels

Two studies in England conducted by Harris Interactive (2006) with 1,983 panellists (of which there were 133 multi-panellists) and 2,071 panellists (of which there were 166 multi-panellists) showed that:

There is no difference in age or sex. There were slightly more unemployed persons amongst the multi-panellists (6.7% as opposed to 5.2%). They respond more but also abandon the survey more often, thus an identical rate. The quality of their responses is slightly higher (consistency, depth). They are not more motivated by incentives. On the other hand they actually show a greater desire to learn new things