Testing persuasion in the real world

The effectiveness of political actors in influencing public opinion by specific persuasive appeals is poorly understood. From laboratory experiments we have learned that persuasive stimuli (i.e. having participants watch a documentary or read a newspaper story) does have effects on attitudes. These experiments also reveal insights into to the psychological mechanisms of attitude change. We know less about the persistence of persuasive treatments over time (which is what matters in actual campaigns) and even less is known about the ecological validity of these experiments. What sounds like a technicality is crucial and addresses the core of persuasion research in political communication: It is unresolved to what degree and under which circumstances findings from artificial laboratory setting can be generalized to the real world, particularly the communication of actual political actors. Since research ultimately aims at understanding the world outside the lab, this is a crucial gap in our knowledge. Field-experiments are the natural candidate to investigate how persuasion works in applied settings. In recent years some scholars have started to conduct this kind of research with increasing intensity (among others, David Broockman has published several interesting studies on this topic)

Treatment Material
Treatment Material, Offline

Financed by Konrad-Adenauer-Stiftung and in collaboration with the German employers´ interest group Gesamtmetall we have conducted a persuasion experiment and present first and preliminary results. Andreas Jungherr is the principal investigator of this project and Matthias Mader and Harald Schoen also work on this. In the following, I will briefly present some slides from a presentation Andreas will give at the International Conference for Political Communication #IKPK16 in Berlin.

In this study, subjects received a mailing sent on behalf of ‘Gesamtmetall’ on  the issue of a free trade agreement (TTIP). Subjects (N=8.000) were randomly exposed to either a postal mailing under field conditions or to an identical facsimile as part of an online survey or no to treatment at all.

Treatment Material, Online
Treatment Material, Online

Attitudes on the issue in questions -TTIP- vary considerably. More people (42,9%) oppose the proposed trade than than people who support it (32,1%). 25 percent of respondents chose the scale mid-points of our three-item-construct on TTIP-attitudes.

Preliminary analyses of the treatment effect shows substantial persuasion among both experimental groups with larger effect sizes in the survey-experiment. In post-treatment wave 2 48% of respondents opposed TTIP and 27% supported TTIP. Those who have received a postal letter from the interest group which argued for the benefits of TTIP have considerably more favorable attitudes towards this trade deal. The share of supporter grew to 32% and the share of opposed respondents shrinked to 44%. (Yes, the effect is statistically significant at p<0.001). As expected, the effect is even stronger when the panelists have not received a physical letter, but have read the letter as part of our online survey. 45 percent (!) supported TTIP with only 32.5% opposing TTIP in this condition. So, having read the arguments by the German association ‘Gesamtmetall’ basically reverts the opinion among respondents in the survey-experimental-condition from a solid plurality against TTIP to a solid plurality of ttip-supporters.

The lower (but still substantially important) effect size in the field-experimental condition compared to the survey-experimental condition has a variety of (very interesting) causes. Among them are non-compliance (simply not opening or reading the letter) and the bigger distance in time between treatment administration and measurement in the field-experimental mode. However, importantly, while the effects of the treatment naturally decline over time, they are persistent one week after the administration of the treatment.

attitudes towards TTIP across experimental groups in wave 2
attitudes towards TTIP across experimental groups in wave 2 (Order: Support TTIP, Neither/Nor, Oppose TTIP)
attitudes towards TTIP across experimental groups
attitudes towards TTIP across experimental groups in wave 3 (Order: Support TTIP, Neither/Nor, Oppose TTIP)

The persuasion effects are highly statistically significant in the field- and in the survey-experimental-condition (and no, the result is no artifact of our trichotomization of the DV). Proper statistical analyses were already conducted. However, the relieving point is that even without fancy regression analyses our results are robust and effects can be detected with simple instruments of comparisons of means that are comprehensible without statistical education.

This posting gave a glimpse into the persuasion effect of our experiment. More generally, this is an exciting project because it allows tapping into a variety of  different kinds of unanswered questions about how attitudes develop and change. We have conducted extensive analyses on the predictors explaining attitudes toward TTIP and free trade which will be reported some time later. The research design of parallel experiments is very interesting from a methodological perspective and hopefully leads to a better understanding of differences and commonalities of field- and survey-experiments. One example for this is the chance to investigate how selective exposure and motivated reasoning (established phenomena in lab experiments) will behave in the field and how this will influence the actual treatment effects in the field compared to the effect in the survey-experiment. The wealth of data we have will also allow us to tap into the psychological mechanisms behind the persuasion we have observed. Lastly, we had included another small experimental variation concerning the content of the latter (one-sided vs. two-sided).

Andreas will talk about some of these topics this Monday at #IKPK16 in greater detail. I just want to give a peek into one phenomenon I find particularly interesting. Pundits and scholars alike often argue that the findings on persuasion would not replicate outside the lab because citizens can often choose what message to consume and people would be motivated to avoid political messages that do not match their pre-existing beliefs. Therefore, persuasive appeals would mostly preach to the converted.

Our data does not support this notion of selective exposure. The pre-treatment attitude toward TTIP or union-membership is not or only weakly correlated with whether an individual has decided to read the letter. This is the case, although we had printed on the envelope “PRO TTIP!” in the biggest letters possible and it also states that it was sent by a pro-business interest group.  We have also conducted some analyses showing  not much evidence for biased processing (motivated reasoning) either. So, there are interesting weeks ahead for those involved in project, in which we can learn more about how persuasion works in an applied setting.

 

self-reported rates of having read the postal letter by pre-treatment-attitude and by union-membership
self-reported rates of having read the postal letter by pre-treatment-attitude and by union-membership

Launch of the German Longitudinal Election Study´s panel component

This week we have launched the first wave of the German Longitudinal Election Study´s panel component. What makes the German Election Study so appealing is its extensive reach into different facets of voting behavior. For each of these dimensions (media effects, campaign effects, long-term trends etc.) different research designs were developed and tailored to better understand each of these elements. The GLES is a service to the community of electoral researches. The GLES produces high-quality social scientific data sets on voter behavior (and media coverage) and make these publicly available. Noteworthy is the over-arching framework of the GLES which integrates these components in order to facilitate research across disciplinary boundaries. For example, identical coding schemes and question formats are used across all components.

I work for the panel component of the German Election Study.  Prof Dr Harald Schoen is the principal investigator of this component. After a comprehensive pre-test in the summer, this week we have launched the first wave of the panel component. The component itself consists of three studies:

  • Long-term panel: In this study, voters are surveyed over the course of three elections. From the past election we have retained a sample of about 2,000 respondents. One interesting methodological aspect: It´s a multi-mode study with parallel online and postal surveys. The long-term panel is an amazing data source. It allows to see if and how voters react to the changing political context they are embedded in: Whether shifts of party platforms actually make voters switch their vote choices (or even turnout decisions? Hint: Yes, but to a small extent), if characteristics of the specific election mobilizes citizens who previously abstained to turn out to vote this time or how long-term changes in political attitudes (such as political efficacy or satisfaction with democracy) interacts with voting behavior. These every-day-questions of electoral research consider temporal variation and should therefore be tested with long-term panel data sets which are heavily under-utilized (not just this one, but long-term panels on elections in general). It is surprising that in electoral studies we still rely so much on cross-sectional data (despite all their problems with biased retrospection and causal identification) where it is sometimes obvious that long-term data sets would be the more adequate choice.
  • Long-term panel of the campaign panel: This is in unplanned but beloved child. The GLES started an online campaign panel in the weeks before the 2009-federal election. When the colleagues were setting up a replication for the 2013-election they discovered that many of the 2009-panelists were still active. So, they decided to recruit them again which is why for >1,000 individuals we can observe with amazing detail individual stability of dynamics not only within an electoral campaigns but also across electoral campaigns (2009-2013). We follow up on that and survey about 5,000 respondents that were active participants of previous campaign panels. Beginning with the second wave, they will be treated like ordinary members of our biggest project, the 2017 campaign panel.
  • Campaign panel: Compared to its predecessors, the 2017 campaign panels evolves both in quality and in quantity. This campaign panel starts with a drastically expanded sample size of 15,000 individuals, allowing for very fine-grained analyses of even small subgroups of the electorate (strategic voters, young voters and so on). Second, this time the campaign panel starts long before the actual campaign. The campaign panel aims at better understanding campaign effects. To achieve this we need a benchmark to which stability and dynamics during campaigns is compared.

For more information on the GLES and to obtain the data sets, visit www.gles.eu.

Winterschool on Elections and Electoral Behavior

Winter School: Bill Jacoby
Bill Jacoby on Ideology and Values

Some weeks after my research stay at André Blais´ Chair for Electoral Studies had begun, the Winter School on Voting Behavior happened to take in Montréal as well. It´s the second edition of a unique enterprise that brings together young students on electoral studies from across the globe with leading scholars in the field. PhD students from Latin- and North-America, China and Europe presented research such as on the effects of polls or cross-pressures on voting behavior.

Liran Harsgor presented a comprehensive and thought-out study on the gender gap in theWinter School: Lecture Blais United States for which she was awarded the Victor d´Hondt prize (well deserved!). For me, it was the first time to present a sketch of the Self-Determination Theory of Political Motivation. It was an experiment as the talk was bereft of any empirical analysis and focused exclusively on theoretical arguments. It was worth the effort for the positive and valuable feedback I received. You are invited to have a look at the slides here.

Winter School: Curling
Political scientists outside the office

While the afternoons were reserved for student presentations, the day began with a lecture by a senior scholar. André Blais gave an blunt (“Economic Voting is overrated”) and impressive wrap-up 100 years of electoral studies that closed with a plea for a more imperialist discipline (“My plea is: Study elections outside politics! Us, political scientists, we have to be more imperialist!”). Bill Jacoby talked about Values and Ideology and as the editor of the American Journal of Political Science he offered a spontaneous Q&A on Publishing (“If you think about including a graph in a paper, there is only one statistical software you should use: R”).

Further talks were given by Ruth Dassonneville, Patrick Fournier, Richard Johnston, Sona Golder, Richard Lau and Marc Hooghe.

 

From Mannheim to Montréal

After exciting 15 months at the Mannheim Centre for European Social Research and the Chair for Political Psychology, it is a time to start a new chapter. It will begin in Montréal, Québec.

Thanks to the amicable support of André Blais, I´ll have the opportunity to spend the upcoming months at the University of Montréal as a visiting researcher. Everyone interested in turnout behavior is familiar with the work of André Blais´ and his team of researchers at the Chair in Electoral Studies. Best known is his discussion of the merits and limits of rational choice theory with regard to voting behavior. In recent years André Blais and Carol Galais have taken up the challenge to investigate the perceived duty to vote more closely (1, 2, 3), as one of the variables with the highest explanatory power for electoral participation. Despite its central role in electoral studies, this concept of voting as a civic duty is not well understood. As my dissertation deals with the psychological mechanisms whereby social and moral norms influence political participation, our research interests intersect offering diverse opportunities for intellectual exchange. 

Apparently, the stay in Montréal will not only be of intellectual benefit. According to my guide, Montréal is said to be the cultural capital of Canada and even the New York Times has recently published a piece on the new, ‘hip’ Canada. The author lists Justin Bieber and Ryan Gosling as prime examples for Canada´s new hipness..

If Justin Bieber doesn´t convince you of hip Canada, may be these fun facts will:

  • 1 out of 10 of the world´s trees is Canadian.
  • When you are in New Foundland (Eastern Canada), it´s closer to Ireland than it is to Vancouver.
  • Canada is cold. This doesn´t mean you´d a jacket. Just walk in the tunnels below the surface. 32 km of them. From shopping malls, theaters to concert halls: You can reach everything without seeing daylight. And, yes, if you feel the urgent for absolution after your shopping trip,  the Cathedrale Christ Church is also connected to the network of tunnels of course.