This article is available in: PDF HTML Potential sources of bias in the reporting and interpretation of gambling research findings

Journal Information
Journal ID (publisher-id): jgi
ISSN: 1910-7595
Publisher: Centre for Addiction and Mental Health
Article Information
Article Categories: opinion
Publication date: November 2020
Publisher Id: jgi.2020.45.10
DOI: 10.4309/jgi.2020.45.10

Potential sources of bias in the reporting and interpretation of gambling research findings

Paul Delfabbro School of Psychology, University of Adelaide, Adelaide, Australia
Daniel King School of Psychology, College of Education, Psychology and Social Work, Flinders University Adelaide, Australia
Alex Blaszczynski School of Psychology, Brain and Mind Centre, Gambling Treatment and Research Clinic, The University of Sydney, Camperdown, Australia

Introduction

Over the last decade, increasing attention has been directed to specific problems confronting the social sciences. These concerns have included not only well-documented difficulties in replicating major research findings (Open Science Collaboration, 2015), but also problems regarding the nature of the scientific process itself (Chambers, 2017). A number of these concerns have been articulated by Chambers (2017) in his book The Seven Deadly Sins of Psychology. This book was written not only to highlight the potential causes of the “replication crisis,” but also to call attention to important sources of bias and unreliability in social science research. Chambers provided a detailed account of the numerous ways in which the validity and reliability of research can be compromised. Certain of these “sins” were generally self-evident, and included fraud (e.g., the fabrication of data) and the withholding of data from independent scrutiny. Other practices, however, were more subtle. Examples here included the practice of massing or “data tuning” until it yields the results required; “HARKing,” in which the study’s hypotheses are reframed after the results are known; and various forms of “p-hacking,” in which data are analysed or collected to ensure statistical significance. Common examples of “p-hacking,” Chambers observed, included the selective addition of cases to a sample to obtain significance; selective non-statistically-justified removal of cases to increase effects; and the use of multiple analytical test strategies until one yields significance.

To avoid this type of confirmation bias, Chambers argued for a revision of existing practices towards greater transparency of method; the pre-registration of hypotheses; and the greater sharing of data and other research information. Support for Chambers’s concern was further evidenced in the current social and scholarly drive towards open science. Increasingly, funding agencies such as the National Institutes of Health, Welcome Trust and certain European Commission frameworks mandate open access, including open access to data (Masuzzo & Martens, 2017). The emerging zeitgeist is for increased accessibility and cross-validation of data through repositories allowing access to data, software codes, papers, and open review. This process can be aided through the Internet and its capacity for rapid download, reanalysis and upload facilities (Masuzzo & Martens, 2017).

All of these issues are likely to be relevant to the field of gambling studies. As an area which spans such disciplines as psychology, public health, the social and the medical sciences, gambling researchers face many of the same indicated dilemmas. The research topics studied often involve human interactions within complex environments. Situations also arise in which researchers search for small effects in environments that are not ecologically valid (e.g., simulated laboratory experiments using analogue populations). Researchers may also experience difficulties obtaining sufficiently large samples of participants or will be faced with data that feature outliers (e.g., exceptionally high expenditure figures) that require decisions to be made about the inclusion or exclusion of certain data points. Moreover, as with their counterparts in other fields, gambling researchers are faced with the same needs to publish in the most prestigious journals in the full knowledge that their findings are more likely to be accepted and cited if they are novel and statistically significant.

Gambling does, however, differ in one important way from many of the classic studies of social and cognitive psychology that feature heavily in Chambers’s book: it is a social issue of relevance to government policies, and therefore to its funding bodies. Thus, unlike certain of the often more abstract topics reported in laboratory psychology experiments (e.g., priming or social conformity), researchers are more likely to conduct gambling research because of its wider population-level policy impacts. In this sense, gambling research has much more in common with public health and medical science, where important findings can attract government interest and publicity because of the potential social utility of the findings. In response, there is now a dedicated body of research that examines the extent to which findings of studies (however conducted) are presented to the public accurately (Sumner et al., 2016), particularly in the context of social equity, justice and allocation of scarce resources. Evidence suggests that the media and press-releases will often overstate the findings, leave out important details (e.g., tests were on animals and not humans), or omit the results from control groups (Yavchitz et al., 2012). In effect, the pressure to gain recognition often results in ethically questionable premature reporting of findings resulting in unfulfilled longer-term expectations. However, much of this research usually assumes that the biases arise largely in the translation of findings to the development of press-releases and media stories.

In this paper, we argue that there is another form of significant bias that science communications research has perhaps given less attention. This bias is the one present in the reporting of findings themselves in the original source documents. In our view, this is emerging as a significant area of concern in the field of gambling studies and can be observed in papers that otherwise may display few of the serious problems outlined in the Chambers book. Our concerns build principally upon Chambers’s concerns about confirmation bias in research. In particular, we express concerns about the extent to which researchers inflate the importance or implications of their findings. We argue that because gambling is a highly politicised and ideologically emotive area this bias is likely to be driven by multiple influences. Not only may researchers be tempted to report findings consistent with a study-specific hypothesis, but they also may be influenced by broader ideological and pragmatic considerations. These considerations include the need to render findings consistent with the funding body’s agenda (which may lead to further funding) or to confirm a particular position on the nature of gambling (e.g., that it is harmful and exploitative).

We argue that several factors have contributed to gambling research becoming increasingly prone to these biases. The first is the heavy reliance of gambling research on government funding. For example, in Australia, New Zealand and the United Kingdom, many projects are funded by government foundations or departments that have a strong focus on research with implications for reducing harm. As a result, there is often a need to demonstrate “impact” or policy significance as part of meeting the specifications of the project brief.1 A second and related issue is that many major studies into gambling are in the form of government reports. Thus, even if the findings are ultimately reported in peer-reviewed journal articles, the original presentation of the material may appear in a report that has to meet the specifications and needs of the government funding body. In such situations, built-in safeguards may exists that require government pre-approval prior to release or submission for publications, and rights to modify aspects of the report. In fact, direct influence by the government funding agency over design or measures to be used, may sometimes be reported as “collaborative” interactions. A third important factor is the growing influence of public health approaches and agendas. Public health approaches have the potential to benefit gambling studies by helping to provide an integrated framework to organise ideas relating to prevention and intervention (Korn & Shaffer, 1999), and many of these now include exhortations for political action. For example, in certain papers (e.g., David et al., 2019, 2020) research has been described as a vehicle for “creating urgency” or to inform action. The link between research and policy is also articulated by van Schalwkyk and colleagues (2020), who refer to a need for “an evidence-based joined up response” to inform “gambling control” (p. 1681).

Aims of the present paper

In this paper, we examine three principal areas of reporting bias in gambling research which we believe will be symptomatic of, or likely to be exacerbated by, the increasing trend towards agenda-driven research. Although this analysis is unlikely to be exhaustive, we believe that it covers a range of concerns that we have observed in a number of reports and papers. We have divided these into three categories: (1) rhetorical argument; (2) confirmation bias and its forms (building upon Chambers’s, 2017 arguments); and (3) overgeneralization and what we term “denominator neglect.” The rhetorical argument section refers to the misleading over-sell of findings, selective reporting and biased framing of information to create a false sense of urgency. The confirmation bias section refers to argumentative sleights of hand used to strengthen the reliability of findings and to over-extend the implications of findings. The final section examines the care that must be taken when extrapolating estimates to populations without a careful consideration of the base rate of the phenomena (which sometimes may be quite low). Where possible, we have tried to illustrate our points with examples drawn from good quality journals or reports, but part of our critique is also drawn from our experiences as reviewers where it has been possible to address certain identified biases prior to publication.

Section 1: Rhetorical argument

A principal tenet of scientific communication is the need to be independent, objective, transparent (disclosing financial conflicts of interests, moral or ideological positions and membership of advocacy groups), balanced and to avoid emotive statements in the justification of research studies and in the description of findings. When studies are introduced, the aim should be to highlight the “research problem” or “gap,” and to describe the evidence that supports a particular line of investigation. Certain of this evidence may, or may not, be entirely consistent with the aims of the project. In essence, whether the research is confirmatory or exploratory, the aim is to build up a conceptual, logical or theoretical argument to justify the investigation and interpretation of data. This justification should not be sententious (overly moralizing) or tendentious (leading towards a particular conclusion). By avoiding these styles, the author avoids begging the question, or “over-selling” the study, and encourages the reader (and perhaps the researchers) to interpret the research in an objective way. In effect, the authors avoid situations where arguments are being replaced by rhetoric. An objective and more neutral style makes it easier to accept the results as they are even if they do not appear to support the direction of the funding body. In this section, we draw attention to a number of rhetorical trends in gambling research that compromise these ideals and principles.

(a) Building urgency

As discussed, in public health a sub-field termed “advocacy research” is now operating, one that promotes the importance of using research findings to inform important policy issues and political agendas (David et al., 2019, 2020). In principle, this is not a problem. Translating academic findings to the needs of broader stakeholders is important. However, the term “building urgency” may imply that research is potentially more valuable if it shows something, reveals policy-relevant findings, and strengthens the need for action. In our experience, several questionable ways exist in which this urgency is being created. One way is by using emotive or exaggerated language (“dangerous or unhealthy consumption or formats,” ‘harmful commodity industries,” “coercive commodities,” “companies that profit from this misery” and “toxic”) (Abbott, 2020; Hancock & Smith, 2017; van Schalkwyk et al., 2019; Young & Markham, 2017) in the place of “gambling.” In our view, this practice is somewhat akin to the practice of journalists or lawyers colouring the opinion of individuals using phrases such as “convicted smuggler” or “disgraced politician.” Another common strategy is to overuse declarative rather than evidence-supported narrative styles (e.g., “Gambling has been identified as a threat to health” (van Shalkwyk et al., 2020, p. 1680) or “The production of coercive commodities has become an increasingly significant economic project of fractions of the capitalist class” (Young & Markham, 2017, p. 2762) or “We should not be complacent about this; any problem gambling among this group is too much” (Wardle, 2017, p. 6). Such statements are articulated as rhetorical self-evident truths without any appropriate qualification and, as Shaffer and colleagues (2020) caution, repeating claims of this nature frequently could lead to its uncritical acceptance as fact.

A third, and perhaps more common trend, is what might be called the “avalanche of hypothesis-consistent statements” approach, in which an initial proposition is followed by a series of connected statements to convince the reader that a problem exists. A common example occurs in youth gambling research using the following argument. Young people are vulnerable. They are more likely to take risks. Gambling involves risk. Gambling opportunities are expanding into new technologies. Young people use many new technologies. This means that they are more likely to gamble, experience harm and that there is a need to take steps to control gambling. In reality, there may only be evidence for each of these statements in isolation. Although reviews do show that there are adolescent risk and protective factors that may predict problem gambling in adulthood (Dowling et al., 2017), individual level analysis in fact suggests that adolescent gambling may often not be a strong predictor of adult behaviour (Delfabbro et al., 2009, 2014, 2016; King et al., 2020). Such qualifications need to temper studies of adolescent gambling and its potential long-term consequences.

A final method to create urgency is to use direct rhetoric. The government or other parties might be chastened for “failing to act” (Thomas et al., 2018) or there may be calls for advocacy from organisations to advance public health agendas (Abbott, 2020). Papers may also refer to certain lines of inquiry as being “vital” or that certain trends “cannot be denied” (Wardle, 2019).

(b) Inferred consensus

Certain papers and reports resort to rhetorical devices such as inferred consensus to bolster their claims. Such techniques are consistent with known measures of persuasion (Cialdini, 2007) in which claims are thought to be more persuasive if they demonstrate the existence of broader social consensus or imperatives. Such rhetorical strategies are evident in generic phrases such as “most people would support,” “the community expects/demands” or “action is required.” Alternatively, this rhetoric may take more subtle forms such as in phrases such as “there is a need” or “it/this is concerning,” which immediately raises a question for whom the need exists or who might be concerned (examples in van Schalkwyk et al., 2019). The former statement could be interpreted as implying the existence of some unnamed party (presumably policy makers or the community) whose position on the topic is being inferred, but not investigated in the research. It creates a false sense of consensus or inferred obligation to take the results more seriously because other people or the community agree with the views expressed.

(c) Validity of parallels with tobacco

In the field of gambling studies, this rhetorical strategy occurs most commonly when researchers draw parallels between gambling and tobacco and use argument by analogy. Gambling is addictive; so is nicotine. Both can be harmful. Both are supplied by large companies with considerable lobbying power. Both raise issues about the safety of the products and the extent to which the burden of harm falls on more vulnerable users. Public health researchers, in particular, point to the successful controls placed on the supply of tobacco, the reforms to the industry and the nefarious tactics used by the tobacco industry (Big Tobacco) to suppress information and block reform. Based on this logic, advocates of reform (e.g., Cassidy, 2014; David et al., 2019; Thomas et al., 2016) argue that controls similar to those imposed on tobacco should be imposed on the gambling industry. The advocates even imply that the two are connected. A study by Nyemcsok et al. (2018), for example, refers to “[p]otential synergies between tobacco and gambling research” (p. 1076). Without qualification, this could easily be interpreted to imply a direct connection between the two industries, but where the intention may have been more modest: namely, to draw parallel between the modus operandi of both industries. For balance, it would be important to draw attention to changes in ethical requirements and research practices that have occurred as a consequence of the influence of tobacco industries to prevent repetition of documented malpractices. Gambling industry operators are under scrutiny to maintain arm’s length distance and industry funded researchers’ studies are exposed to greater levels of critique.2

However, as we have argued before, gambling is not the same as smoking. Many forms of gambling (e.g., bingo, lotteries) have little association with harm. Even the most generous estimates suggest that only 3% of people gamble on EGMs every week in Australia (Delfabbro & King, 2020) and, of these people, around 15–20% are likely to experience problems. Around 1% of the adult population are classified as problem gamblers at any point in time and not all of these experience significant harm (Browne et al., 2016). Although 1% is still of significant public health concern and parallels certain other serious disorders (e.g., certain forms of mental illness), the ratio of high risk cases to general gambling involvement is low. Indeed, as Shaffer and colleagues (2020) point out:

addiction does not reside in the object of interest. Addiction is the relationship between the user and the object or activity, a relationship modulated by the intensity of its use (e.g., dose). If “addictiveness” resided in gambling or psychoactive drugs, then many, if not most, users would evidence addiction. This is certainly true for tobacco, but it is far from true for gambling. (p. 3)

In other words, while certain parallels do exist between tobacco and gambling in how the industry might operate, those differences are considerable. Gambling does not inevitably give rise to harm; it is only harmful for a small minority who gamble to excess (Abbott, 2017). Although certain products are probably more harmful than others (e.g., EGMs), the development of addiction is not as likely as is the case with cigarettes, which have addictive additives and carcinogens that affect all users. Despite this, advocacy researchers will attempt to apply that the two activities are quite similar and, in so doing, justify greater urgency and restrictions on the assumption that harm is an inevitable consequence of consuming gambling products (e.g., David et al., 2019). In fact, the probability of harm for any given individual who engages in gambling is quite small. Moreover, it seems unjustified or an overgeneralization to express concerns about young people growing up to gamble if the activity does not entail an inevitable element of harm (which is the case with tobacco). Gambling can be consumed at healthy levels and this appears to be the case for at least 90% of people who try gambling products across their lifetimes (Abbott, 2017).

(d) Molehills into mountains

Another rhetorical strategy that is particularly common in youth gambling research is to discern problems irrespective of the prevalence of the phenomenon. In adolescent gambling research this can occur more easily because there is a tacit assumption that all gambling in this population is problematic or potentially harmful because young people are not legally able to gamble and they are a “vulnerable population.” Thus, even when figures might be extremely low or a strong declining trend emerges in certain major variables (e.g., participation or problem gambling) (Wardle, 2017), a temptation to focus on any figures which are increasing (e.g., engagement in new activities) may nevertheless operate. We have faced this dilemma ourselves (Delfabbro & Thrupp, 2013; Gainsbury, King, Russell, Delfabbro, & Hing, 2016). At what point does a “significant minority” become large enough to be of interest in research and policy? Another tendency is to make small percentages seem important by multiplying them by the population of young people to show that the problem is large when expressed in absolute numbers. In this way, it is easy for researchers in this area to make the findings sound “concerning” and a justification for policy action. An example is observed in Thomas et al. (2018) when they write: “60–80% of young people engage in formal or information gambling prior to the legal age ... and are vulnerable to harmful and problem gambling .... a survey in the United Kingdom (UK) estimated that around 0.9% of 11- to 15-year-olds were problem gamblers (equating to 31,000 young people)” (p. 2).

(e) Altering the frame

A careful analysis of evidence throughout the world and in countries like Australia and New Zealand shows that gambling participation rates for many activities have declined over the last 10–15 years (Abbott, 2017, 2020; Delfabbro & King, 2020). Despite increases in Internet gambling and sports betting (Gainsbury, 2012), participation rates and real expenditure has declined (Queensland Treasury, 2019). Problem gambling rates have similarly shown some decline or stabilized in many countries (Abbott, 2017) and, in response, the industry is looking for new ways to recover its revenue (e.g., skill-based EGMs, virtual activities) (Delfabbro et al., 2019; Pickering, Philander, & Gainsbury, 2020). In countries such as Australia and New Zealand, the major period of growth was in the 1990s and early 2000s with declines experienced thereafter (Abbott, 2017).

Such declines are not convenient for those who might want to create urgency. Nevertheless, it is possible to convey the view that gambling is increasing in importance by focusing only on certain types of gambling (e.g., Internet or sports gambling is growing), that are often increasing from an exceptionally low baseline level. Another strategy is to cast the frame of reference back to earlier periods. For example, Abbott (2020) recently describes increases in gambling availability, participation and expenditure as “unprecedented” and “rapidly expanding,” despite exceptionally detailed work by the same author which shows the reverse trend over the last 10–20 years (Abbott, 2017) for the majority of land-based forms of gambling. In effect, the growth is only unprecedented when one widens the frame to encompass the 1990s (which is correct). In our view, policy action or research directions would need to be based on trends observed in the 2010s and 2020s. Continual referral back to the modern era becomes more and more tenuous as this interval expands from 30 to 40 years over the next decade. It creates an avenue for maintaining the argument that we are in an era of “growth” or a unique period of history in which concerns about gambling must remain paramount.

(f) Slippery slopes

A slippery slope argument is one where an initial claim or event is seen as precipitating a longer chain of events (usually negative ones) where there is usually no evidence or proof that the concatenation of events will follow. A good example is the belief that same-sex marriage leads to an erosion of traditional family values or that immigration leads to an erosion of the dominant culture. Both statements have little evidence supporting the connection between the first claim and the second. In our view, such arguments are common in gambling research and particularly in research involving young people and gambling. A useful illustrative example is a study by Bestman et al. (2016) in a study that audited the content of webpages of New South Wales clubs. The aim of the study was to examine whether family and child-friendly content was present on the pages. The study showed that one in five clubs mentioned child or family facilities on their front-end pages (6% had children’s dining and 9% encouraged adults to bring their children). They then found that 86% had family-related marketing on their secondary pages. From this they then concluded that “The prominent presence of child and family related promotions on home pages is indicative of the importance of this target group for these venues” (p. 159). At no stage did the authors emphasise the fact that these gambling clubs also had restaurants and other facilities that might encourage visitation for reasons other than gambling. Nor did they establish that children were even looking at the webpages, entering gaming areas or contemplating gambling. Nevertheless, they conclude:

The frequent use of images of children suggests that these are places where children “belong” and are welcome ... the marketing strategies identified here may increase the likelihood that the venues will be normalised among children as positive environments, resulting in higher rates of patronage in adulthood. If this is the case, it also seems likely that children who regularly attend these venues may transition more seamlessly into the range of gambling activities that are offered within that environment. (p. 159)

Although we generally agree with views of the same team about other issues (e.g., the saturation of sports advertising in modern sport) (Thomas et al., 2012, 2016, 2018), we believe that the passage above makes far too strong a statement based on the available evidence about the influence of gaming venues.

(g) Flexible definitions

A misleading technique which we have observed in a number of reviewed papers and reports is the use of what we term “slippery” or “flexible definitions.” The most common is to conflate gambling with problem gambling. Although only higher risk gambling leads to genuine or serious harm, certain authors will position gambling itself as “problem” behaviour. Indeed, in studies of younger people, it is often hard to find anything gambling-related at all that is not harmful. Exposure to gambling advertising or parental gambling is bad. Knowledge of gambling (e.g., brands or betting offers) is bad. Having intentions to gamble when one turns 18 is bad. When young people (even young adults who are now called “emerging adults”) gamble, this is also bad and attempts will then be made to identify “risk” factors.

In our view, this represents an extension of the slippery slope argument described above in that it does not inevitably follow that gambling is bad. Problem, pathological or disordered gambling and the associated harms is the principal concern. Thus, it is important for researchers to differentiate discussions of risk to harmful gambling or problem gambling and to avoid the inference (implied in certain studies) that gambling is the first step towards problem gambling. Unless one is talking about “risk factors” in the technical statistical sense, then it is misleading to talk about the risks for a behaviour that is legal and not harmful for many people who partake of the activity in moderation (alcohol, soft-drinks or fast food being the obvious parallels). One example that where the authors appear to stray into this type of logic is a recent UK study by Melendez-Torres et al. (2019). In their paper, the authors report the findings of the School Health Research Network Survey which generally reports quite low prevalence figures for previous week gambling (e.g., around 5% had played slot machines) and a small percentage had been involved in private or lottery-style gambling (1–3% for any individual type of gambling). The authors conclude: “Given the widespread opportunity to gamble and lack of education regarding its associated risks, adolescents are vulnerable to poor outcomes associated with gambling.” Here, it almost appears that merely the opportunity to gamble could lead to young people experiencing negative outcomes.

Section 2. Confirmation bias and its forms

Confirmation bias is a well-recognised cognitive bias that involves unbalanced emphasis on information consistent with an existing idea or hypothesis. It occurs when people downplay or fail to search for disconfirmatory evidence (bias against disconfirmatory evidence or BADE), over-emphasise or search selectively for confirmatory evidence (Balzan et at., 2012; Woodward et al., 2006). As Chambers (2017) points out, a common form of confirmation bias in research is when authors develop hypotheses after the data are known (the HARK effect). In effect, this involves the development of hypotheses that are consistent with the findings. For example, a particular study might fail to obtain many of the expected effects, but obtain an interesting finding relating to age or gender. A hypothesis relating to these variables is then inserted into the introduction of the paper to make it appear planned or anticipated.

Such practices are likely to present in gambling research as well, although it will be difficult to detect this type of practice because papers can easily be back-engineered to make these findings appear to be original components of the paper. However, we argue that there are other more transparent forms of confirmation bias that are more easily detectable in papers and that can be addressed in the review process. These include the practice of “cherry picking” in introduction and discussion sections and a process which we refer to as “splicing” which, in essence, involves the construction of arguments using various and independent data sources to fill in the gaps in evidence.

(a) Cherry picking

Cherry picking is most common in introduction and discussion sections. When observed in introductions, the pattern is similar to “creating urgency” described above. The researcher builds up an argument by reporting only that evidence which supports the proposition being investigated. Most commonly, the position is that gambling is bad. It is a major public health issue. It causes harm and a particular group is vulnerable. Gambling and problem gambling are at unprecedented levels and are “increasing” or becoming worse. Participation rates are high. Expenditure rates are high. New forms of gambling are emerging. There are new forms of technology emerging which are going to make it much worse. Gambling affects the most vulnerable (in the absence of operationally defining “vulnerable”). Much of this may be true, but there is often no qualification. For example, papers which adopt this style will not cite more nuanced interpretations (e.g., Abbott, 2017 about the recent downward trends in prevalence). A tendency towards finding the highest and most extreme figures my also be present. As an example, Thomas and colleagues (2012) report that “up to 80% of adolescents will have engaged in gambling by the time they are 18, with up to 8% categorised as problem gamblers, and up to 15% at risk of developing problems” (p. 145). Having conducted numerous studies of youth gambling (and during a time when the figures were probably higher than they are now) (e.g., Delfabbro & Thrupp, 2003; Delfabbro & King, 2014; Delfabbro, Winefield, & Anderson, 2009; Delfabbrom, King, & Derevensky, 2016) we believe that these figures are inflated. Actual problem gambling figures are likely to be a fraction of these figures (closer to 1%–3% for problem gambling and at-risk gamblers combined (e.g., see Wardle, 2017 for recent UK results or Volberg et al., 2011). Citing such extreme and unlikely figures (which are likely to be because of problems with sampling or measures) works well in the paper because it fits with a narrative designed to highlight the potential threats associated with sports advertising on young people. Although this is a useful line of research, the narrative framing makes it harder to cite more cautionary work that reveals low rates of sports participation in adolescents and the lack of continuity between adolescent and adult gambling (e.g., Wardle, 2017).

Another form of cherry picking occurs in discussion sections. The most common of these occurs when researchers investigate multiple hypotheses, but devote much of their time to discussing only the significant ones. For example, an experimental study might have four conditions, a pre- and post-comparison and include multiple covariates. All of the main effects might be non-significant, but one finding (e.g., males on Condition X in the post-test condition are different from women). Instead of writing that the hypotheses were “generally not supported,” much of the discussion and the abstract might be devoted to the isolated significant effects. This selective reporting of the findings makes the study sound more impressive than it really is, and can also lead to tenuous findings being advanced as the basis for policy reform.

(b) Splicing

Gambling occurs all over the world. Many different types of gambling exist. Countries also differ in their mix of gambling product, the prevalence of problem gambling, and when the market began to develop. An exceptional amount of gambling research also exists. Pertinent findings have been published in related disciplines (e.g., tobacco, alcohol, fast food). As a result, it is possible to find a study that provides an illustration of a point that is required to support an argument. A good argument is one where there is a common point of reference. For example, if one looked at a single type of gambling, in the same country and analysed the same variables over the same period then a clearer picture emerges as to what has happened over that period (e.g., Abbott, 2017). By contrast, splicing involves taking information from different sources and combining them. Thus, for example, a person wanting to show that video gaming in adolescence leads to problem gambling in adulthood might not be able to find any study that shows this effect. However, there might be a study in Country X using a convenience sample that shows a video game and gambling correlation for adolescents. Another country (Country Y) might show that people who gamble as teens are more likely to gamble as adults. In Country A, a study might show that problem gambling is linked to technology use in adults. Another study in Country B (a prevalence study) shows that technology use is more common in video-game players. By splicing all these findings together it could be possible to argue that video games are potential risk factor for problem gambling, even though no study in a single jurisdiction has shown this.

Splicing can also take several other forms. One form of splicing, already alluded to in the section on slippery slope arguments, is to find “evidential” or “analogy” patches to form connections between a sequence of related claims. For example, if no convincing evidence exists that advertising exposure leads to gambling and then harmful consequences in teenagers (A–B–C), one might then argue that A–B and B–C is likely to be so as a result of findings from smoking research. Another strategy is to patch over and turn non-significant results in one’s favour. For example, a study might not show that a particular effect occurs for problem gambling, but applies equally to all gamblers. However, since problem gamblers are a subset of all gamblers then the effect holds for them as well. Problem gamblers will then be described as vulnerable and highly likely to engage in Activity X and therefore the findings justify the need for a policy response.

Section 3. Overgeneralization

The purpose of many studies is to draw inferences about populations. Strictly speaking, the only time one can usually do this confidently is when the sample is representative of a given population and sampled in a way that minimises bias. In an ideal world, the best sample is a random sample where participants are given an equal probability of being selected and where the characteristics of the sample match the population. In reality, such a sample is rarely achievable (as indicated by the low response rates in many public epidemiological surveys) (e.g., 22% in the recent Victorian prevalence study, Rockloff et al., 2018). Accordingly, what usually happens is that researchers try to make the sample representative by careful sampling (e.g., by demographics) or post-weighting the data to make them appear representative of the population. In our view, this sort of procedure generally works well if the original sampling adopted a probability approach. Good examples include the Health Income Labour Dynamics in Australia (HILDA) Survey (Armstrong & Caroll, 2017), or the Quinte survey in North America (Williams et al., 2015). On the other hand, we raise concerns about studies that attempt to make inferences about the prevalence of phenomena based on panel surveys or convenience samples (particularly where inclusion criteria are modified after commencement of recruitment to increase sample size). Although it may be possible to weight the sample to reflect broader demographic characteristics (age, gender, area), there are still likely to be biases that cannot be controlled. For example, those persons who volunteer for gambling or gaming surveys are more likely to be interested in the topic, have higher levels of co-morbidity and be influenced by how the survey is promoted and its content (Ladouceur et al., 1997). Such bias is not likely to be as strong in broader population surveys where the topic is not usually so transparently about gambling.

In our view, reviewers and policy makers need to be alert to papers that present findings that encourage readers to interpret the results as prevalence figures, but where the study is not a prevalence study. Such findings are indicative and are principally correlational in nature and have to be generalised cautiously. This applies to many studies of gaming (e.g., loot boxes) conducted online (e.g., Zendle et al., 2019, 2020) as well as studies of adolescent gambling in schools (Delfabbro & Thrupp, 2003; Derevensky & Gupta, 1998; Gupta & Derevensky, 1998). Adolescent gambling studies conducted in schools, for example, can often be challenging because consent might not be obtained from all students; students give mischievous answers; and not all selected schools want to take part. Potential biases in the use of panel surveys are identified by Howe and colleagues (2019) show how prevalence rates obtained from panel surveys tend to be higher than population surveys, even after applying demographic weights.

We also believe that care must be taken when interpreting the results of purposive samples and the extent to which they can be related to broader populations. For example, in a study of young people’s exposure to sports advertising, Nyemcsok et al. (2018) studied the brand awareness of young basketballers who played in community stadiums. The study used “a range of convenience and purposive sampling” and “did not aim to be representative (and be generalizable) to all young people in Victoria” (p. 1070). All this is sensible qualification and the sample appears well suited to the nature of the investigation. However, the paper concludes:

This study shows that the current regulatory structures in Australia appear to be ineffective in preventing young people’s recall and awareness of gambling brands and provides further evidence that a range of significant restrictions may be required to prevent young people’s exposure to gambling advertising. (p. 1076)

To our mind, such statements seem quite strong given that (a) the researchers acknowledge using a selective sample of young people who are probably more likely to exposed to sports advertising and (b) have an interest in sports. We note, too that (c) the sample was quite small (n = 111).

Another form of overgeneralization occurs when rare phenomena obtained for small subgroups or sub-questions are presented as being substantial, but where their significance might be less policy-relevant when the base rate of the phenomena is considered. These are not necessarily flaws in the research or reporting, but where certain qualifications might be needed. For example, in a national study of adolescent gambling (n = 2760) in the UK, Wardle (2019) reports that “39% of children who bet on skins had gambling on other activities” and “those who bet on skins had higher rates of at risk and problem gambling (23%)” (p. 1109). Here, the results would be clearer if at least some base rate figures were presented. The abstract presents large figures, but does not tell the reader that only 7% of young people gambled on skins. The 39% therefore refers to 2760 x .07 x .39 = 75/2760 or 2.7% of the sample. A similar logic can, of course, be applied to the second figure.

Sometimes, in complex projects, a risk of misinterpretation can occur if figures are taken in isolation without reference to the larger project. A good example is an important study of sports betting by Russell et al. (2019) who reported that “[o]ver 77% of micro event bettors met criteria for problem gambling (compared to 29% of non-micro event bettors), and only 5% were non-problem gamblers.” (p. 218). This is an important and policy-relevant finding with implications for regulation. However, one needs to read other papers by the same authors (Hing et al., 2018, 2019) to determine to what extent banning micro-bets might have on problem gamblers. As the authors carefully show in other papers, only 20% of all bets placed by problem gamblers were classified as “during the match” and these included micro-bets, exotic bets and others. Thus, the actual percentage of total bets by problem gamblers that fall into the impulsive “on-the-day” category may be quite low. Careful re-reporting and interpretation of these findings (e.g., by policy makers) therefore requires reading of the larger body of work to place the finding in context.

Conclusion: Improving research practices

The principal focus of this paper has not been on academic malpractice or misleading practices in research methods and analysis (the principal focus of Chambers’s important 2017 book). Instead, we have endeavoured to identify sources of bias in academic reporting and how findings from gambling studies might be misrepresented. We argue that certain of these are unintentional, but others occur intentionally because of (a) the increasing imperative to promote strong harm-reduction agendas via so-called “advocacy research,” (b) the growing politicisation of gambling research, and (c) the significant role played by government bodies in funding research. We also acknowledge that this style of writing might also be a “carry-over” from other contexts (e.g., grant, tender and promotion applications) in which researchers have to “sell” their ideas to receive advancement.

Having acknowledged these points, we believe that gambling research will be of a higher quality and more balanced if certain of the issues identified in this paper are avoided wherever possible. These include (a) overuse of emotive language, (b) unbalanced and over-extended arguments, (c) selective reporting, and (d) overgeneralization or interpretation of findings without reference to base rates or the nature of the sampling. Many of these issues can often be remedied and addressed before submission and during the review process. Authors should also, wherever possible, avoid allowing their findings to be misrepresented in other forms of communication such as press releases, media stories and in social media. However, there are many pressures and directives in the world that may serve to run counter to this vision of academic research. University research and its benefits are being increasingly marketed and commodified. There are three-minute thesis competitions where students have to sell their research to an audience. Grants and tenders are often seen as more competitive if they can sell or promote the ideas, even if the work may be in the preliminary stages. Academics are praised for popularizing their work and making it more relevant to the community to show that universities have “impact” and the ability to translate their findings to the community. However, if this process occurs at the expense of objectivity and findings are allowed to be misrepresented, then it weakens accordingly the status of academic research. In the field of gambling studies, the process may also serve to make “advocacy work” less effective if academics are seen to engage in the same practices often criticized in industry. These practices include the selective understatement of the harms from gambling; the impacts on vulnerable groups; the greater risks of certain products; and the extent to which various harm minimisation strategies are effective in addressing problem gambling.

In conclusion, as authors in the field of gambling studies, we should acknowledge that these are all principles that apply to our own work as well. There will be published works where we can look back and reflect on whether we fully considered all potentially competing perspectives or possibly overstated or understated particular points. We also acknowledge that the work we do will be shaped by our own disciplinary perspectives and so our areas of focus, the variables we study, and how we interpret gambling phenomena may not coincide with the views those scholars working in other disciplines advance.

References

Abbott, M. W. (2020). The changing epidemiology of gambling disorder and gambling-related harm: public health implications. Public Health, 184. Advance online publication. https://www.researchgate.net/publication/341045964_The_changing_epidemiology_of_gambling_disorder_and_gambling-related_harm_Public_health_implications

Abbott, M. W. (2017). Gambling and gambling harm in New Zealand: A 28-year case study. International Journal of Mental Health and Addiction, 15, 1221–1241. https://doi.org/10.1007/s11469-017-9767-6

Adams, P. J., Butzow, S., & Rossen, F. (2010). Vested interests in addiction research and poisonous partnerships: Health sector buy-into arrangements with government and addictive consumption industries. Addiction, 105, 585–590. https://doi.org/10.1111/j.1360-0443.2009.02815.x

Andréasson, S., & McCambridge, J. (2016). Alcohol researchers should not accept funding from the alcohol industry: Perspectives from brief interventions research. Journal of Studies on Alcohol and Drugs, 77, 537–540. https://doi.org/10.15288/jsad.2016.77.537

Armstrong, A., & Caroll, M. (2017). Gambling activity in Australia: Findings from wave 15 of the Household, Income and Labour Dynamics in Australia (HILDA) Survey. Melbourne, Australia: Australian Institute for Family Studies. https://nla.gov.au/nla.obj-571994111/view

Balzan, R., Delfabbro, P., Galletly, C., & Woodward, T. (2012). Reasoning heuristics across the psychosis continuum: The contribution of hypersalient evidence-hypothesis matches. Cognitive Neuropsychiatry, 17, 431–450. https://doi.org/10.1080/13546805.2012.663901

Bestman, A., Thomas, S., Randle, M., Pitt, H., Daube, M., & Pettigrew, S. (2016). Shaping pathways to gambling consumption? An analysis of the promotion of gambling and non-gambling activities from gambling venues. Addiction Research and Theory, 24, 152–162. https://doi.org/10.3109/16066359.2015.1093121

Cassidy, R. (2014). Fair game? Producing and publishing gambling research. International Gambling Studies, 14, 345–353. https://doi.org/10.1080/14459795.2014.971420

Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. New York, NY: Princeton University Press.

Cialdini, R. B. (2007). Influence: The psychology of persuasion. New York, NY: HarperCollins.

David, J. L., Thomas, S. L., Randle, R., & Daube, M. (2020). A public health advocacy approach for preventing and reducing gambling related harm. Health Promotion, 44, 14–19. https://doi.org/10.1111/1753-6405.12949

David, J. L., Thomas, S.L., Randle, R., Daube, M., & Balandin, S. (2019). The role of public health advocacy in preventing and reducing gambling related harm: Challenges, facilitators, and opportunities for change. Addiction Research and Theory, 27, 210–219. https://doi.org/10.1080/16066359.2018.1490410

Delfabbro, P. H., & King, D., (2020, in press). Demographic insights into the decline in electronic gaming machine (EGM) participation in South Australia. International Journal of Mental Health and Addiction. https://doi.org/10.1007/s11469-020-00326-9

Delfabbro, P. H., King, D., & Derevensky, J. (2016). Adolescent gambling and problem gambling: prevalence, current issues and concerns. Current Addiction Reports, 3, 268–274. https://doi.org/10.1007/s40429-016-0105-z

Delfabbro, P. H., King, D., & Gainsbury, S. M. (2019). Understanding gambling and gaming skill and its implications for the convergence of gaming with electronic gaming machines. International Gambling Studies, 20, 171–183. https://doi.org/10.1080/14459795.2019.1662824

Delfabbro, P. H., King, D., & Griffiths, M.D. (2014). From adolescent to adult gambling: An analysis of longitudinal gambling patterns in South Australia. Journal of Gambling Studies, 30, 547–563. https://doi.org/10.1007/s10899-013-9384-7

Delfabbro, P. H. & Thrupp, L. (2003). The social determinants of youth gambling in South Australian adolescents. Journal of Adolescence, 26, 313–330. https://doi.org/10.1016/S0140-1971(03)00013-7

Delfabbro, P. H., Winefield, A. H., & Anderson, S. (2009). Once a gambleralways a gambler? A longitudinal analysis of gambling patterns in young people making the transition from adolescence to adulthood. International Gambling Studies, 9, 151–164. https://doi.org/10.1080/14459790902755001

Derevensky, J. L., & Gupta, R. (1998). Adolescent gambling behavior: A prevalence study and examination of the correlates associated with problem gambling. Journal of Gambling Studies, 14, 319–345. https://doi.org/10.1023/a:1023068925328

Dowling, N. A., Merkouris, S. S., Greenwood, C. J., Oldenhof, E., Toumbourou, J. W., & Youssef, G. J. (2017). Early risk and protective factors for problem gambling: A systematic review and meta-analysis of longitudinal studies. Clinical Psychology Review, 51, 109–124. https://doi.org/10.1016/j.cpr.2016.10.008

Gainsbury, S. (2012). Internet gambling: Current research findings and implications. London, UK: Springer.

Hancock, L. & Smith, G. (2017). Critiquing the Reno Model I-IV international influence on regulators and governments (2004–2015): The distorted reality of “responsible gambling.” International Journal of Mental Health and Addiction, 15, 1151–1176. https://doi.org/10.1007/s11469-017-9746-y

Hing, N., Russell, A.M.T., Li, E., & Vitartas, P. (2018). Does the uptake of wagering inducements predict impulse betting on sport? Journal of Behavioral Addictions, 7, 146–157.

Hing, N., Russell, A.M.T., Thomas, A., & Jenkinson, R. (2019). Wagering advertisements and inducements: Exposure and perceived influence on betting behavior. Journal of Gambling Studies, 35, 793–81.

Howe, D. L., Vargas-Saenz, A., Hilbert, C. A., & Boldero, J. M. (2019). Predictors of gambling and problem gambling in Victoria. PLoS One, 14, e0209277. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209277

King, D. L., Russell, A., & Hing, N. (2020). Adolescent land-based and online gambling: Australian and international prevalence rates and measurement issues. Current Addictions Reports, 7, 137–148. https://doi.org/10.1007/s40429-020-00311-1

King, D. L., Russell, A., Gainsbury, S., Delfabbro, P. H. & Hing, N. (2016). The cost of virtual wins: An examination of gambling-related risks in youth who spend money on social casino games. Journal of Behavioral Addictions, 5, 401–409. https://doi.org/10.1556/2006.5.2016.067

Korn, D., & Shaffer, H. (1999). Gambling and the health of the public: Adopting a public health perspective. Journal of Gambling Studies, 15, 289–365. https://doi.org/10.1023/a:1023005115932

Ladouceur, R., Arsenault, C., Dube, D., Freeton, M., & Jacques, C. (1997). Psychological characteristics of volunteers in studies on gambling. Journal of Gambling Studies, 13, 69–84.

Livingstone, C., & Adams, P. J. (2016). Clear principles are needed for integrity in gambling research. Addiction, 111, 5–10. https://doi.org/10.1111/add.12913

Masuzzo, P., & Martens, L. (2017). Do you speak open science? Resources and tips to learn the language. PeerJ Preprints, 5, e2689v1. https://doi.org/10.7287/peerj.preprints.2689v1

Melendez-Torres, G. J., Anthony, R. E., Hewitt, G., Murphy, S., & Moore, G. F. (2019). Prevalence of gambling behaviours and their associations with socioemotional harm among 1116 year olds in Wales: Findings from the School Health Research Network survey. European Journal of Public Health, 30, 451–457. https://doi.org/10.1093/eurpub/ckz176

Nyemcsok, C., Thomas, S. L., Bestman, A., Pitt, H., Daube, M., & Cassidy, R. (2018). Young peoples recall and perceptions of gambling advertising and intentions to gamble on sport. Journal of Behavioral Addictions, 7, 1068–1078. https://doi.org/10.1556/2006.7.2018.128

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, 943–952. https://doi.org/10.1126/science.aac4716

Pickering, D., Philander, K., & Gainsbury, S. (2020). Skill-based electronic gaming machines: a review of product structures, risks of harm, and policy issues. Current Addiction Reports, 7, 229–236. https://doi.org/10.1007/s40429-020-00309-9

Queensland Treasury. Australian Gambling Statistics. Brisbane: Queensland Government. https://www.qgso.qld.gov.au/statistics/theme/society/gambling/australian-gambling-statistics

Rockloff, M., Browne, M., Hing, N., Thorne, H., Russell, A., Greer, N., Tran, K., Brook, K., & Sproston, K. (2019). Victorian population gambling and health survey (2018–19). Melbourne, Australia: Victorian Government.

Russell, A.M.T., Hing, N., Browne, M., Le, E., Vitartas, P. (2019). Who bets on micro events (microbets) in sports. Journal of Gambling Studies, 35, 205–233.

Shaffer, P.M., Ladouceur, R., Williams, P.M., Wiley, R.C., Blaszczynski, A., & Shaffer, H.J. (2019). Gambling research and funding biases. Journal of Gambling Studies, 35, 875–886.

Shaffer, H., Blaszczynski, A., & Ladouceur, R. (2020). Gambling control and public health: Lets be honest. International Journal of Mental Health and Addiction, 18, 819–824. https://doi.org/10.1007/s11469-020-00240-0

Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., Bott, L., Adams, R., Venetis, C. A., Whelan, L., Hughes, B., & Chambers, C. D. (2016). Exaggerations and caveats in press releases and health-related science news. PLoS One, 11, e0168217. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0168217

Thomas, S. L., Bestman, A., Pitt, H., Cassidy, R., McCarthy, S., Nyemcsok, C., Cowlishaw, S., & Daube, M. (2018). Young peoples awareness of the timing and placement of gambling advertising on traditional and social media platforms: A study of 1116 year olds in Australia. Harm Reduction Journal, 15, 1–13. https://harmreductionjournal.biomedcentral.com/articles/10.1186/s12954-018-0254-6

Thomas, S. L., David, J., Randle, M., Daube, M. & Senior, K. (2016). Gambling advocacy: Lessons from tobacco, alcohol and junk food. Australian and New Zealand Journal of Public Health, 40(3), 211–217. https://doi.org/10.1111/1753-6405.12410

Thomas, S. L., Lewis, S., Duong, J., & McLeod, C. (2012). Sports betting marketing during sporting events: Aa stadium and broadcast census of Australian Football League matches. Australian and New Zealand Journal of Public Health, 36, 145–152. https://doi.org/10.1111/j.1753-6405.2012.00856.x

Van Schalkwyk, M., Cassidy, R., McKee, M., & Pettigrew, M. (2020). Gambling control: In support of a public health approach to gambling. The Lancet, 393, 1680–1681. https://doi.org/10.1016/S0140-6736(19)30704-4.

Volberg, R. A., Gupta, R., Griffiths, M. D., Olason, D.T. & Delfabbro, P. (2011). An international perspective on youth gambling prevalence studies. In J. L. Derevensky, D. T. L. Shek, & J. Merrick (Eds.), Youth gambling: The hidden addiction (pp. 21–56). Berlin, Germany: De Gruyter.

Wardle, H. (2017). Trends in children's gambling 2011–2017. London, UK: London School of Hygiene and Tropical Medicine.

Wardle, H. (2019). The same or different? Convergence of skins gambling and other gambling among children. Journal of Gambling Studies, 35, 1109–1125. https://link.springer.com/article/10.1007/s10899-019-09840-5

Williams, R. J., Hann, R. G., Schopflocher, D., West, B., McLaughlin, P., White, N., King, K., & Flexhaug, T. (2015). Quinte longitudinal study of gambling and problem gambling. Guelph, ON: Ontario Problem Gambling Research Centre. https://opus.uleth.ca/bitstream/handle/10133/3641/QLS-OPGRC-2015.pdf

Woodward, T. S., Moritz, S., Cuttler, C., & Whitman, J. (2006). The contribution of a cognitive bias against disconfirmatory evidence (BADE) to delusions in schizophrenia. Journal of Clinical and Experimental Neuropsychology, 28, 605–617. https://doi.org/10.1080/13803390590949511.

Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012). Misrepresentation of randomized control trials in press releases and news coverage: A cohort study. PLoS Medicine, 9, e1001308. https://doi.org/10.1371/journal.pmed.1001308

Young, M., & Markham, F. (2017). Coercive commodities and the political economy of involuntary consumption: The case of the gambling industries. Environment and Planning A, 49, 2762–2779. https://doi.org/10.1177/0308518X17734546

Zendle, D., Cairns, P., Barnett, H., & McCall, C. (2020). Paying for loot boxes is linked to problem gambling, regardless of specific features such as cash-out or pay-to-win. Computers in Human Behavior, 102, 181–191. http://dx.doi.org/10.1016/j.chb.2019.07.003

Zendle, D., Meyer, R., Waters, S., & Cairns, P. (2019). The prevalence of loot boxes in mobile and desktop games. Addiction. https://doi.org/10.1111/add.14973

*******




1Research agendas established by major government funding bodies make this explicit in their mission statements. Projects are designed to address policy relevant areas within the context of “reducing” or “minimizing harm.” Researchers (including the authors) have had to write policy briefs to demonstrate how the findings have “impact” on policy and practice.

2Speculative claims that industry funded research is ipso facto biased in the absence of supportive evidence remains common (Adams et al., 2010; Andréasson & McCambridge, 2016; Livingstone & Adams 2016) despite certain studies failing to find differences in funding source and study characteristics (Shaffer et al., 2019).

Submitted June 9, 2020; accepted August 28, 2020. This article was peer reviewed. All URLs were available at the time of submission.

For correspondence: Paul Delfabbro, Ph.D., c/o School of Psychology, University of Adelaide, 5005, South Australia. E-mail: Paul.delfabbro@adelaide.edu.au

Competing interests: The authors have conducted research funded by government agencies, provided consultancy advice for government agencies and industry groups /peak bodies relating to harm minimisation or responsible gambling; and have been involved in regulatory matters or court cases.

Ethics approval: Not required.

Acknowledgements: No funding was associated with the preparation of this paper.

Article Categories:
  • opinion

Related Article(s):


Copyright © 2021 | Centre for Addiction and Mental Health
Editor-in-chief: Nigel E. Turner, Ph.D.
Managing Editor: Vivien Rekkas, Ph.D. (contact)