The Methodology and Procedures of the National Media Archive

Lydia A. Miljan (The National Media Archive)

The National Media Archive

The National Media Archive is pleased to have this forum to discuss its methodology and respond to some criticisms of its work. This paper will begin with an overview of the structure of the National Media Archive, its research programme and its holdings. Following that discussion, a description of National Media Archive methodology will be furnished. It is hoped that by providing this description, misunderstandings of our analysis will be avoided. Third, assumptions about news will be provided. The remainder of the paper will address criticisms made by Robert Hackett, Bill Gilsdorf and Philip Savage which appear elsewhere in this volume.

The National Media Archive was established at the Fraser Institute on December 15, 1987, to a) acquire, preserve, and make available a complete resource of news coverage in Canada, so the issues and their representation by the media can be assessed from a number of points of view; and b) provide and encourage other researchers to research the public information function performed by the national media. The Archive was developed because nowhere else in Canada had television news been easily accessible. The National Archives of Canada holds videotapes of CBC and CTV programming, however, those holdings are not indexed. Only the date and name of the program is catalogued. No attempt has been made to index the content of those videotapes. While the networks have some holdings of their own programs, it is difficult for researchers to get access due to time constraints on network personnel.

The National Media Archive's current collection holds CBC's "The National," "The Journal," "Venture" and "Sunday Report" as well as "CTV News." Each of these newscasts is transcribed and converted to a software package that allows for free word text searches. This has eliminated the need for manual indexing and allows for the retrieval of words or combinations of words. The services of The National Media Archive are available to all. The staff of the Archive is dedicated to providing helpful, thorough, prompt and impartial access to the collection. The Archive provides customized computer searching of any topic, name or issue presented in the national news. Verbatim transcripts are made available to the general public for research or study purposes only. The National Media Archive operates as a division of the Fraser Institute. An Advisory Board provides guidance on policy and opportunities for the future. The five-member board is made up of respected members of the academic and journalistic communities.

Methodology Employed by The National Media Archive

The content analysis conducted by the National Media Archive is conducted on television news of major public policy issues. The sources used are primarily CBC or CTV, however, depending on the availability of news transcripts, some analyses have also examined The Globe and Mail. In addition, issues such as the 1988 Federal Election campaign examined CBC radio as well as The Globe and Mail. The analysis conducted by the National Media Archive on the Nicaraguan Election campaign, also examined Maclean's Magazine in order to compare the American television, newspaper and magazine coverage of the campaign. Analyses are usually based on news coverage over the course of a full year. The major exception to this procedure is in the case of elections. In this case, only the campaign period is analyzed. At times, content analyses have also been conducted on current such as the February 1989 analysis of corporate mergers (two weeks). The Archive examined coverage of the First Minister's meetings held on Meech Lake during the first week of June 1990. Because the transcripts can be accessed by a word or combination of word searches, the analysis covers all references in all related stories. The unit of analysis is generally the statement. This is defined as the spoken sentence or a complete grammatical assertion separated by punctuating pauses in speech. However, in some cases the unit of analysis has been reduced to the single word.

Criteria of Objectivity

Within the standards of social science methodology, we establish objectivity by a number of procedures. First, researchers are hired on the basis of their differing political and educational backgrounds. This facilitates different perspectives when designing the research instrument and testing the coding design. This ensures that the content is analyzed and not the researcher's interpretation of what or how they view the statement in question. Each researcher is assigned a topic and prepares a preliminary code book. The researchers are encourage to access public, private and university libraries, and to conduct interviews with interested parties, e.g. academics, special interest groups and professionals. In addition, newspaper articles and television stories are also examined. Rules are designed by a consensus of the researchers and explain the issue and specify where statements are to be coded. All researchers examine the coding design and rules prior to the commencement of coding. Suggestions are made and debated. The code book is tested by having the group code stories and compare results. This familiarizes researchers with the code book and refines any problems with the design.

Coding involves reading the statement, and/or watching the accompanying video and classifying the statement itself. Blind reliability tests are conducted at different stages of coding. During the reliability meetings, the group compares how each statement was coded. Any disagreements are discussed until a consensus is reached. Once the entire sample is coded, coders of opposing viewpoints switch stories and check codes for all variables. Any disagreements are discussed and changes made when agreement between coders is reached.

Variables Analyzed

All stories in the sample are numbered and that number is placed on a spreadsheet. In addition, since the statement is the unit of analysis, each statement is considered a case containing a number of variables. Many of these are self explanatory such as the date, placement of story in the newscast and the program. Other variables are more complex and will be elucidated here. Among them are:

  1. Origin of Statement: refers to who made the statement. If a reporter made the statement it is identified as reporter. If a statement is shortened or a quotation is made, it is coded as a quote. If the statement is attributed to a source, it is identified as paraphrased. Interview statements refer to statements which are preceded with a question and the full answer is provided.
  2. How the story was presented: examines whether the information is presented as hard news, i.e., events of the day, or as background, defined as providing in-depth analysis over time. Usually coded when the story is not a current news event, for example, stories on the 50th anniversary of the bombing of London.
  3. Dateline: This variable refers to where story was filed. If it is an anchor only story, the filing location is studio.
  4. Geographic focus of the statement recognizes that sources can be talking from one location but speaking about another place, the topic of their discussions is identified.
  5. Source: Names of prominent speakers are listed and numbered.
  6. Affiliation of Source: The association the media provide for the source. The media often place a caption on the video or describe the significance of the speaker in the text. The media description is always used.
  7. Coder and Recoder: The researchers are also assigned numbers to ensure accountability. If anomalies exist in the data, the coder can be questioned directly about coding decisions made.

The remaining variables depend upon the issue analyzed. For example, in the study on abortion, the analysis examined the amount of attention devoted to the various aspects of the abortion issue, including the arguments of pro-choice and pro-life groups. Among them are: access to abortion, moral issues and time limits to abortion. Also included in the analysis was the governmental time spent discussing the law, as well as procedural issues such as demonstrations and meetings. In contrast, the analysis of the environment looked at the types of environmental issues overall such as oil spills, PCBs, pollution, nuclear disasters, nuclear energy, the changing atmosphere, dams, pesticides, extinction, and natural environmental occurrences. In addition, the amount of coverage devoted to general issues such as the extent of a crisis, effects on the environment, strategies of major players, solutions to the problems, effects of solutions and causes were also examined. Further information on coding categories and procedures are available upon request.

Assumptions About News

The content analysis of the news is based on the assumption that the western media consistently claim to provide objective information, with a strong emphasis on fairness and balance. In 1919 U.S. Supreme Court Justice Oliver Wendell Holmes, following Mills, proposed that only "the free marketplace of ideas" guaranteed free flow of information. For journalists this proposition provides the foundation for objective, accurate and truthful reporting. Indeed, John Merrill defines journalism as a having "deep loyalty to the truth" (1975, p.8). Canadian notions of objectivity have been largely influenced by these American precedents. According to Peter Desbarats, the Canadian Press (CP) "promulgated a dispassionate and objective style of reporting among English journalists" (1990, p.19). Recently, the Canadian Broadcasting Corporation, in its Journalistic Policy Handbook recognized three principles to guarantee a "balanced and comprehensive service." They are accuracy, integrity and fairness. "Accuracy" is defined as "the information conforms with reality and is not in any way misleading or false. This demands not only careful and thorough research but as disciplined use of language and production techniques, including visuals." "Integrity" guarantees that "the information is truthful, not distorted to justify a conclusion. Broadcasters do not take advantage of their position of control to in any way present a personal bias." "Fairness" refers to the fact that "the information reports or reflects equitably the relevant facts and significant points of view; it deals fairly and ethically with persons, institutions, issues and events" (1988, p.6).

In addition to news organizations making claims of "objectivity," the public itself has come to rely on news, especially television news, to provide them with objective information.

One indication of this is an Environics poll conducted for Canadian Legislatures which found that Canadians consider all news media to be objective and fair in their coverage of issues. More specifically, it revealed that Canadians generally prefer television over newspapers or radio for information on public affairs. Television was also named as the medium most likely to give in-depth analysis (Marketing, 1988, p.5). On Balance takes these expectations into account and therefore examines only the manifest content because audiences only have knowledge of the end product. Thus, as John Soloski points out, "The stress journalists place on objectivity means that news stories are presented to news consumers indexically, that is, divorced from the context of their production. News consumers do not know how a reporter arrived at her/his information nor do they know how procedures used by reporters to gather and report the news determine what news is reported" (1989, p. 870).

Lies, Damn Lies and Scholarly Critique

The critique by Hackett, Gilsdorf and Savage of On Balance, which appears elsewhere in this volume is a multi-level attack on the National Media Archive. Targeted is the funding of the organization, the content analysis method, and application of the method. The remainder of this note suggest that their critique is not scholarly because Hackett, Gilsdorf and Savage misquote, oversimplify and use logical fallacies to misrepresent the Archive's work.

While we have severe misgivings about the integrity of the critique, we also find it questionable that only the National Media Archive was subject to scrutiny. For example, MediaWatch does content analysis and is a self-described advocacy group and its research appears in this volume, yet their funding, methods and reporting practices are not challenged. The questions which ensues is, why are these particular individuals so interested in the work of the Archive? While our critics admit they "support the principles of public broadcasting" (Hackett, 1992, p. ) this cannot account for their vehemence in attacking the National Media Archive. Interestingly, each of our critics has a relationship with the CBC which benefits them in different ways. Bill Gilsdorf's research is dependent on CBC goodwill to allow him to observe journalistic practices (Hackett, 1990). Robert Hackett has been in discussion with the corporation for the past four years to make arrangements to set up a permanent archive (similar to the National Media Archive) at Simon Fraser University. Philip Savage is an employee of the CBC.

From the above it is evident that there are a number of self-serving interests at work here, the least of which is the long-standing adversary relationship between the critical and empirical schools of communication (Blumer, 1985, p.185). In other words, we are not being judged by other empiricists whose outlook and approach are similar, but may differ on specifics; we are being judged by scholars with completely methodological and epistemological approaches to communications. In short, Hackett, Gilsdorf and Savage use the assumptions of the critical school to critique research grounded in the empirical tradition. While the potential exists for a rigorous exchange of ideas and approaches, this potential is never actualized because the critique takes such extreme measures to vilify the work of the National Media Archive. For example, the syllogism is provided that since the National Media Archive receives media attention it should be subject to critical scrutiny. Several questions come to mind from this assertion: is one to assume that the authors are not pre-disposed to media attention? does the popularization of a study diminish its credibility? is raising awareness of journalistic practices wrong? Why the suspicious reaction to media coverage by communication scholars? Perhaps the answer can be found in a 1986 letter to the editor penned by then Associate Professor of Communications Studies at Simon Fraser University, Robert Hackett, complaining about the printing of a study which analyzed CBC radio: "My chief concern, though, is with The Globe and Mail's judgment in prominently reporting this study, while ignoring other research equally publicly available, which reaches different conclusions. As just one example, my own analysis of national TV news suggested that there were few statistically significant differences between CBC and CTV, that in many respects labour was portrayed more negatively than business, and that if any region was under-represented it was rural Ontario, not the West" (Hackett, 1986, p. A7). Aside from the fact that a radio study is refuted with television data, this example illustrates that at least one of the authors is not contrary to media attention, he is opposed to media attention given to research which comes to different conclusions than his own.

Apart from these questions about the objectivity of the critique, we are sceptical of the scholarly criterion used to attack the Archive. There are an extremely high number of factual errors, unsubstantiated claims and selective examples used throughout the paper. This rebuttal will examine the more glaring mistakes and point to the many logical fallacies employed to disgrace the work of the Archive.

Critique of Association

Hackett, Gilsdorf and Savage begin the critique with a questionable classification. The basic premise of the article hinges on categorizing The Fraser Institute as political advocacy group. This thread is woven from the title to the concluding statements. Nothing, however, could be further from the truth. The Fraser Institute is a registered economic and educational research institute. Research conducted for the Institute by over 300 authors have been published in 105 books. Over 200 of Institute's title's have been used for course adoption in universities and colleges world wide. The Institute has on its board of advisors three nobel laureates and is widely regarded for its research in economics and social policy. The classification that the organization is a political advocacy group is not only false is it ignores the fact that if it were a political organization, the Institute would not have research institute status.

Following the mis-classification is the attempt to use the logical fallacy of guilt by association. At one point the critics charge: "...the reader of On Balance is sometimes confronted with gaps between evidence and conclusions, gaps which are seemingly bridged by interpretations which more often than not are consistent with the ideological stance of the Fraser Institute" (Hackett, 1992, p. ). Later, they assert: "It is funded by an advocacy group whose very purpose is to promote laissez-faire economics, and by presumably like-minded (and affluent) subscribers to the $95.00 per year newsletter" (Hackett, 1992, p. ). (This statement is odd, considering that CBC holds 3 subscriptions. Surely, the CBC is a not like-minded subscriber to On Balance). Aside from the fact that the Fraser Institute does not fund the Archive and that the Fraser Institute is not an advocacy group, the critics make the claim that because the Institute examines how markets best work to solve public policy issues, the results of the Archive will be tainted by its association with the Institute. The Archive is a separate division of the Fraser Institute which autonomously hires its own staff on the basis of differing political and educational backgrounds. As a consequence, in the three and a half years of operation, there have been 28 researchers ranging from self-described feminist-marxists to libertarians working for the Archive. Those researchers were not uncritical of the procedures or the analysis of the research. In fact, the scrutiny of the researchers in designing and implementing the research has led to significant changes in the type and direction of research undertaken at the Archive.

Critiques of Application

Many of the criticisms of the National Media Archive research are in fact criticisms of content analysis itself. For example, the authors suggest that since content analysis is not foolproof, National Media Archive researchers cannot claim objectivity. To date there has been no social science method that is absolutely flawless. Survey research, audience testing and content analysis, methods which many communications scholars employ, are all to varying degrees flawed given the nature of the subject matter and the constraints on researchers. However, when these studies are reported in reputable journals, there are rarely any caveats concerning the value of the method used or indeed any scepticism of the objectivity of the findings. Hackett, Gilsdorf and Savage allege that The National Media Archive research does not use a standard research tool, since none is provided in the methodology section of our reports. As was stated at great length in the preceding discussion, the National Media Archive establishes, before analysis, a clear set of coding categories and rules. At least one of the authors, Robert Hackett, was aware of this fact since he supervised two National Media Archive researchers in directed studies courses which were based on the content analysis of the National Media Archive. One of those students received an "A" in the course.

Furthermore, the Archive has always made its coding criterion available to researchers who have requested them. In the press package accompanying the first issue of On Balance the coding sheets were reproduced and distributed to all persons requesting information on the study. When the National Media Archive researchers asked the same of Professor Hackett in his recently published article in the Canadian Journal of Political Science, he stated that he did not "believe" in codes. The criticism that On Balance does not reproduce its coding sheet is a misleading one in that rarely, if ever, do scholarly journals reproduce coding sheets. This argument appears to be a case of special pleading whereby the authors want to discredit the work of the Archive by requiring the Archive to provide more information that academic journals do not demand of their authors.

The critique also makes several erroneous assumptions about the Archive's methodology. Many of these errors could have been avoided by simply requesting documentation. For example, in the discussion of On Balance's selection of topics the critique charges, "the selection of categories often appears to be arbitrary and not informed by previous research" (Hackett, 1992, p. ). The 1988 election study is cited as an example and the following is described: "the On Balance issues on the federal election (November 1988, January 1989) have the coverage broken down into various topics. Leadership is not one of those listed, although other content analysis-based studies of the last four election have identified leadership as consistently one of the most frequently mentioned topics" (Hackett, 1992, p. ). There are three glaring problems with this analysis. First, the critique misrepresents the findings of the Archive by omitting that the list of topics was the top ten issues. Second, leadership, although analyzed, was not mentioned frequently enough to be included in the top ten list. If any of our three critics had bothered to examine the code-sheet for the election they would have seen that leadership was in fact a topic which we examined. The third problem is more disturbing since this criticism complains that On Balance is not informed by past research. As many readers will recall, Fletcher did find leadership to be dominant themes in the 1979 and 1980 election campaigns (Fletcher, 1980). While they cite Soderlund, Romanow, Briggs and Wagenberg as evidence to support that the last two elections were dominated by leadership they neglect to mention that Soderlund and his associates examined only mentions of issues in stories and did not differentiate between major and secondary issues. Albeit Soderlund and his associates found that 26 percent of the stories mentioned leadership, it is not the same as saying that the coverage was dominated by discussion about leadership. When the main topic is analyzed, as was done by Frizzell and Westell, a very different conclusion is reached: "Studies of previous elections have suggested that voters are influenced more by the leadership qualities of the candidates than by any other factor when deciding which party to support. It may have...but what is clear is that the press had little to say about leadership as a specific topic" (1985, p.65). Further, and even more compelling, is the fact that both Frizzell, Pammett and Westell and ourselves found that leadership was not a main issue in the 1988 federal election campaign (Miljan, 1989a, p.5). Therefore, it appears that the critique is not informed by past research.

Another false assumption made in the critique is the assessment that our analysis uses overlapping categories. The critique states that the findings on the environment are ambiguous since a person could be both an environmentalist and a scientist. There is an additional reproach that no definition for scientist or environmentalist was provided in the analysis. Again in this instance none of the authors requested our coding rules. Nonetheless, our studies have always made it clear that we examine the manifest content. Consequently our categorization of sources was dictated by the media themselves. It is true that a person can be both a scientist and environmentalist, however, the networks defined individuals either as scientists or as environmentalists and it was those descriptions that formed the basis of the findings. In this case, our critics should question why the networks did not provide more details for their audiences. It is ironic that the while Hackett, Gilsdorf and Savage allege that The National Media Archive is guilty of methodological errors, they make methodological errors to prove their case. In one instance, our critics confuse effect analysis with content analysis. They argue: "Consider, for example, whether the following statement by a CBC reporter should be regarded as `favourable' or `unfavourable' towards free trade, by American and Canadian TV viewers respectively: `The U.S. Congress is being told that the agreement is a sell-out.' If the agreement was an American defeat in a situation of bilateral negotiations, then surely it was conversely a Canadian victory" (Hackett, 1992, p. ). The National Media Archive's free trade analysis did not ask whether the statements described were examples of Canadian or American victories, to do so would require audience testing. The National Media Archive examined whether statements were favourable or critical of the free trade deal. Whether or not Canadians could imply that the statement was a Canadian victory was not addressed. What was addressed was what was stated, that is, "that the agreement was a sell-out." Clearly this is a statement critical toward the free trade agreement. Similarly, in the discussion on the analysis of labour, our critics confuse content analysis with effects analysis: "On Balance described a picketer yelling on national television, `Hey you slimy scabs, wait till you're out on the street, you little pigs' as an opportunity for labour representatives `to communicate about other players'; yet the woman's anger, language, and implicit threat especially in the eyes of viewers not predisposed to support union militancy, probably do more to hurt labour's image than to enhance it" (Hackett, 1992, p. ). The statement provided in the example was of a woman being critical of replacement workers. The researchers did not attempt to look at the statement from a person not disposed to trade union "militancy," or indeed from the position of a person disposed to trade union "militancy." The researchers coded what was said. And what was said was clearly critical of replacement workers.

The questioning of the way statements were coded illustrates a contradictory quip made in the second footnote which reads: "On Balance has unfortunately been reluctant to reveal to its readers such examples of its coding decisions" (Hackett, 1992, p. ). If On Balance has not given examples of coding decisions, then how could the critique dispute so many published in the pages of On Balance?

Not only do our detractors confuse effects analysis with content analysis; they also invent issues which they pretend On Balance was meant to address and then they criticize On Balance for not addressing those issues! For example, the following premise is asserted: "...if we are interested in analyzing the ways in which news is oriented towards power and discourse, the factual/opinion dichotomy ignores the ideological selectivity, categorization and labelling within the realm of factual itself" (Hackett, 1992, p. ). This is a straw man argument because the National Media Archive never has stated it is interested in power and discourse.

The discussion of On Balance's use of the favourable/unfavourable dichotomy is also galling. The critique oversimplifies the Archives approach by ignoring the fact that our news analysis involves participants, their actions, strategies, and policies or the issue in question. The National Media Archive has also extended the favourable/unfavourable dichotomy to probe participant's attitudes toward a policy. In the case of the environment study a 4-point scale was developed to indicated a range of opinion on environmental issues. It included a scale on the severity of the problem going from serious to limited to no problem. In addition, all of our studies explicitly identify the range of comment in the categories. Hackett and his associates claim: "the category of "unfavourable" statements seems to involve any comment that is critical on a given issue" is false.

The second problem with the discussion on the favourable/unfavourable dichotomy is the comment: "it conflates the distinction between the "efficacy" (power) and "morality" of news actors" (Hackett, 1992, p. ). They assume that On Balance will address all aspects of news production, even areas that cannot possibly be answered with content analysis. This is not the only instance in the section where the critique essentially poses a question which differs from the one asked in the research and then critiques the research for not answering the new question. For example, the discussion on abortion argues: "both `pro-life' and `pro-choice' groups opposed the government's legislation. Surely it is misleading to combine them into the category `favourable' " (Hackett, 1992, p. ). The conclusion that government received negative attention on the abortion debate is hardly misleading, since, as the critique points out, both pro-life and pro-choice groups were critical of government's handling of abortion legislation. The section on the government in the abortion issue did not ask "Who was critical of government?" it asked "What type of coverage did government receive?" On Balance is not being chastised for using the wrong data to answer questions, it is being chastised for asking its own questions.

In a similar vein, the criticism of our analysis of labour issues over-simplifies the analysis and then proceeds to discuss the reason for the findings in On Balance. The assertion is made: "On Balance (December 1988, p.5) concluded that because labour sources are quoted more frequently than management sources, `management was given the worst press.' " (Hackett, 1992, p. ). As careful readers of the December 1988 issue would have known On Balance explained that the critical coverage given to management was due to the finding that labour sources made more statements than did management sources. It then provided the proportion of coverage each received: "As a result of the enhanced attention paid to labour sources, management got the most unfavourable coverage. In The Globe and Mail, corporations received well over twice as much negative as positive coverage. CBC provided critical coverage of management nine times out of ten. Labour received more favourable coverage from both sources. Interviewees in The Globe and Mail were twice as likely to be favourable than unfavourable toward labour. On CBC favourable coverage was only slightly more frequent than unfavourable" (Miljan, 1988, p. 4). This finding is not disputed in the critique and in fact, evidence is provided to show that other studies have come up with similar results. What is disputed is that On Balance did not explain why this occurred. The reason On Balance did not explain why management received more negative coverage than labour was because a manifest content analysis cannot provide that level of analysis. Why was On Balance criticized for remaining within the logical parameters of its method?

The section on the favourable/unfavourable dichotomy is built on the clever strawman that "all issues have two antagonistic and equally weighted sides" (Hackett, 1992, p. ). By presenting On Balance assumptions in such a simplistic manner, our critics can tear down the strawman with emotionally charged statements such as the following. "If news emphasizes the negative effects of sexual assault, should these be counterbalanced by portrayal of positive effects?" (Hackett, 1992, p. ). To present the argument in this manner is to reduce the debate to absurdity. On Balance has never made the argument that all issues have only two equally weighted sides. We have argued that because the media makes claims of objectivity, all sides of an issue should be presented in a fair and accurate manner. Since the issues and positions on any public policy issue are on the public record the presentation of the arguments can quite easily be subject to scrutiny. On Balance recognizes that certain groups or events will not received balanced coverage, nor should they in some cases, and notes when and why this occurs as we did in the coverage of abortion. It is true that balanced arguments about the government's handling of abortion would have been misleading. But we never required that the coverage on the government should be balanced, we only noted the coverage, and provided the context in which it occurred.

Similarly, the discussion on health care makes a claim that On Balance examined only two perspectives: user pay versus public funding; and then complains that the relative weight of the arguments were not provided. The issue on health care actually examined three means by which health care could be paid for, the third being a mixed system. In terms of the "weight" of the alternatives, it is not our role to direct public policy issues in any direction. The question posed in this case was: "How is the health care system to be paid for" by the media who found that the quality of care was diminishing. The alternatives exist in the United States, Canada and England. The weight of each is dependent on many issues--issues which are outside the scope of our analysis. By providing such an editorial decision we would certainly be guilty of stepping over the bounds of our methodology. One cannot help but be cynical and conclude that if such an analysis were conducted on the pages of On Balance, our critics would attacked that as well.

With respect to the argument that defining statements for or against an issue such as free trade "conflates the range of possible comment" (Hackett, 1992, p. ) this is not only misleading, it ignores the reality of the free trade debate. For one, the assessment of favourable and unfavourable arguments on free trade was comprised on a range of factors including arguments about culture, tariffs, jobs, investment and sovereignty. However, each of these issues had an argument which either was in favour of or opposed to the agreement. To say that there was a wider range of views ignores the historical fact that free trade was a dichotomous issue.

While our critics maintain we see all issues as dichotomous they turn around and criticize instances when a dichotomy is not presented. Which is it? Do we only see two sides or do we examine the issues as they warrant? Even with this argument, the critique misrepresents On Balance studies. The example is provided that because no dichotomy could be presented in the presentation of poverty that On Balance "decries the sympathetic depiction of poverty" (Hackett, 1992, p. ). In reality, On Balance noted the delicacy used by the networks in their descriptions of the poor. We stated: "Natives were given almost no attention on CTV but 15 percent on CBC. Stereotypical images of the poor as substance abusers or the mentally ill were avoided by both networks" (Miljan, 1989d, p.2).

Further in the paper when the same argument is repeated, the critique uses our examination of advocacy journalism to illustrate our "simplistic" approach. The critics state that we define objective journalism as reactive and then complain that we ignored the complexities of the issues. The National Media Archive's methods for assessing "advocacy journalism" such as that practised by Time magazine where Charles Alexander, the science editor, admitted they had "crossed the boundary from news reporting to advocacy" assessed a variety of measures. Among these were: absence of certain kinds of stories, the assumption that the environment was threatened and lack of verisimilitude. The November/December 1989 issue of On Balance summarizes these points. "The lack of reporting the range of scientific views and the presentation of one side to the exclusion of the other provides strong evidence that the networks practised advocacy rather than objective journalism in environmental coverage...Both networks relied overwhelmingly on the view that the environment faces serious problems which should be addressed. For example, coverage on the greenhouse effect emphasized the seriousness of the problem and gave opinions which supported this view. Neither network presented countervailing perspectives from the scientific community or even the degree of such opinion in the scientific community. Similarly, in the coverage on PCBs the networks focused on the indestructibility of the material. Neither discussed the range of methods known to safely destroy PCBs...Another measure to gauge advocacy journalism is the extent to which networks focused on different environmental problems" (Miljan, 1989e, p. 7). In addition to misrepresenting On Balance research, the critique adds a non-sequitur: "objective journalism as On Balance defines it would simply accord even more power to the social and political groups most capable of initiating newsworthy events or of making statements which can be translated into public policy -- notably government" (Hackett, 1992, p. ). Given that our analysis of advocacy did not address social and political groups, how could we define objective journalism as providing more attention to government? We actually questioned why so few scientists were interviewed on scientific issues.

It is ironic that while the critique charges that several methodological errors are committed by the National Media Archive, our results are not disputed. What the critique in fact does is take our findings and develop non-sequiturs to make the findings and our analysis appear unreasonable. In one instance, the critique strongly objects to the analysis of AIDS which examined how accurately CBC depicted persons with AIDS. Hackett, Gilsdorf and Savage claim that because On Balance does not theorize the analysis "it comes very close to inviting the media to indulge in intensified homophobia" (Hackett, 1992, p.). Not only is this illogical, it is also a false classification of the study on AIDS. Despite the claims made in the critique the study on AIDS carefully theorized the benchmark. The very first paragraph of the issue examined Dr. William A. Check's criticisms of the American's media coverage on AIDS. One of his criticisms was that in its desire to popularize the disease it promulgated the theme that "anyone can get AIDS" (Check, 1987). He argued that this theme was so prevalent that it was done to the detriment of informing the public on the disease. Our analysis on the methods of transmission which compared CBC descriptions of those with AIDS and real world statistics found that, "in promoting the theme that AIDS is not a gay disease, CBC obscured the fact that homosexual and bisexual men are most at risk" (Miljan, 1989c, p. 2). We found that rather than honestly approaching how the disease is spread in the majority of cases, CBC national television news focused on the transmission of AIDS through blood transfusions in 22 percent of its descriptions despite the fact that all contacts through blood transfusions happened before 1985--the year in which the Red Cross began screening all blood supplies. Our analysis can hardly be charged with inviting the media to indulge in homophobia. A more logical conclusion, and one which we hoped would be encouraged, is that the media would become more responsible in informing the public about how AIDS is transmitted rather than engaging the public in fear campaigns.

In discussing sampling, the critique charges that the sampling decisions on the free trade debate were motivated to "maximize the probability of finding imbalanced coverage" (Hackett, 1992, p. ). Not only does the critique discount the rationale that in order to make appropriate comparisons between television and newspapers that editorials and speciality sections need be excluded from the sample, but it ignores research which differentiates the public's use of such information. Research on public opinion formation indicates that news reporting influences this process more than editorial information. The reasoning is that the public is more aware of the persuasion tactics used in editorials pieces (Entman, 1989, p.81). Conversely, by the very definition of objective news reporting, the public believes news stories to provide only the facts and little interpretation. In order to make a comparable analysis and to examine balance in news stories, not in editorials, not in business reports, only the front section of the Globe and Mail was compared to the CBC "National" and "Journal." Contrary to the assumption raised by the critique, our expectation, was that the news would not be slanted in either direction, but balanced, both on the CBC and in the Globe and Mail. In an attempt to illustrate sampling inconsistencies on our part the critique adds, "Later On Balance studies, however, do include the "Report on Business with a significant impact on the reported results" (Hackett, 1992, p. ). We would hope that if researchers examine a certain media in one study that would not preclude them from conducting research on different media in subsequent studies. True, The National Media Archive did examine the "Report on Business" in the privatization study. However, in this study the "Report on Business" was compared only with that of the front section, and never compared with CBC. Part of the reason for the analysis was to examine how the two sections compared to each other. We maintain that there was no inconsistency on our part since CBC was only compared with The Globe Mail news stories in the front section.

In addition to questioning the decisions of media we sampled, the critique adds that our sampling decisions skewed results. The study on health care is singled out because the analysis only examined stories pertaining to the health care system and not stories on specific illnesses or ethical questions. As was mentioned in the introductory statements of that issue, we were specifically interested in the coverage of the health care system in terms of the rising costs and increased demands on the system. It is unclear what role reports on medical advances would play in the analysis of how much attention the health care system received. While the media may have given attention to ethical issues, unless those ethical issues specifically mentioned the system, it was not relevant for the study and should not be included in the discussion on the costs of the health care system.

The argument that findings are overgeneralized to characterized whole media organizations is wrong. Every issue of On Balance clearly indicates which programs and media are analyzed. No generalizations were made beyond what the data could bear.

Another tactic used in the paper to discredit our work to selectively omit information that would contradict assertions made in the critique. One such example is the strategy of examining only the first year of On Balance. This particular practice allows for a generalization of On Balance material that is false. That is, the critique passes off as common practice things that were done only a few times and completely ignores changes made to the format over two years ago that directly address the concerns raised in the critique. This selective use of examples wrongly implies that On Balance is a rigid publication that is not concerned with presenting material in the most reliable manner and which cannot adapt and learn from past errors. The brief footnote on page 9 hardly provides adequate explanation for why the changes were not fully accounted for in the critique. As a point of fact, the publication went through numerous changes during the year that was used for the critique as well as during the two years that the critique did not bother to mention.

For example, on page 11 the claim is made: "percentages are often reported without indicating the actual total of statements on which they are based" (Hackett, 1992, p. ). While it is true that the number of statements used were not always reported in the first year of publication, a change was made in the second year to ensure that the number of statements were provided in all graphs and charts. This change is not acknowledge in the critique. Similarly, the critique overemphasizes reporting practices and exaggerates the point in question. For example, it states, "One disturbing tendency is the use of graphs suggesting that the reporting of particular, named journalists is overwhelmingly imbalanced on a particular issue" (Hackett, 1992, p. ). The only case where we found that opinionated statements were directed was in the free trade agreement and we did place these findings on a graph. To claim that one occurrence is a tendency is somewhat of a double standard since, they critique the National Media Archive with generalizing journalist reports with few cases. A similar error appears on page 11 where Hackett, Gilsdorf and Savage state: "The proportion of statements which are "neutral" rather than "favourable" or "unfavourable" on an issue is often excluded or underemphasized..."(Hackett, 1992, p. ). Only three issues out of the 26 that were published neglected to identify the number of neutral statements. This can hardly been seen as a practice which was done "often." No doubt, because the critique ignored the issues which provided neutral statements this error is made: "On Balance distinguishes between "factual" and "opinion" statements, and codes only the latter in its efforts to measure the achievement of balance in news reports" (Hackett, 1992, p. ). This is simply a false assertion since all On Balance material clearly states that all statements are coded. In a related area, the critique attempts to make controversial, even mysterious, the difference between fact and opinion. Nevertheless there really is a difference between fact and opinion. For example, the quote provided by Hackett, Gilsdorf and Savage is commonsensically an opinion, "Many in Congress feel (the current trade situation) allows Japanese and Korean cars made in Canada to sneak into the American market, under cover of the Canada-U.S. Auto Pact" (Hackett, 1992, p. ). The word "feel" clearly demonstrates that this is the opinion of the "many in Congress" and in no way indicates an interpretation of events or issues.

In another instance the authors omit information included in On Balance and complain that it was On Balance which omitted information. They allege that our finding that CBC was provoked to cover health care issues by the Alberta nurses strike was not supported by the data. Not only do they neglect to provide the context for that statement, they also neglect to report the data which supported the finding. The actual statement reads, "However, The Globe and Mail consistently covered health care issues during the year and was not, like the CBC, provoked to cover health care issues by the Alberta strike" (Miljan, 1989b, p. 4). In the preceding page a graph and argument was provided to support that conclusion. The graph provides the amount of coverage CBC and The Globe and Mail gave health care issues over the year time analyzed. The analysis went as follows: "The amount of coverage on health care was most prevalent around the time of the Alberta nurses' illegal walkout. CBC did not air any stories on the health care system in the month prior to the labour dispute. Indeed, coverage on health care seemed to peak just after the nurses's strike. After the walkout, health care stories did not appear on CBC until three months later" (Miljan, 1989b, p. 3). Therefore, the quip that "some claims should at least be qualified" is unsubstantiated.

Methodological Considerations

In addition to the many factual inaccuracies in Hackett, Gilsdorf and Savage's paper many of their assumptions about On Balance's methodology are simply misleading or erroneous. First, Hackett, Gilsdorf and Savage critique On Balance for a "poorly developed theorization of the news process" (Hackett, 1992, p. ). Elsewhere the authors state, "Nor does the National Media Archive formulate any sort of understanding of the nature of news production and distribution" (Hackett, 1992, p. ). The National Media Archive is not concerned with the "news process" (in the sense of production) but with the product. Questions about why certain material is present in the news program and what effects those stories have on the public are left to other researchers to examine. It should be noted, however, that those questions cannot be broached until one has a clear understanding of the content. Thus, the content analysis is a very important first step in understanding those issues.

Second, Hackett, Gilsdorf and Savage question the National Media Archive's examination of balance yet fail to acknowledge journalistic practices and paradigms which strive for balance. The CBC Journalistic Policy states: "CBC programs dealing with matters of public interest on which differing views are held must supplement the exposition of one point of view with an equitable treatment of other relevant points of view" (1988, p.7). In a similar vein, CBC's ad campaign of two years ago featured Peter Mansbridge who states, "Fair. That's what I'd like people to see me as. Fair, credible, trustworthy, accurate--but fair." While Hackett, Gilsdorf and Savage do not think that news organizations hold to the "two sides to every issue" approach the public is led to believe that they do.

Third, the authors on the one hand criticise the National Media Archive for being "right-wing" and impugns the motives of the archive on the basis of its funding sources. For example, "Contrary to the repeated claims of the NMA, the techniques of content analysis are not a guarantee of pure objectivity" (Hackett, 1992, p. ). An elementary methodological social sciences is the trade-off between reliability and validity. The National Media Archive runs inter-coder reliability tests and regularly reports them. Validity depends, in part, on sound judgement regarding categories as well as on sound judgement by coders. On the other hand, the authors criticize the Archive's attempts at ensuring objectivity in the coding design and implementation. "The subjective preferences of individual coders are a secondary problem" (Hackett, 1992, p. ). The assumption here is that the coders have no impute into the categories and rules created. It is precisely the coders who must work to construct the code book and rules and only with their differing viewpoints can we assure that personal biases are not introduced into the code book design. The whole point of selecting a cross-section of coders is to minimize any systematic coder subjectivity through discussion and agreement regarding categories and coding decisions. The National Media Archive selects coders on the basis of their differing political and educational backgrounds to ensure that researchers go beyond their own perspectives and examine exactly what was said. Again, this is commonsensical and standard procedure.

One consistent error throughout the Hackett, Gilsdorf and Savage paper is to imply that Archive studies bias. They claim: "There are a number of problems with this approach, which, to be sure, On Balance shares with some other mass communication research done on news `bias' or `balance' " (Hackett, 1992, p. ). They also allege that "strong implications of editorial partisanship are made" (Hackett, 1992, p. ). What is significant in this section is that although charges of On Balance studying bias runs thick, the proof is rather thin. Rather than provide evidence from the publication, they quote from an editorial. Other editorials have found the exact opposite, why weren't those quoted? Further they make the statement: "On Balance can be seen as part of a process of `ideological mobilization' by the political Right and its corporate allies against the media -- particularly public broadcasting" (Hackett, 1992, p. ). Not only do our critics neglect to provide evidence for such claims they carefully avoid discussing issues of On Balance that could be viewed as left leaning. For example, On Balance did an extensive analysis of how women's issues are reported on network news. How can the finding, "Substantive issues concerning feminism were given little attention" (Miljan, 1990, p. 6) be seen as a right wing attack on public broadcasting?

It appears that our critics are unable to separate the Fraser Institute from the National Media Archive. Why else would they persist in guilt by association arguments such as the one that parallels the Fraser Institute with AIM? It is difficult to understand how this discussion has anything remotely to do with the work of the National Media Archive.

While the authors questioned why the National Media Archive were concerned about the political orientation of its researchers they later go on to state that the interpretations of the researchers are ideologically similar to that of the Fraser Institute, "These gaps are bridged by subjective interpretations by the NMA researchers themselves, whose interpretations generally appear to be consistent with the ideological stances of the Fraser Institute" (Hackett, 1992, p. ). The researchers employed by the National Media Archive are from the Simon Fraser University Co-operative Education Programme. One requirement of the researchers is to write a report on their experiences at their work placements. Consistently, the researchers have stated that the research of the Archive is objective and does not require them to make subjective decisions to correspond with the Fraser Institute. For example, one researcher wrote: "The Archives is part of the Fraser Institute, but is separate from the Institute in that it does not serve the same free market mandate" (Greebe, 1989, p.2). Similarly, another researcher wrote: "The Media Archive is a division of the Fraser Institute but separate from it in that it does not attempt to influence or inform from a free market perspective" (Cushner, 1989, p.1).

In addition to arguing that the National Media Archive slants its work in favour of the Fraser Institute, they imply that because the National Media Archive is funded from the private sector that this ipso facto impugns its independence. The CBC is funded completely with public funds. Does this mean that the CBC product is controlled by the government? Obviously not.

The Hackett, Gilsdorf and Savage piece criticizes the work of the Archive for research it never claimed to conduct and does not take into consideration the type of publication it produces. The critique claims, that On Balance has "failed to position its work in relation to other research on news production, and content" (Hackett, 1992, p. ). On Balance is a general-circulation report, not a specialized social science journal. It is addressed to a wide, not a narrow, audience and strives to be both intelligible and accurate. It also strives to meet the normal standards of social science though it does not have space to report on the state of the literature; nor would the audience of On Balance be particularly interested in it. The general public is comparatively indifferent to the state of the literature, and the social science specialists ought to know it anyhow. Nevertheless, On Balance has made use of the occasional footnote and has referred to recent advances in the social science literature where it was deemed to be relevant. The same criticism is repeated: "On Balance research seems to take in account neither the complexity of the production of news and its meanings nor the relevant literature" (Hackett, 1992, p. ). Following the statement a series of authors are cited, presumably ones which we should have quoted. What is comical that at least one of the authors they note that we should have cited is one in which we did cite (Ericson, 1987; 1989). In addition, several other authors are cited in On Balance such as: Surlin, Romanow and Soderlund (1988), Greenberg, Sachsman, Sandman, Salomane (1989); Taras (1989); Gill (1989); and Soloski (1989).

The National Media Archive has always made available to the public its methods, practices and results as well as access to its database. The National Media Archive welcomes inquiries from members of the public to use the Archive. That is one of its most important purposes. In view of the factual errors, wrong observations and special pleading it is difficult to seriously consider the comments made by Hackett, Gilsdorf and Savage in their critique of On Balance. While their paper is an attempt to critique the content analysis employed by the National Media Archive, no evidence has been put forward that demonstrates their ability to seriously consider such methodological questions.


Blumer, Jay. (1985). European-American differences in communications research. In Everett M. Rogers, & Francis Balle (Eds.), The media revolution in America and Western Europe (pp. 185-199). New Jersey: Ablex Publishing Corp.

CBC (1988). CBC Journalistic Policy. Montreal.

Check, William A. (1987). Beyond the political model of reporting: Nonspecific symptoms in media communication about AIDS. Reviews of Infectious Diseases, 9(5), 987-999.

Cushner, T. (1989). Co-op work report: The National Media Archive. Report prepared for the Simon Fraser University Education Program.

Desbarats, Peter. (1990). Guide to Canadian news media. Toronto: Harcourt Brace Jovanovich.

Entman, Robert M. (1989). Democracy without citizens: Media and the decay of American politics. New York: Oxford University Press.

Ericson, Richard, Baranek, Patricia, & Chan, Janet. (1987). Visualizing deviance: A study of news organization. Toronto: University of Toronto Press.

Ericson, Richard, Baranek, Patricia, & Chan, Janet. (1989). Negotiating control: A study of news sources. Toronto: University of Toronto Press.

Frizzell, Alan, & Westell, Anthony. (1985). The Canadian general election of 1984: politicians, parties, press and polls. Ottawa: Carleton University Press.

Frizzell, Alan, Pammett, John H., & Westell, Anthony. (1989). The Canadian general election of 1988. Ottawa: Carleton University Press.

Gill, Donna. (1989). REAL women and the press: An ideological alliance of convenience. Canadian Journal of Communication, 9(2), 1-16.

Gilsdorf, William, O. (1990). The organizing processes of the CBC-TV national election unit in the 1988 Canadian federal election. Draft paper presented at the ASCRT meeting in Victoria.

Greebe, V. (1989) Co-op report of a work semester at The National Media Archives. Report prepared for the Simon Fraser University Education Program.

Greenberg, Michael, Sachsman, David, Sandman, Peter, & Salomone, Kandice. (1989). Risk, drama and geography in coverage of environmental risk by network TV. Journalism Quarterly, 66(2), 267-284.

Hackett, Robert. (1986). Study of bias at the CBC found wanting. Globe and Mail, p. A7, July 9.

Hackett, Robert A., Gilsdorf, William O., & Savage, Philip. (1992). News balance rhetoric: The Fraser Institute's political appropriation of content analysis. Canadian Journal of Communication, 17(1), 15-36.

Merrill, John C. (1975). Ethics and journalism. In John C. Merrill & Ralph D. Barney (Eds). Ethics and the press: Readings in mass media morality (pp. 8-17). New York: Hasting House Publishers.

Miljan, Lydia (Ed.). (1988). A look at labour: News as entertainment. On Balance, 2(8).

Miljan, Lydia (Ed.). (1989a). Campaign '88: Election wrap-up. On Balance, 1(3).

Miljan, Lydia (Ed.). (1989b). Healthcare: Looking after ourselves. On Balance, 2(2).

Miljan, Lydia (Ed.). (1989c). AIDS: Epidemiology, fear, education, and human rights. On Balance, 2(7).

Miljan, Lydia (Ed.). (1989d). Poverty: Networks coverage differs. On Balance, 2(8).

Miljan, Lydia (Ed.). (1989e). Environment II: Oil spills, pollution and the changing atmosphere. On Balance, 2(10).

Miljan, Lydia (Ed.). (1990). Women and the media: Women's issues in television news examined. On Balance, 3(8).

Soloski, John. (1989). Sources and channels of local news. Journalism Quarterly, 66(4), 864-875.

Surlin, Stuart H., Romanow, Walter I., & Soderlund, Walter C. (1988). TV network news: a Canadian-American comparison. American Review of Canadian Studies, 18(4), 465-475.

Taras, David. (1989). Television and public policy: the CBC's coverage of the Meech Lake Accord. Canadian Public Policy, 15(3), 322-334.

Taras, David. (1988). Television is most believable medium, poll finds. Marketing, February 15.

  •  Announcements
    Atom logo
    RSS2 logo
    RSS1 logo
  •  Current Issue
    Atom logo
    RSS2 logo
    RSS1 logo
  •  Thesis Abstracts
    Atom logo
    RSS2 logo
    RSS1 logo

We wish to acknowledge the financial support of the Social Sciences and Humanities Research Council for their financial support through theAid to Scholarly Journals Program.