2. Methodology

This chapter describes the specifications for the evaluation, the lines of enquiry used to fulfil the terms of reference, as well as the data sources and analysis involved in the study. It also comments briefly on the quality of the data and the limitations of the evaluation.

2.1 Evaluation design

An evaluation of this nature does not lend itself easily to a conventional experimental research protocol that measures program variables in a tightly controlled environment in order to scientifically demonstrate the existence of causal relationships between the parameters (for example, the design of Katimavik and the program’s results). The complexity of the program, the number of elements it features, the broad spectrum of themes it covers, the holistic character of the approach and the length of time it takes to see results are impediments to that type of approach. Determining the exact proportion of results that can be clearly attributed to the program is a huge challenge in the case of Katimavik, because many other factors have a bearing on results, starting with the social and economic conditions in which participants live once they go home after their placements. For all these reasons, the evaluation was based on quasi-experimental research design in which data were collected in a semi-controlled environment in order to gather as much relevant information as possible. The indications on hand were then cross-referenced to identify with reasonable certainty Katimavik’s contribution to the achievement of results observed during the evaluation.

2.2 Lines of enquiry

Four lines of enquiry were needed to carry out the summative evaluation of the Katimavik program.

Firstly, a document review was completed to identify the various aspects of Katimavik, place the program in its historical and current settings, compare it to similar programs, and compile information that would be useful in analysing key issues. The document review included the examination of: core documents provided by PCH and Katimavik-OPCAN; documents gathered by the consultants when they visited the head office of those two organisations and three of Katimavik-OPCAN’s regional offices (Halifax, Calgary and Montreal); documents found on the Internet; and documents obtained in the course of the evaluation. The review also included analysis of the various databases currently used by Katimavik-OPCAN.

Secondly, interviews with key stakeholders were conducted to determine whether it was appropriate for the federal government to contribute financially to Katimavik in the current climate and gather comments and impressions from close observers on a number of subjects: relevance of the program; fit between program objectives and Government of Canada priorities and needs of youth and communities; program design and delivery tools; and results achieved. In all, 32 informants (see Appendix 4) were interviewed, including PCH officials, Katimavik-OPCAN managers and staff at the head office or in regional offices, an expert from the community sector, and past associates of the program.

Thirdly, using the FluidSurveys web application, the consultants conducted three online surveys of stakeholders groups involved in Katimavik at some point between fiscal years 2005-06 and 2008-09, namely former participants, unsuccessful applicants and community partners.⁷ The purpose of the surveys was to gather the views of respondents throughout Canada regarding aspects of their experience deemed relevant to the summative evaluation, taking into account life events that occurred before, during and/or after their time in the program. Surveys were completed over the Internet by: 645 former participants with a margin of error of ± 3.5% 19 times out of 20; 131 unsuccessful applicants with a margin of error of ± 8.5% 19 times out of 20;⁸ and 134 community partners with a margin of error of ± 8.2% 19 times out of 20.

Finally, three series of focus group sessions, each lasting two hours, brought together ordinary Canadians, former program participants and engaged youth who had never taken part in (or applied to) Katimavik. The purpose of the sessions was to validate certain findings of the online surveys and gather perceptions and opinions regarding specific elements of the program, including: the public’s awareness of and level of support for Katimavik (discussions with ordinary Canadians); the long-term impact of the Katimavik experience (discussions with former participants); and factors likely to motivate young Canadians to involve themselves in community service (discussions with engaged youth). This line of enquiry made it possible to consult 98 individuals split in 10 sessions, as follows: \

• five sessions involving ordinary Canadians—two in Montreal (one in English, one in French), one in Ottawa (bilingual), one in Halifax (bilingual) and one in Calgary (in English);
• four sessions involving former participants—two in Montreal (one in English, one in French), one in Halifax (bilingual) and one in Calgary (in English);
• one session in Montreal (bilingual) involving engaged youth.⁹

2.3 Data sources and analysis

To prepare this report, the consultants analysed data gathered between February and June 2010 by way of the aforementioned four lines of enquiry. The data were obtained using hybrid methods combining both qualitative and quantitative research techniques, then consolidated and triangulated in order to identify concurring observations as well as complex or controversial points on which sources did not always agree. This strategy resulted in valid and reliable findings on the various issues covered by the evaluation. The consultants were thus able to draw conclusions and make recommendations based on proven facts and informed judgments.

While the summative evaluation covers the fiscal years from 2005-06 to 2008-09, this report takes into account the many changes undergone since 2008-09—and especially since early 2010—by the program and the entity that delivers it, Katimavik-OPCAN. These recent developments did not radically alter the nature of the analyses underlying the report, but they did have a bearing on some findings and, more importantly, on the recommendations based on those findings.

2.4 Data quality and limitations of the evaluation

In general terms, the summative evaluation was carried out under difficult conditions because of the complexity and scope of the Katimavik program, the constraints associated with tight evaluation timelines, and above all the predicament in which Katimavik-OPCAN landed starting in early 2010. The data collection phase roughly coincided with the departure of influential members of Katimavik-OPCAN’s management team—including the Executive Director and the Chairman of the Board of Directors—and with the implementation of strong measures made necessary to modernise the program and cut back costs following a significant decrease in financial contribution from PCH. Consequently, despite all their good intentions, Katimavik-OPCAN staff did not always find time to quickly provide the information requested—or to check the accuracy of information submitted—by the consultants.¹⁰

Despite these constraints, the findings of the evaluation are based on trustworthy sources and information. Furthermore, where it was possible to compare their data with the data collected for the 2006 summative evaluation, the consultants found similarities that tend to confirm the consistency of some of the findings made at five-year intervals, particularly with respect to the surveys. That said, the content found in the next chapters should be interpreted with circumspection for the following reasons:

• Because they used a quasi-experimental research protocol, the consultants came up with findings that are based on solid indications, yet cannot serve to demonstrate with absolute certainty the existence of causal relationships between the variables measured by the evaluation. Moreover, it would be unwise to draw overly general conclusions on the sole basis of the outcome of focus group sessions or surveys of unsuccessful applicants or community partners, given the limited representativeness of the samples used for those lines of enquiry.
  
  
• Some findings are based on a comparative analysis of the opinions expressed by the former participants and unsuccessful applicants who responded to the surveys. This type of analysis inevitably entails a risk of bias because it focuses exclusively on youth who are drawn to Katimavik (as opposed to youth for whom the program holds no interest). Consequently, care must be taken in extrapolating the resulting findings to all young Canadians.
  
  
• The consultants were unable to analyse all the information on hand regarding Katimavik because they were late in receiving some of the data needed for the evaluation. Over the years, the program has generated a very large number of documents and reports, not all of which are used to their full potential by YPD and/or Katimavik-OPCAN. Furthermore, apart from the information provided through the Internet portal currently being operated by Katimavik‑OPCAN,¹¹ older data were hard to access and difficult to use. As a result, the findings made for program years 2005-06 and 2006-07 are based on statistics of limited scope.
  
  
• The sources consulted disagreed on a number of points related to the design and delivery of the program. In fact, Katimavik fuels philosophical debate between advocates of the status quo, who like to invoke the legacy of a longstanding tradition, and advocates of change, who are always willing to challenge the foundation on which the program is built. To a large degree, this polarisation of opinion accounts for the differing views seen throughout the findings of this evaluation.
  
  
• Finally, the report discusses the possible impact of changes the program and Katimavik-OPCAN have undergone since 2009-10, but does not offer a comprehensive or documented analysis of those changes. Events that took place after the 2008-09 program year are beyond the scope of this evaluation. Furthermore, at the time this report was written, the information needed to study the impact of changes that have been made since 2009-10 was still too scarce or fragmented to allow for an informed opinion on the subject.

[ Previous Page | Table of Contents | Next Page ]