Information identified as archived on the Web is for reference, research or record keeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you may request alternative formats by contacting the Department of Canadian Heritage.
3 Key Findings
This chapter sets out the main observations resulting from the evaluation. It examines in detail the four (4) areas on which the analysis focused: the relevance of the Aboriginal Peoples’ Program; program design and delivery; performance, cost-effectiveness and alternatives. The observations come from synthesizing the information gathered using all lines of evidence described in the methodology.
3.1.1 Consistency with PCH and Government Priorities
APP Linkages to the Government of Canada Budget and Other Priorities
The stated PCH Strategic Outcome: Canadians Have a Sense of Their Canadian Identity (PCH Program Activity Architecture (PAA) 2009-10) recognizes:
“…the need for Canadians of all origins (as members of communities and society) to connect with one another, to better understand their country and its basic shared values, and to come together and celebrate as co-contributors to Canadian society, thereby giving them a sense of Canadian identity.
These objectives support the mandate of fostering a stronger Canadian identity through active, engaged, inclusive citizenship, and they recognize the importance of linguistic identity and a shared civic identity among Canadians to the Canadian national interest.”
“…The outcome implies a unity based on shared democratic values such as respect for human rights and diversity. These values are underscored by the strengthening and promoting of common democratic institutions and civil society, and by the formal and informal recognition of three (3) historical pillars: Canada's Aboriginal peoples, the French and English linguistic communities, and ethnocultural and religious minority communities formed as a result of immigration over many generations.”
This PCH Strategic Outcome and related objectives are directly linked to the following current Government of Canada Outcome Areas:
- Primary: A diverse society that promotes linguistic duality and social inclusion.
- Secondary: A vibrant Canadian culture and heritage; Safe and secure communities.
Alignment with PCH Mandate
Created in 1993, the Department’s legislative mandate is set out in the 1995 Department of PCH Act that identifies the responsibilities for the Minister under the heading of “Canadian identity and values, cultural development, and heritage.”13 The roles and responsibilities of the Department of PCH have evolved since its creation. The current roles identified for PCH are in the 2009-2010 Report on Plans and Priorities (RPP).14.
The RPP notes that PCH, along with Canada’s major national cultural institutions; “… plays a vital role in the cultural, civic and economic lives of Canadians by working together to promote culture, the arts, heritage, official languages, citizenship and participation, Aboriginal, youth, and sports initiatives.”15 Currently, PCH’s main activities involve funding for community and other Third-party Organizations to promote the benefits of culture, identity, and sport for Canadians.16.
Since its creation, PCH has been responsible for delivering a range of Aboriginal-specific programs and initiatives that serve Inuit, Métis, Non-Status Indians and First Nations people living primarily off-reserve. Some of these programs have been in operation for over 30 years: Support for Aboriginal Friendship Centres, the Women’s Community Initiatives, and Aboriginal organizations dates back to the early 1970s. Support for northern broadcasting and the territorial languages accords dates back to the early 1980s.
Alignment with PCH Canadian Identity Objective
As noted above, Canadian Identity is the focus of one of the three (3) current (2009/10) departmental strategic objectives. Engagement and Inclusion, as one of the three (3) programming areas under Canadian identity, focuses on leveraging the benefits of diversity and to promote inclusiveness and engagement of all Canadians in all aspects of society. The APP is aligned with this overall objective and seeks to strengthen Aboriginal cultural identity and participation in Canadian society:
- Aboriginal Communities Component supports the efforts of Aboriginal communities to develop innovative and culturally appropriate solutions to the social, cultural, economic and other obstacles that impede community and personal prospects, by funding the network of Aboriginal Friendship centres across Canada, urban Aboriginal youth projects including Young Canada Works for Aboriginal Urban Youth, and projects undertaken by Aboriginal women's organizations; and
- Aboriginal Living Cultures Component supports the preservation and promotion of Aboriginal languages and cultures through community-based projects, and funds the production and distribution of Aboriginal radio and television programming by northern Aboriginal communications societies. (PCH PAA 2009-10).
Fit in Canada’s Urban Aboriginal Strategy
The Urban Aboriginal Strategy (UAS), specifically targets off-reserve Aboriginal peoples but only in 13 Canadian cities, whereas UMAYC projects are scattered in 149 communities across Canada. Under the direction of the Federal Interlocutor for Métis and Non-Status Indians (who is also the Minister of Indian and Northern Affairs), the UAS seeks partnerships with other federal departments and with the private sector, provincial and municipal governments, and community and Aboriginal organizations The UAS responds to local priorities and advances the UAS national priority areas of: improving life skills, promoting job training, skills and entrepreneurship and supporting Aboriginal women, children and families.
While the UAS and UMAYC share similar goals vis-à-vis strategically focused investments to enable urban Aboriginal communities to be more self-reliant within strong and vibrant cities, the UAS is not specific to youth, nor does it employ a cultural and identity focus. Several examples of UAS projects include life skills training for single mothers, funding feasibility studies for economic development projects, pre-employment initiatives for Aboriginal youth transitioning to the work force, and responding to the needs of Elders in urban communities.
The 2007/08 RMAF/RBAF on the renewal of the UAS, indicates that the aim of the renewed and enhanced UAS is to promote self-reliance and increase life choices for Aboriginal people in urban centres. (UAS RMAF/RBAF, Page 7) It also states that the renewed UAS is linked to INAC’s strategic outcome of ‘improving the socio-economic conditions of Métis, non-status Indians and urban Aboriginal people. The objectives of the Renewed UAS are:
- Targeting urban Aboriginal socio-economic needs;
- Improving access and coordination of programs and services;
- Co-ordination of policy research knowledge and information sharing;
- Improved horizontal linkages and policy harmonization federally;
To address these objectives the renewed UAS was strategically focused on investments in three (3) priority areas:
- Improving skills;
- Promoting job training, skills and entrepreneurship; and
- Supporting Aboriginal women, children and families.
The relationship between the PCH APP and the UAS exists at several levels. At the broadest policy level, all of the APP programming elements relate to the UAS in that they address the wide-ranging conditions affecting Aboriginal people living off-reserve. Some elements of the APP have been identified as being directly related to the Renewed UAS in the formal documents for accountability purposes. At the same time, some of the APP programming elements relate to other, horizontal initiatives (namely the YES and FVI), and accountability for these elements is shared with other (and different) groups of federal departments and agencies.
Fit within the Department of Canadian Heritage
Since 2005, the federal government has, through various decisions, clearly continued to assign responsibility for the APP to PCH. The distinction between Aboriginal persons living either on or off reserve lands is an important one as that is how the boundary lines have been established between PCH and INAC. INAC is the lead department for First Nations living on reserve lands and Inuit living in Inuit communities and hamlets. The distinction between INAC and the mandates of other federal government departments, including PCH, changed in 2004 when the Minister for INAC was assigned responsibility as the Federal Interlocutor for Métis and Non-Status Indians (not living on reserve lands). From its inception in 1985 until 2004, the role of Federal Interlocutor had always been assigned to another Minister with a non-Aboriginal mandate.
INAC’s website states:17 The Federal Interlocutor helps to find practical ways to improve federal programs and services for Métis, Non-Status Indians and urban Aboriginal people. In support of this goal, the Office of the Federal Interlocutor is responsible for:
- Maintaining and strengthening the Government of Canada's relationship with national Aboriginal organizations that represent Métis, Non-Status Indians and urban Aboriginal people, including the Congress of Aboriginal Peoples and the Métis National Council;
- Participating in negotiation processes with these organizations and the provinces; and
- Coordinating the Federal government's Urban Aboriginal Strategy (UAS).
As noted earlier, the Government approved the restructuring of the APP in April 2005 and the transfer of the Aboriginal Organizations component to INAC in 2007.
Findings [R1]: The APP is linked with Government of Canada priorities. APP is aligned with PCH mandate, objectives and priorities.
The APP relates directly to established Government of Canada strategies and priorities for urban Aboriginal programs and services.
There are rationales for programs such as the APP to address the specific needs documented in the literature to improve conditions for a growing urban Aboriginal population as well as to enhance social inclusion and cohesion in Canada.
The APP is aligned with the current mandate of PCH as it relates Canadian identity, engagement and inclusion. The 2005 consolidation of the APP and the subsequent implementation strategy on integration have provided an opportunity to realign the APP to the overall PCH mandate, including the transfer of the previous Aboriginal Organizations Component to the Office of the Federal Interlocutor in 2007.
While there are linkages at a strategic level between the APP and the UAS; the extent of coordination and complementarities between the APP and the UAS is difficult to identify from the available background documentation.
3.1.2 Relevance of APP to Aboriginal Stakeholders
Relevance to Urban Aboriginal Peoples Needs
The literature review identified a wide range of urban Aboriginal Peoples’ needs varying from social, cultural and economic, through to specific needs such as increased programming in the areas of urban Aboriginal youth. The literature review did not identify a structured analysis that gathered or identified the needs of urban Aboriginal Peoples, and how those needs differed from the needs of non-urban Aboriginals and the Canadian population as a whole.
The literature and document review, supported by interviews with PCH staff confirmed that, with the exception of the UMAYC program element, PCH has not conducted a formal needs assessment; relying on the ultimate recipient organizations to design projects that meet the understood needs of urban Aboriginal People living within the area served by their organization.
PCH staff explained that they rely on the needs analysis undertaken by Third-party and Ultimate Recipient Organizations regarding the extent to which the needs of urban Aboriginal People are being addressed. That said, with few exceptions, project file and document reviews did not find that Third-party and Ultimate Recipient Organizations undertook needs assessments to support their projects.
APP Current and Future Relevance Based on Demographic Trends18
As of 2006, the Aboriginal population is over one million, at 1,172,790. This population reflects a 45% growth rate over the last decade, in comparison to an 8% increase for non-Aboriginals. Aboriginal Peoples comprise almost 4%of Canada’s total population.
The greatest population per capita is within Nunavut where 24,920 Inuit represent 85 % of the territory’s total population, followed by the Northwest Territories (50%) and the Yukon (25%). Eight (8) in ten (10) Aboriginal people live in Ontario and the western provinces.
The greatest population increases have occurred in Nova Scotia (95%), New Brunswick (67%), Newfoundland and Labrador (65%), Quebec (53%), and Ontario (68%). Among the western provinces, Manitoba had the greatest growth at 36%, followed by Saskatchewan (28%) and the Yukon Territory (23%). The greatest increase since 1996 has been amongst the Métis; a 91% growth rate has brought their population to 389,785. First Nations and Inuit populations grew by 29% and 26 % respectively.
The 2006 Census data indicates that the Aboriginal population is becoming more urban. Approximately 54% lived in urban areas (large cities and smaller urban centres), up from 50% in 1996. The metropolitan areas with the largest Aboriginal populations include Winnipeg (68,380 representing 10% of its population), Edmonton (52,100 or 5% of population), Vancouver (40,310 or 2% of the population), Toronto (26,575 or 0.5% of its population) and Calgary (26,575 or 2% of population).
Several smaller Western urban centres had relatively large percentages of Aboriginal people, such as: Thompson, Manitoba; Prince Rupert, British Columbia; and Prince Albert, Saskatchewan (36%, 35%, and 34% of the respective populations). This increasingly urban Aboriginal population includes mostly First Nations (50%) and Métis (43%) people, while very few Inuit live in southern urban areas.
In terms of age, the Aboriginal population is much younger than that of non-Aboriginals, 27 years versus 40 years as the median age, but particularly in Nunavut, Saskatchewan, and Manitoba (20, 22, and 24 respectively). Almost half (48%) of the Aboriginal population is comprised of children and youth 24 years and under.
With respect to urban centres, this age group has large representation in Regina, Prince Albert (both 56%) and Saskatoon (55%). Like the national trend, the Aboriginal population is getting older. Although seniors only represent 5 % of the Aboriginal population (versus 13% with non-Aboriginals), this population has doubled between 1996 and 2006.
Projected demographic trends show a continuing growth in the urban Aboriginal population, remaining younger than the Canadian average, and also aging and reflecting urban residential patterns. While the total population in each age-grouping is growing in absolute numbers, the percentage of Aboriginal children under the age of 15 years, as a part of the total Aboriginal population, is projected to decline from 33% to 29 % by 2017. This decrease reflects declining birth rates, although still higher (1.5 times higher) than the overall Canadian birth rate. In addition, decreasing mortality trends are contributing to an aging population.19.
The case studies and discussions with PCH staff point to a consensus that needs related to culture, identity and language evolve slowly over time, sometimes measured in generations and that changing demographic trends mostly impacts upon the volume and urgency of needs, and not the needs themselves.
Programming Alignment with Organizations’ Objectives
The respondents, all of whom currently receive APP funding, were of generally two (2) group types. One group included those that existed and were seeking additional project funding that matched their interests and objectives. A second group included those organizations that were ‘assembled’ to seek funding to deliver a project. This was often the case for UMAYC projects.
When asked the question "To what extent is the APP aligned with the objectives and activities of the recipient organizations?" the respondents were in general agreement that they had sought APP project funding because the APP program criteria most closely matched all or a portion of their organizations’ mission statements and objectives.
With respect to project funding and not core funding, respondents stated that the program criteria were not the limiting factor, as there was considerable flexibility in the types of projects that PCH will accept and fund. Respondents also stated that the major limiting factor is the lack of available funding; that they had excellent projects that addressed the urgent needs of urban Aboriginal Peoples and PCH officials indicated that there was not enough money to fund all proposed projects.
Aboriginal Community Language and Culture Needs
A national study conducted by EKOS Research Associates Inc. (2006) provides a starting point to examine the issues and concerns amongst the urban population. Two thousand and eighty-three (2083) Aboriginal people who were not currently residents of a reserve participated in a telephone survey or personal interview regarding various activities within the previous twelve months. Examined in the EKOS study were:
- Participation in community;
- Cultural and creative activities;
- Language; and
- Internet use.
The results of participation in community groups, organizations or events varied, however participation was greater in Aboriginal-based activities; 20 %participated weekly; and 25 % participated on a monthly or yearly basis.
The cultural identity of Aboriginal Peoples living off-reserve was strong (60%) in the EKOS study, even though a lower number (40%) indicated a strong sense of cultural identity within their community. While most mentioned they participated in a cultural activity, they preferred live performances such as concerts, plays, and dances. Approximately half attended either exhibits of visual art or historical artefacts. Participation in creative activities, such as crafts, dance, acting, singing, or music was less common with less than 50% engagement. Roughly half (53%) watch the Aboriginal Peoples’ Television Network (APTN) daily, mostly for 1-2 hours (41%) and viewership increases with age.
The EKOS study examined languages spoken, views on preservation, and learning approaches. In terms of the languages used at home and within the community, almost all participants (93%) in this study spoke English, with one-fifth speaking French; about 7% speaking both English and French. Smaller percentages of Aboriginal languages were spoken: Cree (13%), Ojibway (6% ), and less of other Aboriginal languages.For those who speak an Aboriginal language, just over three-quarters acquired the language as a child, while 20 % learned their Aboriginal language as an adult. This occurred more frequently for younger respondents under the age of thirty-four and/or those with post-secondary education.20
Importance of language maintenance is greater amongst language speakers living off-reserve than those living on-reserve. Not only was intergenerational learning a factor of importance, 21 but that language skills are maintained through speaking, writing, storytelling and ceremonies. Importance of language maintenance was greater among:
- Inuit and First Nations people;
- Women; respondents between the ages of 35 and 44;
- The more educated;
- Those with strong cultural identity;
- Those with greater household incomes; and
- Residents of areas of large Aboriginal populations.
Although the language maintenance was reported to by strongest need amongst Aboriginal peoples living off-reserve, Inuit youth desired increased access such as learning, hearing and using Inuktitut.22 Furthermore, these youth think governmental initiatives should facilitate, not replace, home and community-based efforts.23 As one of the most viable Aboriginal languages, the number of speakers is also decreasing (69% from 72% in 1996) with low percentages outside Nunavut (15%) or in CMAs (19%).24
In 2001, the Canadian Census reported 3,100 Aboriginal people as artists, primarily artisans and visual artists. Today, Aboriginal artists are integrating old traditions with new media forms, such as the popular modes of expression of circus arts and hip-hop among Aboriginal youth (2008).25 An unknown future of participation in arts as mediated through demographic data such as ethnic origins and rates of population growth and decline has been stated by the Canada Council for the Arts (CCA).
Changing Aboriginal Language and Culture Needs
There is high mobility of First Nations people, especially amongst the young and women, from reserves to cities, when compared to non-status Indians, Métis, and Inuit (Norris, Cooke, & Clatworthy, 2003). Norris et al also indicated high residential mobility within cities amongst all groups of Aboriginal peoples, but moving back to reserves is generally only an option for First Nations people.
From research examining the relationship of Aboriginal language use and the low socio-economic status of First Nations people, O’Sullivan (2003) could not claim to produce definitive evidence that demonstrated that language use either helped or hindered communities. Thus, he suggests that future research focus on the “relative importance of language use in itself, and change in language use (and related cultural changes)” (p.159), of which longitudinal research may be most effective to ascertain this information.
Norris (2003) states that the language continuity is determined by the average age of mother tongue speakers and the age of home language speakers. A linear inverse relationship is observed as younger speakers, stronger language. Thus the language outcomes of children are quite significant not only for maintenance, but for survival. This statement is strengthened by UNESCO’s position that at least 30%of children in a community must speak the language or it becomes endangered, and that it must be spoken in the home for maintenance and survival for future generations (1996).
Findings [R2]: APP is and will continue to be relevant to Urban Aboriginal Peoples’
Using a wide range of methodologies, including file review, documents review, literature review, interviews and case studies, this evaluation concludes that the program is relevant to the needs of urban Aboriginal Peoples.
With the exception of UMAYC, neither PCH nor the Third-party and Ultimate Recipient organizations have conducted a formal needs analysis of urban Aboriginal Peoples as it relates to APP and its objectives.
An increase in the population of urban Aboriginal Peoples may bring a corresponding increase in the demand for services and support from APP.
APP efforts will continue to provide support relevant to urban Aboriginal Peoples in the future by providing services that address specific opportunities such as maintaining the use of Aboriginal language.
3.1.3 Relevance of Territorial Language Accords
Canada-Territorial Language Accords (TLA) are government-to-government agreements with each of the Governments of Yukon, Northwest Territories and Nunavut that fund services in Aboriginal languages.
Aboriginal Languages in the Territories
Analysis by Statistics Canada in 2006 provided the following language breakdown presented in Table 8.
Table 8: Language Characteristics, Aboriginal Population in Territories, 200626
Total Population in each Region (Aboriginal plus non-Aboriginal)
Total Aboriginal Population
Mother Tongue is an Aboriginal Language
Speak an Aboriginal Language Most Often at Home
Knowledge of Aboriginal language(s)
The Government of the Northwest Territories recognizes 11 official languages including English, French, and nine (9) Aboriginal languages belonging to three (3) different language families: Dene, Inuit, and Cree. Dene, part of the Athabaskan family, is the language spoken in the majority of the Northwest Territories (NWT). Other languages in this family include Chipewyan, Dogrib, Gwich’in, North Slavey and South Slavey.Three (3) Inuit languages, Inuvialuktun, Inuinnaqtun and Inuktitut, are also recognized in NWT, with 20% of the Inuit population speaking either Inuvialuktun or Inuinnaqtun. Inuktitut speakers of NWT primarily live in Yellowknife. Cree, belonging to the Algonquian family, is spoken by a small number of people living mostly in the areas of Fort Smith and Hay River. English and French is more common in Yellowknife and the regional centres.27
Just over a third of the Aboriginal population speaks an Aboriginal language, with slightly fewer learning it as their mother tongue, and 13% using it in the home.In Canada, there are ten (10) Inuit languages spoken, including Inuktitut (Nunavut), Inuttitut (Nunavik in northern Quebec), Inuttut (Nunatsiavut region of Labrador), Inuinnaqtun (western Nunavut), and Inuvialuktun (Inuvialuit region of the NWT) 28 From the 2006 Census, Inuktitut continues to be one of the most spoken languages, and its use is declining. Mother tongue has declined by 4 % in the past decade and its use as the home language has decreased by 8%. Speaking ability varies by Inuit region with the majority (99%) speaking Inuktitut in Nunavik compared to two (2) in ten (10) in the Inuvialuit region. Only 15% of Inuit living in urban areas could carry on a conversation in their mother tongue. Among Yukon First Nations, there are eight (8) languages from two (2) major language families: (Inland) Tlingit and Athapaskan. The Athabaskan family is the largest language family in Canada and has seven (7) dialects being Gwich’in, Han, Upper Tanana, Northern and Southern Tutchone, Tagish, and Kaska.29
Although the Yukon has the smallest Aboriginal population of the three (3) regions, its Aboriginal languages face the greatest challenges. In 2006, just 16% could speak an Aboriginal language, and fewer than 12% learned an Aboriginal language as their mother tongue, with 2% using it in the home (Statistics Canada). Older people, especially over age of 65, were more likely to have the ability to conduct a conversation in an Aboriginal language. While considerable effort by the Yukon is channelled to language maintenance, such as school-based program and the Yukon Language Centre (teacher training and resource development), it is felt that the impact is minimal.30
Trends in Use of Aboriginal Languages
UNESCO’s 2010 report; Atlas of the World’s Endangered Languages notes that: “Canada’s Aboriginal languages are among the most endangered in the world.” The trends in the use of Aboriginal languages are clearly articulated in a discussion of trends and perspectives on second language acquisition (Norris, 2007). Today there are 86% or more Indigenous languages, belonging to eleven Aboriginal language families within Canada. These languages are perceived to be steadily declining as ten (10) have already been lost over the last 100 years and less than one (1) in four (4) individuals speak an Aboriginal language.
Cree, Inuktitut and Ojibway are the only viable languages with populations large enough to sustain extinction over the long term (Norris, 2004). One of the most common methods of analyzing language shift is to examine home-language and mother-tongue ratios. Despite the decreasing trend of intergenerational transmission of Aboriginal mother tongues, an increase in second language learners may slow rapid erosion and possible extinction. Language viability is largely influenced by the number of people who speak it within the home (Norris & Jantzen, 2003; RCAP, 1996).
In the 2006 Census, slightly less than 12% of Aboriginal Persons reported speaking an Aboriginal language as the primary home language. Other trends include declining mother tongue transmission, especially amongst women in the “child-bearing and working-age years” (Norris and Jantzen, 2003).
PCH’s Aboriginal Languages Role in the Territories
PCH has a national role for Aboriginal languages and the extent of PCH’s role in the Territories has been evolving, in an environment where there are known decreasing trends in the use of Aboriginal languages within the territories.
While the TLAs in the NWT and Nunavut are used to fund Aboriginal language activities for government services and to support languages projects in communities; the Yukon government withdrew from its TLA in 2007.
Findings [R3]: There is a noted decrease in the vitality and use of Aboriginal languages. PCH plays a role with regards to Aboriginal language in territories
There is a clear trend in the decreasing vitality of Aboriginal languages in Canada. This trend reflects smaller populations of those who speak an Aboriginal language or who learned it as a child.
The trend indicates that Aboriginal languages are less likely to be used in the home or other domains such as places of education and employment. Less use of a language reduces opportunities for intergenerational transmission.
PCH’s Aboriginal Languages Initiative includes the Territories.
3.2 Design and Delivery
3.2.1 APP Elements Supporting Program Effectiveness
Objectives of the APP Consolidation
Following a two-year comprehensive review in April 2005,31 the Government approved the findings of the review and the ‘restructuring of programming elements within the Aboriginal Affairs Branch of PCH under a consolidated policy framework to better define programming, and allow for administration and reporting efficiencies and the ability to monitor and report on results strategically.’
APP implementation moves programming towards strategic change by providing a bridge from the former administration of individual programs to the consolidated APP. Work continues on implementing the new program and PCH reports that much of the data collection, consultation, program framework development, and programming operations and delivery tasks have been completed along with expected training and capacity building tasks.PCH staff commented that the APP implementation strategy focuses on integrating program components and programming elements and that the integration and search for administrative efficiencies focused on headquarters level and less so in the regions. The changes that PCH staff members felt were most visible were at the project level include the provision of additional tools, better defined reporting requirements and refinements to the program eligibility and other criteria.
APP Consolidation and Expected Changes in Effectiveness
Two (2) internal audits, the first of the Aboriginal Communities Component (2008) and the second of the Aboriginal Living Cultures Component (2009), identified concerns about the program accountability systems in place to adequately monitor and measure program performance. In addition, the audits identified that in some cases (for example, in the Aboriginal Languages Initiative (ALI)), the guidelines for eligible recipients and eligible projects were unclear which creates the risk that projects funded may not be aligned with program objectives.
The Aboriginal Living Cultures audit also noted that, although program delivery standards for some programming elements were developed internally, they were not implemented and not communicated externally. In other programming elements, the audit noted the absence of program delivery standards. As the Audit (2009) notes: “…when internal service delivery standards are not developed for a program, it increases the risk that program objectives and expected turnaround times are not being met and the Program is not able to evaluate its performance on the internal turnaround time for application assessments.” (Audit of ALC Component, 2009, Page 11)
The Aboriginal Living Cultures audit included recommendation to improve mechanisms to measure and manage the performance of the ALC component. While noting that the APP Directorate has started to develop recipient performance reports in 2006-2007 and started the collection of performance related information in 2007-2008; analysis of this information had not yet been initiated in 2008-2009.
The Aboriginal Communities audit identified concerns about the lack of performance management data. The first of 9 recommendations for improvement was that:
“The Director General of the APP must develop and implement a formalized performance management and reporting process. The tools .. must include , but not be limited to, a performance reporting process with the Regions with accountability between the Regions and HQ, performance targets specific to the ACC that derive from the umbrella RMAF of the APP as well as a project work plan for the contemplated program performance management database if approved.” - Audit of AC Component, 2008, Page (i)
Perception that the Consolidation and Program Transition Increased Linkages and Improved Effectiveness
There are linkages between APP programming elements by virtue of the delivery model using Third-party Organizations, and the target populations for various programs. For example, the NAFC and other P/T associations are directly involved in the delivery of youth program funding for UMAYC. In addition, various APP programming elements relate to other government strategies and initiatives (such as the Urban Aboriginal Strategy, the Youth Employment Strategy, and the federal Family Violence Initiative.
PCH key informants stated that while some linkages were created between programming elements from an administrative perspective; from a program perspective, new linkages were not created from the consolidation of APP. PCH staff explained that most of the linkages that existed before the integration of the programming elements into APP are better identified with the integration of the APP bringing the separated program teams together.
PCH key informants also indicated that for the most part effectiveness has not improved as increased administrative and financial controls have increased the amount of work needed to manage the APP programming elements. Key informants stated that there was a need to improve administrative and financial controls, and that there was a cost associated with the changes.
Some improvements were noted in that staff was provided with some additional tools to support them and their work. While it is not clear that overall program effectiveness has advanced, project management and financial control have improved.
Findings [D1]: Administrative and reporting efficiencies are currently being addressed. Changes in program effectiveness have not yet been assessed. There is no perception of increased linkages between the APP program elements.
The consolidation of the previous programming elements within the APP is in the process of implementation to address the objectives of improved administrative and reporting efficiencies.
As the APP continues to mature, performance data will become available and facilitate an assessment of changes in program effectiveness. Once fully implemented, this performance measurement framework will produce a set of effectiveness measures tracked over time.
PCH continues to work on updating the APP programming element criteria that may result in the identification of additional linkages. The program will continue to examine the extent that consolidation has created new linkages between program elements.
There is the perception that project management and financial control has improved.
3.2.2 Effectiveness of APP Transition
APP Transition Implementation
The implementation of the APP Strategy is ongoing. Challenges in implementing the APP Strategy include obtaining new staff; as well as recognizing that many proposed changes require multiple levels of consultation before implementation.
Since 2009/10 PCH has undertaken and completed numerous activities that have influenced some of the findings identified for the evaluation period 2005/06 through to 2008/09. PCH recently finalized and will be implementing its APP Transformation Plan, June 2010.
The APP Transformation Plan, June 2010 is both strategic as well as operational and the implementation plan moves forward from the original APP Implementation Strategy to identify a schedule of activities to help APP continue to evolve from a collection of small programs into a single program that supports urban Aboriginal Peoples.
Quality of Administration, Information and Reporting
From an administrative and financial control perspective the quality and completeness of APP internal project files has gradually improved during the years following the implementation of APP.32 As stated by one Key Informant, we now have strong useful paper trails for our projects. This is aided by administrative and other standard operating procedures and some standardization between programming elements.
Perception of Recipient Organizations to Program Transition
There were a number of well-documented consultations about the program transition with Third-party and Ultimate Recipient Organizations making the program transition an open process. The size and diversity, and the turnover of the individuals working within these organizations, meant that some organizations or individuals might not have been aware of these consultations. Moreover, based on the sample used for this evaluation, there is still a subset of organizations or individuals who are relatively unaware of the program transition.
The focus of the transition has been on the policy development, the establishment of program criteria and the creation and integration of management and financial controls. This work has been focused primarily in Ottawa; therefore, Third-party and Ultimate Recipient Organizations are less likely to notice any major impacts at the project level. The exception is the ongoing work to improve reporting for those programming elements where PCH has revised the reporting requirements at the project level.
Considerations for Further Operational Effectiveness
Third-party and Ultimate Recipient Organizations commented that operational effectiveness could be improved if PCH funded projects for multiple years rather than single year projects. It would only be necessary during the first year of a multi-year project to expend effort developing project plans and negotiating as well as training new Aboriginal organizations on PCH requirements.
PCH key informants commented that while multi-year agreements may reduce some of the start-up administrative burden, such arrangements would limit access to the program for new projects as most of the funding would be tied up with ongoing projects. In the case of an agreement where a project was not delivering well, it may be more difficult to terminate or adjust an agreement.
PCH staff indicated that working directly with Ultimate Recipient Organizations rather than through Third-party organizations may be another consideration. This type of arrangement may help to address the following:
- Third-party organization(s) do not use PCH criteria for the identification and selection of Ultimate Recipient Organizations;
- Third-party organization(s) reports are not systematically provided to NAFC and PCH since the NAFC provides a single synthesized report; and
- Better information for PCH staff about the purpose and intent of the projects, as well as the planned outputs and expected outcomes, as is the case for the W6 UMAYC that are managed by PCH regional staff and not Third-party organizations.
Consideration could also be given to instances in which Third Party Organizations continue to be involved (for example where a project is implemented at great distance from a PCH regional office). In such cases, PCH could tighten management practices to ensure that the PCH criteria are used for the identification and selection of Ultimate Recipients.
As the APP continues to mature, PCH may consider a formal in-depth examination of the issues related to direct versus Third-party delivery to establish administrative and program effectiveness benefits of each approach.
Challenges to Improving Effectiveness
PCH key informants recognize that government funding is dependent on clear program and project criteria and sound stewardship of the funds provided. That said, key informants generally felt that administration procedures and practices that were too rigid, and that accounting and program criteria were major obstacles to future improvements to increased effectiveness.33
While recognizing the need for controls and the value of improved reporting of results, some key informants commented on the cost of administrative procedures and controls. While they did not provide specific examples, they made general statements that indicated that, for example, that some program admissibility criteria are unnecessarily restrictive and limit the potential of some projects. They also believe that the volume of paperwork to administer projects seems to be too high, which reduces employee effectiveness as well as unnecessarily prolonging the entire project approval.
PCH manages the program in keeping with the Treasury Board Policy on Transfer Payments; recognizing the need to apply risk management principles, sound and effective administration of the APP, sound stewardship of resources, and rigorous performance reporting.
The ongoing work to develop a performance management framework for these programs and the continuing work on implementing APP transformation is an excellent opportunity to reassess and confirm whether:
- existing administrative arrangements are adequate,
- alternative means of delivery may be more effective or efficient (such as direct delivery by PCH in place of Third Party delivery in certain circumstances),
- there are opportunities to apply additional risk management principles, and
- there is an opportunity to institute higher quality or more effective data collections, measurement of results, or reporting.
Information Quality and Reporting Consistency
At the request of PCH, the evaluation included a limited, non-legal, review of the PCH contribution agreement with NAFC for the development of the AFC and UMAYC databases. The focus of the database review was to determine if the databases contained information appropriate for PCH program-level policy and management purposes and included the examination and analysis of two (2) of the National Association of Friendship Centres (NAFC) databases and PCH’s APP reporting database. The database review for NAFC covered all the evaluation period while the database review for APP covered fiscal years 2007/08 and 2008/09, the years for which project information is contained within the APP database.
Data Base Integrity
PCH’s APP Database
PCH staff developed the APP database in 2008 with the expectation that it would contain project results and other information needed to support the policy and program management of the APP. The APP database contained electronic files for 76 (46.3%) of the 164 reviewed project files. After careful examination, only 18 electronic project files (23.7% of 76 files) contained information that is comparable with the paper-based files. More specifically, the other 58 electronic files (76.3% of 76) contained information from for example, the wrong fiscal year, or from other projects. This meant that 76.3% of the electronic files contain data quality issues and should be used with caution.
A side-by-side examination of the 18 higher-quality electronic files and their original paper-based files, along with interviews with PCH staff, revealed that the results reported within the APP database are essentially verbatim copies of the results identified in the annual reports submitted by the Ultimate Recipients. Further comparative analysis between the 18 electronic and paper-based files identified 205 significant errors and omissions, an average of 11.4 errors per electronic file.34 Even with these flaws in the electronic files, the results reported in the annual reports by the Ultimate Recipients and in the database are important. These databases do provide a record of the program activities and outputs (rather than "strategic outcomes" as defined by Treasury Board).
Reporting by recipient organizations is timelier than in the past and Ultimate Recipients and Third-party Organizations more closely follow the prescribed format and content requirements. This timeliness of reporting varies between APP programming elements as reporting requirements are being sequentially developed and implemented for each APP programming element as the adjustment process continues.
NAFC’s AFC Database
NAFC’s AFC database contains forecasted, not actual, activities, revenues and expenditures information about the Aboriginal Friendship Centers (AFCs) for use in program management and decision-making. The APP database contains information for about 25% of the projects. While the information is important and useful for management decision-making, planning and performance reporting, some data quality issues exist. In the sample of projects assessed for this study, evaluators found an average of 11 errors (such as transposition or incorrect field errors) per project; these databases should be used with caution until data quality improves.
Interviews and discussions with AFC staff at various AFCs across Canada indicated that there is often a significant variation between forecasts and actuals. Some AFCs reported that they continue to refine their forecasts based upon what happened the previous year, recognizing that due to uncertainties, variances can sometimes still be significant.
Type of Information Reported
Treasury Board policies and guidelines discuss results as immediate, intermediate and final or ultimate outcomes. Annual end-of-project reports provide information regarding activities, expenditures, and outputs. In some cases, intermediate outcomes for cultural awareness and increased leadership capacity are measured and reported. The resulting database contains useful information about large numbers of projects, delivered through a variety of methods. Improvements to the database to improve data quality and to include more intermediate and final outcomes information would make it even more valuable.
PCH staff expressed dissatisfaction that much of the information they receive is qualitative in nature. The information generally describes the project and the activities that were undertaken. While it is seen as beneficial to have such qualitative information, reports generally lack quantitative information about the outputs and outcomes of the project. Reporting qualitative and quantitative measures would enhance the information available regarding the achievements of the program. Project successes and failures are currently discussed in qualitative terms, and often in terms of personal observations about one or two (2) of the project participants.
Key Informant Discussions about Reporting Results
PCH has defined reporting requirements, developed tools to assist Ultimate Recipient Organizations in producing their annual end of project reports. PCH’s reporting requirements are included in all contribution agreements. One of the requirements is that Ultimate Recipient Organizations report on the results they have achieved.
PCH key informants noted that identifying longer-term outcomes could be feasible at the level of the program and that these should be compiled by PCH at the program level (rather than by funding recipients at the project level):
- PCH could build on the information received from funding recipients and Third Party organizations as an important source of information;
- Project-level reports would continue to measure and report outputs (immediate outcomes) in order to feed into this work by PCH; and
- PCH could augment the project level outputs with additional data collections, literature reviews and documents reviews in order to measure and report on intermediate and final outcomes.
Based on the interviews conducted, a rigorous approach to results measurement and reporting requires agreed measures of project success (results) to be set out clearly in the agreements and then monitoring, measuring and reporting during the course of the year. The agreed measures may include an appropriate mix of activities, expenditures, outputs, intermediate outcomes and if feasible, a contribution to the program's final outcomes.
PCH’s APP Database
The APP database is not an integrated relational database, but rather a number of independent databases created for most of the APP programming components. Each of the databases has a different structure, with varying numbers of data fields, with few of the data fields being common between the different databases.
Since the APP database copies information from the annual project files, the APP reported results are of the nature of activities and some outputs. This information is both valuable and useful, and the department is encouraged to continue work on the APP performance measurement framework to further enhance database value and usefulness. To that end, PCH staff also continue to focus efforts on:
- Developing more specific detailed reporting requirements for inclusion in contribution agreements; and
- Drafting contribution agreements that will tie the final payment(s) to the quality and timeliness of reports submitted by third party and recipient organizations.
While there is a great deal of valuable information available regarding activities, expenditures and outputs, reporting quality and content is not at the desired level. When preparing reports to executive management and other organizations including central agencies, additional, supplementary information is obtained on a case-by-case basis, as the information is not in the APP database. Seeking this supplemental information makes the reporting process longer than would be the case if the information were available in the database. The program is encouraged to conduct a review on the type of additional information needed for these purposes and determine the benefits of including them as part of routine requirements set out in the performance measurement framework and contribution agreements with Third Parties and Ultimate Recipients.
NAFC’s AFC Database
NAFC does not require individual AFCs to report on actual results achieved in their annual reports. When NAFC undertakes the preparation of its annual report, it contacts AFCs to identify success stories. NAFC’s AFC database contains information regarding activities and outputs, but no data regarding actual results achieved. The success stories collected each year are valuable and provide important insights regarding the important benefits of the program.
While there is some good information in the database, key decisions have to take into account the major variances between forecast and actual information. Even with this issue, which can be corrected in the planned ongoing improvements to the database, the database has demonstrated its value. There have been a number of benefits to having the database, such as being useful in the annual APP performance reporting, having information available for the MCs and TB submissions, as well as information for best practices that can be provided to potential applicants.
The information in the AFC database is useful for PCH’s strategic policy and program management purposes to assess Friendship Centres financial needs in terms of core funding. As future versions incorporate the actual results achieved, compared with the forecast results, this database will be even more valuable for policy and program management purposes. Maintaining a picture of results achieved (compared with the forecast) over a period of perhaps five (5) years, would be an additional benefit. Such a time series would show the results achieved over time.
NAFC’s UMAYC Database
During interviews and discussions held with NAFC staff, and a review of some electronic files, it was determined that NAFC’s UMAYC database contains information about large numbers of projects delivered through different methods including data on activities, expenditures and output information. The database does not contain information regarding the longer-term outcomes. The UMAYC database has significant potential value for program administration and decision-making as it contain actual and not forecast information.
The information contained in the UMAYC database comes from the UMAYC project reports that AFCs and other Ultimate Recipient Organizations submit to NAFC. In addition, at the end of each fiscal year, NAFC asks their regional organizations to identify notable UMAYC project success stories. PCH staff commented that UMAYC project funding is not core funding, and that there is an expectation that individual UMAYC projects should report on project results. The NAFCs do not agree with this interpretation and this issues should be clarified for future contribution agreements.
The UMAYC database contains important information regarding actual achievements of these projects, and project details that are useful for decision making, performance reporting, and preparation of Treasury Board Submissions and Memos to Cabinet.
Findings [D2]: Implementation of APP Strategy continues and improvements are noted. Some recipient organizations remain unaware of program transition. There are issues with data quality and reporting consistency.
Continued progress on an up-dated implementation plan should help APP to continue to evolve from a collection of small programs into a program that supports urban Aboriginal Peoples.
The quality of processes supporting administration and financial control is increasing. The quality of reported results has not improved over the period of transition. Improvements are expected as implementation of the program’s performance measurement framework is completed.
Third-party and Ultimate Recipient Organizations were consulted during the transition and these consultations were documented. Turnover of staff may have taken place in some organizations and the new staff may be unaware of the earlier consultations that were conducted.
The focus of the program transition until now is at the national level so it is unlikely that many of the changes would be seen yet at the project level, except for efforts to improve reporting.
Ultimate Recipient Organizations would prefer multi-year project funding agreements rather than single-year funding of projects to reduce the burden of making applications and reporting performance.
Replacing Third-party organizations and delivering contributions directly may have potential. Pilot projects may be used to assess the extent to which this could work well and improve effectiveness of program.
Annual project reports are more consistent in format and are being delivered on a more timely basis. While this information has value and is useful for management and reporting purposes, due to the data quality issues identified in this study, the information should be used with caution by PCH for program and project management.
PCH staff are developing more specific reporting requirements and tightening requirements in contribution agreements to ensure timelier and more consistent reporting.
Availability and Distribution of Program Information
APP programming elements information and funding applications are available on PCH’s website and printed copies of all documents are available from PCH in Ottawa and the Regional offices.
Ultimate Recipient Organizations stated that they were able to find and obtain from PCH staff and website the majority of the information they needed to complete most applications for funding. They stated that there was little information that described how their applications would be assessed.
Funding Allocations by Need and Geographic Area
PCH has not conducted a formal, documented needs analysis for Urban Aboriginal Peoples, and it was not possible to draw that information from project files or annual project reports. The review of the project files, with few exceptions, did not identify that a formal, documented needs analysis had been conducted and used to support project funding applications.
While there is agreement about the need for ongoing dialogue with Aboriginal organizations and Ultimate Recipient Organizations on APP programs, this is not a formal, program-level needs analysis. There has been excellent take up of the program funding and more applications are received than can be funded from available funding. Informal discussions with stakeholders indicate the program continues to address important needs of Aboriginal Peoples.
PCH staff state that they rely on local Aboriginal organizations to identify the needs of urban Aboriginal Peoples, and to design programs that meet the needs of urban Aboriginal Peoples. This input from Ultimate Recipient Organizations is valuable and there are many informal indicators that the program continues to meet real needs. That said, PCH should consider periodic formal and comprehensive program-level needs analysis as an additional input to an ongoing needs assessment process, helping confirm the priorities that have been set.
While there is a general regional distribution of APP projects by population base, there are no specific criteria as to how this is accomplished. Some programming elements are specific to regions such as NAB and the TLAs. Others are national in scope. Funding allocations receive close scrutiny within the department and the department exercises due care in making a funding allocation.
Opportunity, Selection and Contracting Processes
Organizations that receive core funding have been named and approved by the Government; while project funding requires that PCH ensure that the ongoing selection and contracting of Ultimate Recipient Organizations is performed according to standard operating procedures including clear and up-to-date selection criteria.
Prior to 2005, most APP project funding was awarded to the same recipient organizations. Since then three (3) trends have evolved:
- PCH and some Third-party Organizations make a concentrated effort to ensure that different recipient organizations have the opportunity to submit project funding applications.
- With few exceptions, AFC funds (under the core-funding budget) the same AFCs it has funded over the past few years and does not fund new (additional) AFCs. NAFC states there is only sufficient funding to adequately cover those AFCs that have been funded in the past, and, that if new AFCs were added, it would further dilute the level of funding to existing AFCs. As well, NAFC funds essentially the same organizations to deliver UMAYC projects, almost exclusively delivered within AFCs.
- An AFC can apply to other APP programming elements such as ALI.
The key informants of not-previously-funded recipient organizations stated that APP funded projects are generally open to them and that PCH staff has encouraged them to submit funding applications. They also stated that information was readily available on the internet and through PCH offices. It was noted that greatest barrier to getting APP project funding is that they did not have first-hand knowledge gained through working with PCH, and that lack of that specific knowledge made it more difficult for them to submit a winning solution.
Findings [D3]: Program information is readily available. The current approach is practical. A funding allocation based on formal needs analysis is unavailable. Adequate opportunity exists for new ultimate recipients and Aboriginal Friendship Centres.
PCH has not conducted a comprehensive program-level needs analysis. PCH staff rely on Ultimate Recipient Organizations to complete needs analyses relative to their project.
There is no evidence that Ultimate Recipient Organizations have systematically conducted a formal needs analysis for urban Aboriginal Peoples, or a narrow formal needs analysis specific to their proposed projects.
The criteria to determine the distribution of APP projects are not widely known and have not been reviewed in some time. A review of the evaluation criteria may be worthwhile, as part of the ongoing transformation of the APP.
3.2.3 Program Delivery Appropriateness and Effectiveness
PCH relies on the NAFC as Fourth-party deliverer of a portion of the UMAYC to ensure that regional PTA desks issue competitive calls for UMAYC project funding, select UMAYC projects based on merit and provide operational support to Ultimate Recipients. While PCH tends to receive high-level quarterly reports from NAFC, very little information on individual projects is provided in the NAFC’s reports.
In the direct delivery of W6 UMAYC projects, PCH staff do all of the work beginning with identification and evaluation of funding applications, through to signing contribution agreements and following the project to its end. PCH staff with the responsibility for ensuring the delivery of all UMAYC projects are aware of the day-to-day operations and are able to obtain project specific information directly from the recipient organization.
Direct Versus Third-party Delivery
A close examination of this issue indicates the need to consider direct delivery by PCH in place of Third Party delivery is a potential way to improve program efficiency and effectiveness. The table below, presents the views of PCH key informants on the pros and cons of direct delivery by PCH compared to Third Party delivery:
Table 9: Comparison of Direct Delivery by PCH versus Third Party Delivery
Direct Delivery by PCH
| || |
Third Party Delivery
| || |
There was consensus amongst PCH key informants that the strengths of direct delivery versus Third-party delivery are that PCH has direct control over projects and processes employed, with increased access to information.
Findings [D4]: PCH has a high level of involvement in the direct delivery of W6 UMAYC projects. Information is available for PCH Policy and Program-level management. Delivery mechanisms are perceived to be appropriate and achieving desired reach. There is a potential for cost savings from PCH direct delivery versus Third-party delivery.
In addition to PCH and the NAFC, there are also Inuit and Métis third party deliverers of UMAYC.
Annual project reports, as well as the databases contain information regarding activities, outputs and expenditures required by PCH for policy and program-level management.
Due to data quality issues, reported information should be used with caution. Improvements to the database and additional information regarding outputs and outcomes are under consideration for addition in the PCH performance measurement strategy.
92% of key informants found that Third-party and direct delivery mechanisms were either appropriate or very appropriate.
75% of Third-party and Ultimate Recipients key informant organizations stated that current delivery mechanisms are fully or somewhat achieving the desired reach identified for the APP programming elements.
The strengths ranged from the positive impact upon participants and youth though to empowerment and involvement, the projects were well received and the dedication of staff members was noted;
The weaknesses raised were related to funding issues, delays and a desire of funded organizations to be considered for multi-year funding.
A potential improvement in efficiency and effectiveness was identified, namely, having more direct delivery of funding to Ultimate Recipients for projects by PCH. This could reduce administrative costs and improve measurement and reporting of performance. A mixed approach may be appropriate, where PCH delivers directly only in certain geographic areas (for example, proximity to PCH offices), or under certain circumstances (for example, specific types of projects).
The 2008 Audit found additional risks related to multi-year funding and recommended additional risk mitigation strategies to address multi-year funding.
The key strengths of PCH versus Third-party delivery are direct control over the project and increased access to information.
Pilot projects are necessary to estimate the potential cost savings from PCH delivery versus Third-party delivery, and the specific conditions or areas of funding under which there would be maximum savings.
Multi-year funding has pros and cons that must be considered carefully.
- Annual project funding has the advantage of allowing better access to the program from new project proponents.
3.2.4 Reporting System Adequacy to Report Results
Current Reporting Requirements and Implementation
For most APP programming elements, PCH specifies its reporting formats within the contribution agreements as well as providing additional information on its web site and in printed format. Work in completing requirements for all APP programming elements continues and PCH’s commitment to completing the work is included as part of the current Aboriginal Peoples’ Program Transformation Plan (June 2010).
PCH staff state that they are receiving required reports on a more timely basis and that the reports contain all subject headings specified within the report formats. Annual project reports provide adequate financial and qualitative information, and also describe activities and outputs. This information is valuable and of use to manage the program and prepare reports for Central Agencies.
PCH staff state that they make use of the information provided and often need to seek additional information on a case-by-case basis when preparing reports for senior management or Central Agencies. PCH staff would like to see improvements to the quality and content of the information provided within the reports. They would like to see the qualitative data augmented where appropriate with quantitative data. They also state that the reports do not contain information regarding outcomes they could use to support their efforts in policy and program management.
Reporting System Allows for Communicating Results
Table 10 contains responses to the question: Do funded organizations believe that the reporting system allows them to adequately communicate their results?
Table 10: System Allows Reporting of Results
(Third-party and Ultimate Recipient Key Informants)
Very appropriate and effective
Somewhat appropriate and effective
Not appropriate or effective
Did not answer
Table 11 contains responses to the question: Do you have any recommendations to further improve the reported information so it can lead to a better understanding of the programming elements results?
Table 11: Recommendations to Improve Reporting Results
(Third-party and Ultimate Recipient Key Informants)
Improve the reporting and evaluation process
Evaluate (accept) qualitative information
More collaboration between government agencies
Did not answer
An extensive review of the results presented in the annual reports contained in the project files demonstrated that with two (2) exceptions (cultural awareness and increased youth leadership capacity) annual reports listed activities and outputs, not the intermediate and final outcomes achieved. Annual reports identified one or two anecdotal or deduced intermediate or final outcomes.
The review of a sample of the project applications within the project files found that when project applicants identified time and a portion of the budget to evaluate the results of their programs and to report them in their annual reports, PCH staff asked project applicants to reallocate that time and budget to delivering their project.
The decision to fund annual projects means that a dedicated team of individuals is hired to deliver the project. That dedicated team should set up an appropriate regime to measure and report on results at the beginning, at key milestones and at the end of the project. Where a project or a funding commitment extends beyond one year, there should be ongoing reporting each year to provide sufficient support for funding renewal for the ensuing year.
There is a need for rigour and for a systematic approach to performance measurement to meet the contractual obligations of the contribution agreement regarding performance measurement and reporting. At the beginning of the project, there should be agreement on what is to be achieved and how it is to be measured. Interim measurements should be taken and reported at appropriate key milestones. The information needed should be collected in a systematic manner, from the start of the project. If no steps are taken to prepare to meet these obligations until the project has been completed, it is less likely that adequate performance reporting will be completed as required.
The problem is not due to a lack of detailed reporting requirements or tighter clauses in agreements specific to reporting. The issue is to take early and systematic actions at the start of the project to ensure that performance measurement and reporting are undertaken in an orderly manner as part of the normal activities of the project team.
There is a strong consensus amongst Third-party, Ultimate Recipient Key Informants and PCH staff, that Ultimate Recipient Organizations would be able and most likely have the abilities and capacity to report on the outputs of their projects.
There is some consensus that, if the project application process asked potential Ultimate Recipient Organizations to specify what their expected outputs would be, it was likely that the Ultimate Recipient Organizations could report on the output as well as describe any variances between planned and actual outputs.
PCH Perception that System Allows Reporting on Results
PCH staff indicate that the system allows recipient organizations to report, and that recipient organizations have the information, capacity and funded resources needed to report on immediate results. PCH’s perception is that clear reporting requirements must be specified in the contribution agreements; otherwise, some recipient organizations may not report project results. PCH staff is working to better define reporting requirements and to integrate those requirements into contribution agreements. PCH staff has tasked Third-party and Ultimate Recipients to identify results (outputs and immediate, intermediate and/or final outcomes), reasoning that Third-party and recipient organizations are best informed and closest to the projects.
Third-party and Ultimate Recipient Organizations key informants stated that there are two (2) structural problems that hamper comprehensive performance measurement:
- PCH’s requirements that projects are funded for one year only with no follow-up projects means that expected intermediate and final outcomes will not likely be realized within the fiscal year that the project was delivered.
- Project design does not allocate resources to performance measurement.
There was a mixed reaction amongst key informants to the idea that more explicit reporting requirements would result in Ultimate Recipients reporting on results. Some individuals stated that there were many challenges to reporting APP project results, and that making reporting requirements more explicit is not enough to solve the problem.
Use of Information from Annual Reports
PCH key informants indicate the following ways that information provided in the annual project reports is summarized or otherwise used by PCH staff:
- Preparation of documents for submission to the Treasury Board;
- Input information into the APP database;
- Approval of the final invoice submitted by recipient organizations and Third-party Organizations; and
- Occasional inclusion of qualitative information to illustrate success stories.
PCH key informants also stated that the information currently presented in annual project reports is not consistent in either subject matter or structure, and confirmed that intermediate and final outcomes are not reported in the annual project reports.
The information in the APP database, which is based on the annual project reports, provides a rich source of information on activities, immediate outcomes and expenditures that can be used for program management and reporting purposes. Data quality issues require that the information in the database be used with caution.
Findings [D5]: There are noted improvements in timeliness of reporting. Improvements to quality and content are required. Requiring reports on results requires a systematic approach. More explicit reporting requirements will result in recipient organizations providing more useful information as well as reporting on results.
Updated reporting requirements have been completed for most APP programming elements. The information provided in annual project reports is being received in a timelier manner than in past years.
PCH does make use of the information that is provided. Often, additional information needs to be sought on a case-by-case basis when preparing reports for senior management or Central Agencies.
The report format prescribed in the contribution agreements and APP documentation allows for the reporting of results. There is a lack of clarity regarding what immediate, intermediate, or ultimate outcomes are to be measured or reported, and what indicators are expected. Suggested qualitative or quantitative measures are often not clearly identified (or agreed in the contribution agreement or at the start of the project).
Improvements to the quality and content of the information were identified for consideration in the performance measurement framework that is in development.
Increasingly detailed reporting requirements and changes to the contribution agreements as well as the performance measurement framework is expected to be of high value not only to PCH for program management and reporting to Central Agencies, but also to Third-party deliverers and Ultimate Recipients, in terms of meeting their contractual obligations for performance measurement and reporting.
Where an Ultimate Recipient receives funding for a project, that Recipient must organize and plan to collect the necessary data to measure and report results achievement, and implement this plan during the course of the project.
Requiring a recipient to provide its plan for performance measurement and reporting, including the qualitative and quantitative indicators to be used, and including providing interim reports at key milestones are ideas worth considering in the performance measurement framework now in development at PCH.
Performance [P] is the third of four evaluation questions addressed. Treasury Board defines performance as: What a government did with its resources to achieve its results, how well those results compare to what the government intended to achieve and how well lessons learned have been identified.36 This section presents the findings related to six (6) performance-specific evaluation questions (Appendix A).
As recognized in the methodological limitations for this evaluation, determining the success of the APP as a consolidated entity is a challenge. While the evaluation was able to conclude, based on existing evidence and reflection, with regard to the implementation aspects of the APP; establishing a similar base of evidence for the outcomes of the consolidated APP will require more time and opportunity to acquire outcome-focussed information based on reliable and valid sources of data.
For this reason, the following sections have relied largely on the results of previous evaluation work conducted on the program components that now make up or are similar to parts of the APP. Moreover, while most of the work pre-dates the current program; the expectation is that in continuing to deliver these components in a consistent manner, the APP also continues to realize the positive outcomes reported.
3.3.1 Aboriginal Communities Outcomes
Aboriginal Communities Contribution to Outcomes
For the Aboriginal Communities (AC) component, the APP logic model identified the following immediate and intermediate outcomes:
- Access by a network of Aboriginal Friendship Centres to other programs and activities that support urban Aboriginal communities.
- Access by Aboriginal women and youth to culturally relevant programs and activities that improve skills, knowledge and leadership.
- A network of Aboriginal Friendship Centres that has the capacity to support urban Aboriginal communities.
- Aboriginal women and youth improve their skills and leadership and gain cultural knowledge.
NAFC and AFC annual reports identify the number of programs run within AFCs, the dollar value of those programs, which are mostly in health and wellness, and the total number of urban Aboriginal Persons that participate in those programs. While it is clear that urban Aboriginal Persons visit and make use of the AFCs and the programs delivered in their facilities; no reliable quantitative information is available.
This lack of NAFC and AFC data can be remedied in future contribution agreements by requiring performance information that reports actual results compared to forecasts. PCH should consider making such data a requirement in future contribution agreements. This should be considered within the performance measurement framework for the program.
Skills, Knowledge and LeadershipWhen addressing results, best practices and lessons learned with the Ultimate Recipients, information about lessons learned mostly included ways to improve working with community members and reach out to youth. There was only limited additional information, measureable or observed, that would assist in identifying other results and additional best practices.37 In summary, the Ultimate Recipients contacted for the case studies identified valuable activities and some project outputs, as well as lessons learned about ways to improve working with community members and reach out to the youth.
Table 12: Areas Where Improvements Were Identified by Key Informants
Youth employment skills
Youth involvement and capacity building
Enhanced youth leadership skills
Did not answer
The evaluation reviewed the project files, PCH documents and discussed gender equity with Ultimate Recipients and PCH staff. We found that AAP projects are not gender biased; however, some funded projects have a gender focus. Examples include:
- The women’s programs participants were almost all women except for the occasional male guest speaker;
- Some UMAYC projects targeted young women and family violence while other projects targeted young men at risk; and
- Some culture projects focussed on giving birth to and raising children in a culturally appropriate manner. Men were invited but the focus was on women and children.
Findings [P1]: Aboriginal Communities Contribution to Outcomes. Some outcomes (expected results) were achieved
There is a network of AFCs that are available to support urban Aboriginal Persons, but the level and degree of support could not be determined.
Aboriginal women and youth have access to culturally relevant programs and activities that have improved their skills, knowledge and leadership abilities.
3.3.2 Aboriginal Living Cultures Outcomes
The second performance evaluation question(degree to which the Aboriginal Living Cultures program component within the APP has achieved its immediate and intermediate outcomes) is addressed by three (3) evaluation sub-questions (as listed in Appendix A).
Aboriginal Living Cultures Contribution to Outcomes
For the Aboriginal Living Cultures component, the APP logic model identified the following immediate and intermediate outcomes:
- Access to programs and activities that support Aboriginal communities in the preservation, revitalization and promotion of Aboriginal languages and cultures.
- Production and distribution of Aboriginal radio and television programming that support the preservation, revitalization, and promotion of Aboriginal languages and culture.
- Aboriginal communities have increased capacity to enhance Aboriginal languages and culture.
- Aboriginal achievements and contributions are recognized and appreciated in Canadian society.
- Aboriginal communities watch and/or listen to Aboriginal television and radio programming.
Pre-2005 evaluations were completed on NAB, ALI, the Northern Broadcasting programs (NNBAP & NDP), and the TLAs in the 3 territories. Since 2005 evaluations have been completed on: National Aboriginal Day, the Celebration, Commemoration and Learning Program38 (which include comparative data on NAD), the Government of Canada Endowment Fund, the NAAF, and the NWT Languages programs.
The 2006 report on the NAD and the 2007 Evaluation of the Celebration, Commemoration and Learning Program (CCLP) both included information of the awareness of and participation in NAD, and indicate some success in achieving the intended outcomes. The NAD 2006 report indicates that one (1) in three (3) Canadians are aware of NAD and, despite low participation by Canadians in NAD, the large majority of Canadians (69%) and of Aboriginal Peoples (up to 85%) support government funding for the NAD.
The Government of the Northwest Territories commissioned an evaluation of its Canada-NWT TLA in 2009 and found that there was no coherent plan for Aboriginal languages revitalization in the NWT, and that funding was provided on an ‘ad hoc’, project by project basis. Specifically: “The objectives of the Aboriginal component of the Agreement are long-term objectives, and these have not been met in ways that demonstrate the revitalization of Aboriginal languages in the NWT. However, many individual projects undertaken by language communities have achieved short-term objectives.” (Evaluation of Appendix A of the Canada-NWT Cooperation Agreement for French and Aboriginal Languages, 2009, Page 5). The evaluation report noted that it was not possible to evaluate the efficiency or effectiveness of the use of resources ‘because no evaluation criteria were established prior to project operations.’39
The TLA in the Yukon and Nunavut were last evaluated in 2004 and 2005 respectively. Findings of the 2005 Evaluation of the Nunavut Agreement echo the conclusion of the 2009 NWT Evaluation regarding the difficulties of measuring ‘outcomes’ or results, stating that: “As there is no evaluation framework and few performance indicators for the Agreement, the description of benefits is based largely on the perceptions of key informants.” (Page 10)
While key informants noted benefits at the community (and project) level, the report states: “The Agreement is not sufficient to address the larger picture. All informants feel that the level of funding and resources available under the Agreement is insufficient to meet the need, including funding for territorial government initiatives and services.” (Final Report on the Evaluation of the Canada-Nunavut Co-operation Agreement, 2005, Page 10)
It is noted that “Inuktitut is one of the few relatively strong Aboriginal languages in Canada, but is losing ground. Unless a more comprehensive and concerted effort is made by all levels of government, this trend will continue.” (Page 13) Further research would be required to assess if the trend has continued and to what extent any improvements have been achieved since 2005 through APP programming to preserve and/or revitalize the Inuktitut language.The 2004 evaluation of the Yukon TLA reported that 229 projects had been funded from 1998 to 2003, benefitting 8 language groups and every Yukon community.40 The Report concluded that: “It is difficult to imagine what state the languages would be in today and what would be their prospects for the future had there been no funding under this agreement from 1998-2003.” (Evaluation Report Hope for the Future – A Call for Strategic Action, Canada-Yukon Agreement for Aboriginal Languages, 2004, Page 3.)
The Report states that many valuable First Nations directed projects had been undertaken, and some had been very successful in impacting language revitalization whereas others had had limited success. The Report calls for a more strategic approach, ‘a unified vision’, focusing efforts where the potential impact is greatest.
Overall, despite reports of success at the individual ‘project activity level’, evaluations related to the TLAs question the achievement of the APP expected outcomes with respect to languages in the northern territories under the TLAs, and identify the need for more strategic approaches with measurable objectives. In other words, although the activities could be expected to contribute to the APP outcomes, it is difficult to measure this contribution.The 2003 ALI Evaluation for the years 1998-2002 assessed the contribution of about 1200 community projects for Aboriginal language acquisition and retention in the home. The evaluation stated that, during these four (4) years, the ALI had funded many community projects that would more likely have taken place otherwise.41 This finding suggests that ALI had an incremental effect, with more projects and more communities involved in language projects (although the Report noted the lack of baseline data). As well, the Report noted that the ALI had supported the development of long-term strategies to revitalize and maintain Aboriginal languages. However, the expected long-term outcome of preserving and revitalizing Aboriginal languages ‘will take considerable time and more funds than are now available through the program.’42
In the interim, the Report stated that ALI projects had contributed to interim steps such as increased interest in and awareness of languages among community leaders. These findings relate to the immediate expected outcomes of the APP (that is, creating access to language programs), although there are no direct measures of the intermediate and longer-term outcomes in the APP logic model. Additional research would be required to assess the contribution of the ALI since 2002.
With respect to National Aboriginal Broadcasting, the 2003 NNBAP/NDP evaluation found that: “The broadcast production and distribution supported by the programs has been extremely successful. Aboriginal languages and cultures are being documented and transmitted through media, several hundred Aboriginal broadcasters and ex-broadcasters have developed the skills and knowledge to play leading roles in their communities, regions and nationally, and news and information are readily available to the Aboriginal public through their own media. The reach of Aboriginal radio and television has been extended far beyond the north.” (Northern Native Broadcast Access Program (NNBAP) & Northern Distribution Program (NDP) Evaluation Final Report, 2003, Page 6)
The Report goes on to note that the 20 years of programming had developed ‘an invaluable collection of historical and cultural material’, much of it in Aboriginal languages, and that ‘this is an irreplaceable resource for Aboriginal peoples and for Canada.’ (Ibid.) Another positive result noted was the fostering of independent producers, writers and artists who have become role models, and many have gone on to distinguished careers in leadership positions. This evaluation raised concerns about the adequacy of funding formulas to sustain programming and equipment, and the inability of northern broadcasting to become financially self-sufficient due to lack of other revenue sources.
Visibility and Recognition of Aboriginal Achievements
There are no questions or answers that provide direct or indirect evidence to ascertain the extent that the APP increases visibility and recognition of Aboriginal achievements throughout Canada. As determined by accessible literature, this is evident through the annual expenditures and details on transfer payment programs for the department.43
PRA Inc. Research & Consulting (2008) echo a similar sentiment where determining the effectiveness of policy integration relating to language, culture, and urban programming in Canada is difficult because the level of integration is unknown. This position is supported by the discussion of the “extent and structures” resulting in the statement of: “A large portion of programming for Aboriginal languages, culture, and urban issues in Canada falls under the newly formed APP” and continues to note the goals and departmental administration.44
However, there are findings related to Canadians’ perceptions of contribution to Canadian society, understanding of issues and people, performance of federal government, and engagement in events. The 2004 study by Ipsos-Reid found overall positive public views regarding Aboriginal culture and its contribution to Canadian society. Four (4) in ten (10), especially from the North and West, feel they have a good understanding of Aboriginal Peoples and their culture with the same amount thinking a good job is done by the federal government in the handling of Aboriginal issues.
Almost 80 % of Canadians would participate in more Aboriginal cultural or heritage events if there was more awareness of: location, inclusion of non-Aboriginals, and what events are available (p.34). Additionally, internal audits of both the AC and ALC components indicate annual expenditures that indicate a level of financial support, totalling $50.7M and $20.0M respectively. Again, these reports do not reflect the exact nature activity or outcome that might provide valuable information.
The 2009 Horizontal Evaluation of the National Aboriginal Achievement Foundation (NAAF)45 found that the NAAF: “…has increased the visibility and profile of Aboriginal Peoples and culture within Canadian society – a much needed outcome. Overall, NAAF has certainly had a significant positive impact on the Canadian Aboriginal population, especially those who have had direct contact with NAAF’s programming. In general, NAAF program participants hold a high degree of satisfaction with the programs they use. This is a significant finding, as not only does NAAF do well to achieve program outcomes, it does so in a manner that is accepted and celebrated within the Aboriginal community.” (Horizontal Evaluation of the NAAF, PCH, November 2009, pp.iii-iv)
The activity most directly related to this evaluation question is the National Aboriginal Achievement Awards which include an annual, gala event with roughly 2,500 participants, and the ceremonies are broadcast by the Aboriginal Peoples Television Network and Global Television in a 90-minute special. Estimated audiences range from 100,200 to 179,100.46
Other NAAF activities include: the Education Program which has provided financial support for post-secondary education of over 8,400 Aboriginal students;47 Blueprint For the Future (BFF) which sponsors career fairs to inform Aboriginal students on career opportunities; and, Taking Pulse, which works to connect Aboriginal youth with careers in various industries.48
In the period 2005-10, federal funding (through PCH, INAC and Health Canada) was provided through contribution agreements with average funding of $1.5 million annually (19% of the total NAAF revenues.)49 Federal funding plays a major role in the Education Program (accounting for 75% of funding in 2008/09), and in BFF (about a third of funding). NAAF is a recipient of funding from the Government of Canada Trust Fund (also called the Endowment Fund). PCH made two (2) payments to this fund: in 2003, funding of $12 million was provided and in 2007, funding of $10 million was provided. This fund was created to establish an Aboriginal post-secondary scholarship program.50
Among the conclusions from the evaluation, it was noted that the reach of NAAF services to populations in the North and to French-speaking Aboriginal Peoples required improvement to justify federal government funding.51
While the overall findings of positive successes for the NAAF are clearly identified in this evaluation report, the focus of the evaluation was on the achievements of the outcomes in the NAAF program logic model. Federal funding accounts for only a portion of NAAF funding, and PCH is one of three (3) federal departments contributing funding, and the evaluation did not seek to assess the contribution of PCH’s funding to the overall outcomes of the NAAF or to the outcomes of APP.
The Canada Endowment Fund Evaluation (2008) specifically examined the funding agreement between the NAAF and PCH that has invested $22 million since 2003 (namely, $12 million in 2003 and additional $10 million in 2007 from the CEF). The evaluation assessed achievement of three (3) outcomes, namely: access to improved skills and knowledge, cultural identity, and youth skills, knowledge and leadership. It found that the contributions from the CEF were achieving these results.52
More than half of the scholarship beneficiaries said that the funding was essential in their decision to pursue post-secondary education.53 Therefore, there were data to attribute outcomes to the funding provided. High satisfaction rates (over 90%) were reported, and 79% said the funding helped enhance their cultural identity (such as by providing money to help them return home for ceremonies and family events).54
Other findings also note the important contribution for female students in particular, with three (3) times more awards for women than for men. This finding is consistent with the higher proportions of female than male Aboriginal students undertaking post-secondary education, and, the report notes that female students tend to have more difficulty obtaining funding from other sources.55 Therefore, this funding also relates to other APP outcomes in terms of improving the skills and leadership of Aboriginal women and youth.
Canadians’ Perception of Aboriginal Peoples
The Scholarly literature regarding the relationship between APP and Canadians’ perception of Aboriginal Peoples is virtually non-existent. However, two (2) national public opinion studies provide a picture of Canadians’ perception regarding various issues. In 2004, Ipsos-Reid conducted telephone interviews with 3,600 Canadian adults to focus on key issues and for comparison with earlier studies. The final report organized the results into nine (9) core themes: awareness and knowledge of Aboriginal issues; perceptions of Aboriginal Peoples; views on Aboriginal culture; views on racism and discrimination against Aboriginal Peoples; views on land claims; views on self-government; views on treaty rights; priorities and satisfaction with government performance; and confidence and concerns.
The theme of perceptions of Aboriginal People is further divided into the following sections: situation of Aboriginal Peoples; attitudes toward Aboriginal Peoples; Aboriginal and non-Aboriginal relations and discrimination; participation in Aboriginal cultural and heritage events; views on Aboriginal culture and integration; and government investment in Aboriginal culture. Generally, Canadians feel that Aboriginal Peoples do not compare well with the general population. The majority of respondants (63%) contend that the standard of living of Aboriginal Peoples is worse, even though 36% think the overall situation has improved drastically.
The situation of Aboriginal Peoples is thought to be worse for those living on-reserve compared to those living in cities, a result similar to the findings of INAC (2004). The attitudes toward Aboriginal Peoples were generated through questions concerning current and changing sympathy. Three (3) in five (5) Canadians reported being sympathetic, with INAC reporting that sympathy is greatest amongst residents in British Columbia, Ontario, Yukon and the Northwest Territories and least in Quebec. Although there has been a downward trend regarding increased sympathy since 1996, this finding is counter-balanced with an increase (62% from 50% in 1998) in those who report their sympathy has remained the same during the last two (2) years.
In terms of the relations between Aboriginal and non-Aboriginal peoples, just over half of those surveyed believe relations have remained the same, and almost six (6) in ten (10) believe Aboriginal Peoples experience discrimination. Despite only two (2) in five (5) Canadians reporting a good understanding of Aboriginal Peoples and their contribution to Canada, two-thirds concur that as the first peoples of this land, Aboriginal Peoples have a unique status in Canada. Residents of the Yukon, Northwest Territories, and the West are most likely to have participated in Aboriginal cultural and heritage events. Participation is more likely to occur while on a short regional trip (63%) due to a general interest in Aboriginal culture and heritage (45%). The views on Aboriginal culture and integration are quite positive with the majority (77%) believing a great deal can be learned from Aboriginal culture and heritage.
Although just over half feel that Aboriginal languages and artistic expression are threatened, almost three-quarters agree that Aboriginal culture, languages and artistic expression contribute to Canadian society, particularly domestic and international tourism (64%). Thus, the majority (64%) agree with government investments to safeguard Aboriginal cultures, languages and artistic expression for future generations. Sixty-eight percent (68%)of Canadians agree that it is important for the Government to be respectful of Aboriginal Peopless’ cultures and languages when helping them participate fully in Canadian life. At least six (6) in ten (10) agree that empowering Aboriginal Peoples to strengthen their cultures and languages will assist in playing more productive roles in society, and that government investments are important for the maintenance of a strong Aboriginal presence in Canadian society.
When questioned about preferred priorities for federal and provincial governments, there has been a steady decline in support for fostering more respect for Aboriginal culture, language, and heritage (35% down from 48% in 1998), versus the other two (2) priorities of land claim settlement and Aboriginal self-government.
Another report, The Landscape: Public Opinion on Aboriginal and Northern Issues, was released by INAC in 2004. The study gathered views and perspectives of diverse audiences, including the general public along with Northern residents, youth, and First Nations people living on- and off-reserve. In addition to the similar findings as noted above, a follow-up question was asked of Northerners regarding language and culture in media. The majority (78%) thought that it was important for Aboriginal languages and culture to be promoted on television, radio and the Internet.
Findings [P2]: Aboriginal Living Cultures Contribution to Outcomes. Increased visibility and recognition. Increased Canadian awareness and sympathy for Aboriginal Peoples.
Production and distribution of Aboriginal radio and television programming has been successful. Aboriginal communities watch and/or listen to Aboriginal television and radio programming.
Aboriginal communities have increased capacity to enhance Aboriginal languages and culture.
Aboriginal achievements and contributions are recognized and appreciated by many in Canadian society
The extent to which APP has helped to preserve, revitalize and promote Aboriginal languages and cultures could not be determined.
Organizations delivering APP programs report that visibility and recognition has increased.
There is an increased Canadian awareness and sympathy for Aboriginal Peoples but the information could not be segregated for urban Aboriginal Peoples, and it could not be determined if and to what extent the APP contributed.
3.3.3 Unintended Impacts, Positive or Negative
Unintended Impacts from APP Activities
Discussions with key informants and the project file review did not identify any measurable or observable unintended impacts, either positive or negative.
There are some indications that some programming elements have effects that are not specifically identified in the APP logic model expected outcomes. For example, the NAAF Evaluation noted that federal funding provides a stable base of core funding that enhances the ability to leverage funding from other sources, principally in that it provides some assurance of stability and accountability for its activities. Similarly, the broadcasting funding elements appear to have contributed to developing a valuable resource of historical and cultural information that could have other benefits to Aboriginal Peoples and Canada.
Financial supports for post-secondary education (scholarships) have the potential to impact educational attainment and social well-being with wide-ranging benefits to Aboriginal Peoples and communities. In addition, the wider ranging impacts of languages initiatives (such as those related to education, early childhood development, or economic well-being) have not been examined in past evaluations.
Unintended Impacts from Implementation Strategy
Discussions with key informants and PCH staff and the project file review did not identify any significantly measurable or observable unintended impacts, either positive or negative from the implementation of APP Strategy. There was a general agreement that delayed funding does have a negative impact on project operations.
Findings [P3]: Some project unintended impact. Almost no unintended impacts
There is no evidence that APP activities have had any unintended impacts, either positive or negative at the program level.
Some projects have had unintended positive results on youth leadership and cultural awareness.
Delayed, reduced or cut funding does have a negative impact on project operations.
3.3.4 Benchmarking APP Administration Costs
A summary analysis was undertaken to determine how APP’s administrative cost ratio compares to (benchmark against) PCH as a whole, and for each of the larger program areas within PCH. While administrative O&M could be identified, the true O&M costs could not. Benchmarking could not be established.
3.3.5 APP Management or Administration Overlaps
Many of the reviewed evaluations were undertaken prior to the 2005 consolidation of the APP. A key objective of the consolidation (as discussed earlier) was to improve administrative efficiencies, and a considerable amount of work has been undertaken through the Implementation Strategy since 2005. Therefore, results of earlier evaluations would not necessarily reflect the current administrative and operational environment or its effect on the workload of PCH staff. Therefore, this review focuses on evaluations related to the APP since 2005.
The territorial governments are responsible for conducting evaluation of their respective TLA with Canada, as they are responsible for delivering the program. The responsibilities of PCH are limited to concluding the terms of the agreement (on behalf of the Government of Canada), administering the transfer of financial resources, and reviewing the reports and evaluations submitted by the respective territorial governments.
Although the 2009 NWT evaluation of its TLA did consider the administration of the Agreement and provide recommendation to improve the delivery of project funding, the focus was on administrative and operational factors affecting the staff of the GNWT in the Department of Education, Culture and Employment which is responsible for program delivery. The evaluation did propose development of guidelines and criteria for project funding that could be used to evaluation of efficiency and effectiveness and, which, if adopted, could facilitate the annual reporting to the federal government within 6 months of each year end to ensure continuity of funding from year to year.
The NAAF Evaluation (2009) was undertaken to fulfill the requirements of federal funding to the NAAF by three (3) departments (PCH, INAC, and Health Canada). It focuses on the Foundation’s programs and operations and does not include assessment of the effects of current practices and policies on the workload of PCH staff. However, some of the recommendations arising from this evaluation do have implications for government staff involved in overseeing the funding provided to the NAAF. Specifically, Recommendation 3 suggests the need to create a ‘single set’ of reporting requirements for the 3 federal departments concerned: The federal government should create a single set of reporting requirements for NAAF funding. This will allow the federal government to stipulate a comprehensive set of performance data for review and analysis. Accordingly, it will eliminate the multiple, often competing, reporting demands placed on NAAF and improve the efficiency of the resources NAAF devotes to the work. ( Horizontal Evaluation of the National Aboriginal Achievement Foundation, PCH, 2009, Page 41)
The overall conclusion of the 2009 evaluation on these areas was as follows: Findings from the evaluation suggest that the Program needs to improve both efficiency and effectiveness of its use of resources. For example, findings indicate that the Program should make more efficient use of staff time in managing grants and contributions and take steps to ensure the more timely delivery of materials for funded events and activities. (Page v)
- NAD 2006: The Report prepared for INAC covers awareness and participation information. It does not include information on program delivery, costing, or staffing matters.
- UMAYC National Youth Conference Report (2009): This Report does not include information on program delivery, costing, or staffing matters.
- Evaluation of the Government of Canada Endowment Fund (2008): This report deals with NAAF activities under the Conditional Agreement between NAAF and the Government of Canada as represented by PCH. It does not examine the administrative and operational efficiencies relate to PCH staff.
3.3.6 Budget Review within Document Review
For organizations receiving core funding, or the equivalent of core funding, their budgets have remained relatively constant since 2005/06. Some Ultimate Recipients state that their budgets have not increased in more than ten (10) years. The result is that project managers must keep their projects operational within a framework of inflationary cost increases.
Findings [P4]: Benchmarking could not be established.
Findings [P5]: Administrative practices in place. Limited information.
Administrative practices have been and are continuing to be developed and some have been completed and implemented, with reporting requirements and program criteria being the most significant.There is limited information in the post-2005 evaluation reports to assess this question as they generally deal with other organizations involved in program delivery and do not address internal, PCH administration related to the specific APP program areas.
3.4 Cost Effectiveness and Alternatives
3.4.1 APP Duplication or Complementarities
APP Elements Complement Other Elements
PCH key informants, both in the national and regional offices, generally agreed that the projects were complementary to each other. About half stated that there was room to increase the degree that APP elements complemented each other – while ensuring that duplication is avoided.
PCH Key informants in headquarters office stated that program criteria were being revised and samples of the ongoing changes were provided. A key focus of their review is to increase the complementarity of the APP elements as the APP continues to move from a series of separate and distinct elements to a single coherent program.
The project file review identified that there were commonalities between the APP elements as they addressed common themes such as culture, language, broadcasting and youth.
Duplication and Gaps Between Programming Elements
No duplication of objectives and/or program criteria was identified between the APP programming elements. No gaps were identified during the project files and database reviews.
PCH key informants in headquarters noted that ALI and the TLAs both addressed Aboriginal languages. ALI agreements are project-related and the TLAs are government-to-government agreements, and it would make sense to keep the two (2) separate, while recognizing that both address Aboriginal languages in different ways.
PCH staff did not identify any significant gaps between the programming elements or APP programming elements and PCH’s mandate as a whole. Some minor gaps do exist that can be addressed within the existing APP programming elements.
PCH staff indicated that they were approached with interesting concepts for projects that seemed worthy of consideration and funding and fell outside of the scope of the APP program. PCH key informants in headquarters noted that the ongoing 2010 internal program transition is examining all programming elements with an eye to ensuring that the APP is an integrated program with neither gaps nor duplication.
No project duplication was identified during the project file and database reviews.
Findings [C1]: APP Elements Complement Each Other. No Duplication and gaps Between Programming Elements
There are areas of complementarity between the APP elements.
PCH is working to build on the existing complementarities as work continues to move from separate programming elements towards a more integrated APP.
No duplication of objectives and/or program criteria were identified between APP programming elements.
No significant gaps were identified between APP and its programming elements.
3.4.2 Most Effective Way to Achieve Expected Results
The review compared APP with the First Nations Inuit Health Branch’s (FNIHB) of Health Canada Cluster of Children and Youth Programs (CYD). Presented in table 13 are the different key features between the two (2) programs.
Table 13: Key Differences Between programs
Aboriginal Peoples Program
Children & Youth Cluster
Federal Department & Branch
Date of consolidation
Number of programs consolidated
Date of last evaluation
Program target groups
Urban Aboriginal Peoples (First Nations, Métis & Inuit primarily living off-reserve). Includes 3 territories. Special focus on youth, women, & communities
First Nations (on-reserve) & Inuit peoples (living in Inuit communities)
Culture & language
Child health & wellness
Annual program budgets
Community-based program delivery using Contribution Agreements and Transfer Agreements
3rd party agencies receive operating funding & deliver project funding.
4 funding models used (varies by First Nations). Limited use of project funding (in some elements & for program development)
Community project proposals
Programming priorities determined by communities
Based on existing information, it appears that the consolidated APP is not aligned with the existing program delivery infrastructure and funding mechanisms, and major changes are required in this infrastructure in delivering the APP as a consolidated program. This is more challenging than in the case of the CY Cluster.
Based on the information available and the current scope of the APP, it appears that it will be more challenging for the APP than for the comparison program to achieve a more cost-effective delivery mechanism.
The comparison case study indicated that implementation of a consolidated program is a lengthy process and that the time frame is longer than a three-year implementation strategy because of the need for changes to existing funding agreements with other parties involved. The comparison study showed that the APP is in a similar phase of the transition to the CY Cluster in terms of adapting the management structures.
The comparison study showed that program consolidation requires changes in the internal management structures of the departments concerned. It also showed that this adaptation of staffing structures needs to be coordinated with the shifts in the program funding arrangements with third parties, which can several years to implement. No information is available at this time to assess how APP staffing organization is being aligned to the new consolidated APP structure. The program management structures are being adapted to the delivery of an integrated program, and work is ongoing.
Reporting processes for consolidated programs and to allow for measurement of expected outcomes need to be implemented at the appropriate level with the recipient organizations. The comparison case study showed that implementation of revised reporting template needs to be coordinated with renewal or revision to funding agreements. In the comparison case, this process was beginning in year 5 after the program integration.
The consolidated APP has a broader scope than the comparison case study, which suggests that the adaptation of management structures could be more challenging and that a lengthier time period may be involved.
APP is not yet a fully integrated program as there remain a number of tools that are programming element-specific and not applicable or useable for the entire program. For example, the APP database is in reality is a number of independent non-related databases, each designed for specific programming elements. These separate databases are not capable of sharing information, and as noted, the information contained in those databases contains data quality issues that need to be addressed.
Findings [C2]: Changes to the structure
The APP and FNIHB’s CYD Cluster are somewhat comparable.
No other comparable federally funded programs for Aboriginal Peoples were identified.
The necessary adaptations of the APP program management structures continues.
3.4.3 Cost-effectiveness Changes Required for APP
Opportunities within PCH to Improve Cost Effectiveness
About half of PCH staff commented that effective communications with other organizations within PCH would streamline PCH operations by reducing time spent on such items as processing invoices.
The same proportion of PCH staff acknowledged that the addition of new administrative and management controls solved many of the pre-2005 weaknesses. They also indicated that they felt that the controls and paperwork were excessive, potentially reducing cost effectiveness.
Two (2) respondents commented on their perceived lack of consultation with other funding agencies, both within and outside of PCH. It was stated that there is the possibility for increased linkages and leveraging of funding with other organizations.
PCH regional staff stated that small regional projects should be approved at the regional level rather than requiring approval at headquarters. They noted that requiring headquarters to approve all projects was time consuming, required additional paper work, delayed the start of projects. Other departments follow the practice of projects approvals by regional offices.
As part of its new APP Transformation Plan of June 2010, PCH is examining internal administrative and financial controls to identify areas where effectiveness might be increased.
Findings [C3.1]: There Are Opportunities to Improve Cost Effectiveness
Efficiencies could be realized by delegating more authority to APP to sign off small projects and communications with other branches within PCH.
There is potential to increase linkages and leveraging funding with other funding organizations.
- PCH is currently examining its internal administrative, financial controls and current project funding trends to see if there are areas where effectiveness may be increased.
3.4.4 Potential for Transfer of APP for Cost-effectiveness
The evaluation questions on Relevance and Rationale confirmed that APP and its programming elements were within PHC’s mandate, and that the primary focus was urban Aboriginal Peoples and not those residing on reserves or Inuit hamlets. It was also noted that:
- If project and event funding mechanisms remained the same, the type of work and level of effort performed would remain essentially unchanged if it were undertaken outside of PCH.
- PCH is the government identified lead department for culture and heritage. With APP remaining in PCH it would be assured that the PCH would ensure that the focus would remain as designed.
Key informants did not identify any advantages that might be realized by transferring APP to another government organization. That said, two (2) Third-party Organizations thought that the APP should be transferred to them and INAC was identified by a number of individuals as a potential home department for the APP program. Concerns were expressed that INAC would blend the APP funding into other priorities and that the cultural and heritage aspects would no longer maintain their priority.
Findings [C4]: ] No compelling reason to transfer elements. No advantage in transferring APP
While any part or the whole of APP could be transferred to another government organization, there is no compelling reason to do so.
No advantages would be gained or barriers eliminated by transferring APP to another government organization.
PCH is the best location for the program given its current structure and design. The program is working well at PCH and is fully consistent with the departmental mandate.
13 Canadian Heritage, 2007-2008 Departmental Performance Report for the period ending March 31, 2008, Page 10.
14 Canadian Heritage 2009-2010 Report on Plans and Priorities
15 Op. cit, Page 3.
16 In October 2008, the responsibilities related to multiculturalism were transferred from the Department of Canadian Heritage to Citizenship and Immigration Canada.
18 Unless otherwise note, this section based on: Statistics Canada, Aboriginal Peoples in Canada in 2006: Inuit, Métis and First Nations, 2006 Census, http://www12.statcan.ca/english/census06/analysis/aboriginal/index.cfm (link current as of December, 2010.)
19 Statistics Canada. Catalogue no. 91-547-XIE. Projections of the Aboriginal populations, Canada, provinces and territories. 2001 to 2017. Ottawa: June 2005. http://www.statcan.ca/english/freepub/91-547-XIE/91-547-XIE2005001.pdf
20 Language vitality can be determined through the examination of mother tongue and actual use. A discrepancy indicates language shift, where the language cannot be passed to younger generations as it is not spoken within the home. See Royal Commission on Aboriginal Peoples. (1996). Final Report of the Royal Commission on Aboriginal Peoples. Ottawa, Ontario.
21Because urban First Nation, Inuit and Métis people do not use language at home, intergenerational transmission is difficult, thus elders urged for urban language programs amongst immersion and bilingual schools and cultural camps. Towards a New Beginning: A Foundation Report for a Strategy to Revitalize First Nation, Inuit and Métis Languages and Cultures (Catalogue No. CH4-96/2005). Report to the Minister of Canadian Heritage by the Task Force on Aboriginal Languages and Cultures.
22 EKOS Research Associates Inc., 2006; ShelleyTulloch, Inuit Youth: The Future of Inuktitut. Proceedings of the 14th Inuit Studies Conference, Calgary, August 2004.
23 Interviews with 37 Inuit youth between the 18 and 25 years of age were conducted in three communities to examine the role of Inuit youth in determining the future of Inuktitut; Tulloch,298
24 A census metropolitan area (CMA) is an area consisting of one or more neighbouring municipalities situated around a major urban core and must have a total population exceeding 100,000, of which half live in the urban core.
25 Canada Council for the Arts website (Aboriginal Arts Research).
27 Northwest Territories Education, Culture and Employment. (n.d.). Official Languages of the Northwest Territories. Yellowknife, NT: Author. Retrieved on March 4, 2010 from http://www.ece.gov.nt.ca/PDF_File/Official%20Language/024-Official%20Languages%20Map-web.pdf
28 UNESCO (2010). Atlas of the World’s Endangered Languages. 3rd Edn., Paris. UNESCO Publishing.
29 Council of Yukon First Nations, http://www.cyfn.ca/nathist15?noCache=738:1269422671; Ibid, Statistics
30 Council of Yukon First Nations, http://www.cyfn.ca/nathist15?noCache=738:1269422671; Ibid, Statistics
31 The review included previous program evaluation and audit findings, a literature review and targeted research with an environmental scan, and feedback from four consultations with Aboriginal Peoples in focus groups on program restructuring and renewal.
32 The methodologies employed for the evaluation did not include a measurement scale that could measure the quality of information in a quantitative manner. Quality of information was subjectively determined by the key informants and evaluators based upon their combined experience as to what constitutes information that can be used in the management of programs and projects.
33 This finding is similar to the findings of a December 2006 report to Treasury Board concerning Grants and Contributions and the need to apply risk management principles to have both sound and effective reporting of results and an effective set of rules and requirements for grants and contributions. Report of the Blue Ribbon Panel on Grants and Contributions. http://dsp-psd.pwgsc.gc.ca/Collection/BT22-109-2007E.pdf.
34 Significant errors are errors in which the information was either wrong or it did not answer the question being asked. Spelling mistakes and such were not included when identifying significant errors. Omissions occurred when important information was not transcribed from the annual project reports into the database.
35 It is estimated that 60 to 70% of the key informant interviews included general questions and discussion of what constitutes results, often also including discussions about what are activities and outputs. The precise number of times these topics were discussed was not tracked, as the definition of results was not an evaluation or interview question.
37 Identifying ‘best practices’ requires employing a rigorous analytical approach that formally evaluates a number of related projects or programs, with the objective being to identify a ‘best' or 'exemplary' way for accomplishing certain objectives and results. As an example on how to establish best practices for a health program, refer to: http://www.cdc.gov/pcd/issues/2006/jan/05_0133.htm.
38 Include assessment of Canada Day, St-Jean Baptiste Day, Canadian Multicultural Day, and NAD. This Program relates to the broader Canadian Heritage on Canadian identity.
39 Evaluation of Appendix A of the Canada-NWT Cooperation Agreement for French and Aboriginal Languages, 2009, Page 33.
40 Evaluation Report Hope for the Future – A Call for Strategic Action, Canada-Yukon Agreement for Aboriginal Languages, 2004, Page3.
41 ALI Evaluation, 2003, Page 4.
42 Op. cit, Page 5.
43 Achieved results from both the AC and ALC components reflect expenditures and total numbers of projects or hours where applicable. This information has not been included, not to negate the importance of these figures in their contribution toward increasing the visibility, because the information was from 2006-07 and thus does not reflect the current situation. Retrieved from http://www.pch.gc.ca/pc-ch/publctn/dpr/ctzn/tpp_c_h-eng.cfm
44 This Aboriginal Policy and Evaluation Practices Literature Review (draft – discussion only) focused on: policies of language and culture revitalization and urban issues; extent of policy integration in four countries; large-scale evaluation methods; and evaluation practices and performance measurement strategies for Aboriginal programming.
45 The NAAF is a nationally registered non-profit organization that promotes the education and professional development of Aboriginal peoples with a particular focus on youth. NAAF received funding from private sector sponsors, other levels of government, as well as Canadian Heritage, INAC, and Health Canada.
46 Horizontal Evaluation of the NAAF, Canadian Heritage, November 2009, Page 18.
47 Op. cit., Page 20.
48 Horizontal Evaluation of the NAAF, Canadian Heritage, November 2009, pp.3-4.
49 Op.cit., Page 8.
50 Op. cit., Page 11. PCH Report on Plans and Priorities, 2010-2011. For additional details regarding the Endowment Fund and these payments, please see http://www.tbs-sct.gc.ca/rpp/2010-2011/inst/pch/st-ts02-eng.asp. The RPP states: "NAAF is the largest supporter of Aboriginal education outside the federal government. In 2003–2004, Canadian Heritage provided a $12M endowment for the establishment of an Aboriginal Post-Secondary Scholarship Program. An additional endowment of $10M was approved in March 2007. Investment revenues of the endowment are used to award scholarships and bursaries to Aboriginal post-secondary students across Canada."
51 Op. cit., Page 23.
52 Evaluation of the Canada Endowment Fund, 2008, Page 43.
53 Op. cit., Page 42.
54 Op.cit, Pages 45 and 48.
55 Op. cit., Page 34.
- Date modified :