5. Conclusions, Recommendations and Management Response
This final section of the report presents conclusions and recommendations based on the findings presented in the previous sections. The section covers each of NAAF's activity areas. It is structured along the four evaluation issues.
Three of the four recommendations below relate to issues of program design and delivery with the fourth recommendation focusing on program success. Three of the four recommendations (1, 2 and 4) call for direct action from NAAF while one recommendation (#3) requires direct action by the federal government. INAC will take the lead on responding to all four recommendations, including gathering responses from NAAF on three of the four recommendations. PCH and Health Canada involvement is required under the third recommendation. PCH will also offer support in response to the fourth recommendation.
5.1 Rational and Relevance
The overarching vision statement of NAAF is "Enriching Canada by Advancing Aboriginal Achievement." This is a robust match with Canadian Heritage, Indian and Northern Affairs Canada, and Health Canada priorities and within each department's program funding authority. Moreover, both the overarching vision and the specific activity areas of NAAF are well-matched to the current roles and responsibilities of the Government of Canada regarding Aboriginal Peoples. Aboriginal Peoples continue to be under-represented in PSE and the labour force. As an outcome of this disadvantaged social position, Aboriginal Peoples are forced to combat persistent negative stereotypes through most media and other social channels.
Canadian Heritage, Indian and Northern Affairs Canada, Health Canada, and the federal government as a whole have repeatedly expressed a desire to improve the position of Aboriginal Peoples within Canadian society. NAAF has been selected as a mechanism by which this task can be achieved. The findings of this evaluation indicate NAAF does well to approach and overturn several of the negative circumstances facing the Aboriginal population including under-representation in PSE, under-representation in the labour force, and promoting a positive image of Aboriginal Peoples and culture.
There is a wide range of similar programming to that of NAAF available to Aboriginal Peoples and communities across the country—some of these programs are also funded by the federal government. However, NAAF is by far the largest and most comprehensive service provider in the activity areas it administers. For instance, NAAF is the only organization to offer four activity areas on a national scale to First Nations, Métis, and Inuit peoples.
Beyond NAAF's size and scale advantages – advantages that set it apart from other means of approaching these issues – there is more pressing evidence that, despite alternative programming in existence, there is a continued need for attention to these issues. This evidence lies within the sustained social plight of Canadian Aboriginal Peoples. In essence, even with alternative programming available, expert stakeholders and documentation has indicated that demand far outweighs supply for NAAF's services. Then too, even with NAAF, Aboriginal Peoples still seek out and use alternative sources of assistance to finance their post-secondary education, identify culturally relevant role models, and receive career planning and advancement assistance.
In sum, the evidence revealed through this evaluation supports the continued funding of NAAF, as the rationale and relevance for funding the organization and initiatives of this purpose remain strong and necessary.
5.2 Design and Delivery
The issue of NAAF program reach represents the greatest challenge to the achievement of NAAF's objectives and the objectives of federal government funding. Although findings of the evaluation indicate that NAAF has made significant progress in accurately representing the population in terms of geographic distribution and Aboriginal ancestry, two definable gaps persist. First, is service within the province of Quebec and to French-speaking Aboriginal Peoples. Second, is service to the North and to the Inuit. These two service gaps were identified during the 2003 evaluation; thus, in terms of design and delivery issues, they do identify as an area of significant consideration for the federal government.
However, NAAF has made tangible improvements in bolstering its accessibility to these underserved subpopulations. For one, both BFF and TP activity areas have made planned and successful inroads into the North and Quebec. Moreover, NAAF has improved French capacity over the entire organization, making a commitment to provide all communications and program materials in French upon request. Finally, NAAA and Education Program selection juries have included population sub-targeting as a part of the selection process. In sum, although the number of French and northern stakeholders served has not improved substantially, NAAF has made noticeable changes to increase accessibility to these groups.
Recommendation
1. For federal government funding to be justified as serving the Aboriginal Peoples of Canada, NAAF must continue to make progress on distributing services equitably within the entire Aboriginal population. Improving accessibility for hard-to-reach groups, namely the North and French-speaking subpopulations, must lead to improving actual numbers served.
Management Response: Accepted
The Director of International Relations, INAC will take the lead on this recommendation and work with other government departments and NAAF to develop more targeted measures, such as outreach campaigns, specialized advertising, and stay-in-school and high-school equivalency initiatives for Inuit youth, who have lower high-school graduation rates and to produce additional materials in French for French-speaking youth.
Implementation date: April 1, 2010
In addition, the Director of International Relations, INAC will work with other government departments and NAAF to develop annual targets for take-up of NAAF programs as well as consistent and detailed tracking procedures. INAC will meet bimonthly with NAAF to ensure that targets are being met and progress is being tracked.
Implementation date: January 1, 2010
Through the course of this research, one engaging finding was repeatedly uncovered when investigating the issue of program reach. Stakeholders from Inuit groups have suggested a willingness for NAAF to forge stronger relationships with local organizations to administer services to the North. This suggestion is interesting, given that several stakeholder groups, including NAAF officials, have argued that the largest barriers to increased program provision in the North are financial cost and logistical complexities.
In this context, the presence of working partnerships – between northern organizations, which possess an expertise in the needs of the local community, and NAAF, who possesses an expertise in each activity area – represents many potential synergistic benefits, including in the areas of cost and logistics. Although it was not a finding of this study, the same concept might prove beneficial in attempts to improve services within the French-speaking Aboriginal population.
Recommendation
2. NAAF should explore the possibility and potential of partnering with relevant like-minded organizations. These partnerships should be based on the common goal of improving NAAF service to currently underserved groups.
Management Response: Accepted
The Director of International Relations, INAC will work with other government departments, INAC Sectors and Regions and NAAF to develop a strategy for establishing and maintaining working relationships with organizations such as, but not limited to, Inuit Tapiriit Kanatami (ITK) and other regional Inuit organizations as well as L'Assembléé des Premières nations du Québec et Labrador (APNQL) in order to improve ongoing "on the ground” outreach to currently underserved groups, including targets and performance measures.
Implementation date: January 1, 2010
Other issues of design and delivery have been effectively managed by NAAF. For one, among program participants, there has been a high degree of satisfaction with the structure and delivery mechanisms of NAAF programming. External experts and observers (public and private sector funders, other National Aboriginal Organizations) are highly satisfied with the activity areas NAAF operates, and how they are operated. NAAF program information and materials are comprehensive and are of excellent quality and consistency. NAAF activity areas are supported by high-quality background documentation. Evidence shows that all stakeholder groups have been highly satisfied with the quality and accessibility of NAAF program materials.
A review of multiple funding agreements and reporting documentation indicated that NAAF had a strong record of transparent communication with its funders. NAAF officials believe that current reporting requirements allow for a fair and transparent representation of the organization. Nevertheless, NAAF officials expressed a degree of frustration with the reporting requirements associated with the funding arrangements they hold with the federal government. The most significant problem they assert is the lack of coordination from one assessment to another. The requirements of this evaluation were considered a part of this concern. Furthermore, some federal government officials have expressed some concern with NAAF's ability to report directly on the requirements that are made of them. No department cited a high degree of difficulty in overcoming this issue through cooperation with NAAF if it had occurred.
Recommendation
3. The federal government should create a single set of reporting requirements for NAAF funding. This will allow the federal government to stipulate a comprehensive set of performance data for review and analysis. Accordingly, it will eliminate the multiple, often competing, reporting demands placed on NAAF and improve the efficiency of the resources NAAF devotes to this task.
Management Response: Accepted
This recommendation requires that INAC, PCH and Health Canada coordinate and streamline their reporting and evaluation requirements, while satisfying needs with regard to accountability, value for money and measuring impacts. The Director of International Relations, INAC will work with PCH and Health Canada to establish an interdepartmental working group to identify and assess current reporting demands, and to propose options which will reduce the reporting burden while not diluting reporting standards. This working group will report by January 30, 2010.
Implementation date: November 1, 2009
In addition to standardizing reporting requirements across the three departments (INAC, PCH and Health Canada), the Director of APP at PCH will explore opportunities to share data and the PCH reporting templates with other federal government funders, such as HRSDC, in hopes of easing NAAF's reporting requirements while maintaining federal government accountability obligations.
Implementation date: November 1, 2010
5.3 Success
At the outset of this evaluation study, it was decided that measurements of NAAF success would be made within the program participant group rather than the entire Aboriginal population. The exception was the NAAA.
In this light, evidence gathered in this evaluation indicates how NAAF activity areas have been proficient in achieving and approaching the majority of the outcomes laid out in the program's logic. The organization has increased the visibility and profile of Aboriginal Peoples and culture within Canadian society – a much needed outcome. Furthermore, among program participants, NAAF has significantly assisted Aboriginal Canadians, notably youth, in improving participation rates in PSE and, moreover, in terms of total educational attainment, two areas in which the Aboriginal population lags significantly behind the non-Aboriginal population. NAAF has also made significant progress on presenting Aboriginal young people with a broad range of career perspectives and the motivation and self-belief necessary to achieve these outcomes. Overall, NAAF has certainly had a significant positive impact on the Canadian Aboriginal population, especially those who have had direct contact with NAAF's programming.
In general, NAAF program participants hold a high degree of satisfaction with the programs they use. This is a significant finding, as not only does NAAF do well to achieve program outcomes, it does so in a manner that is accepted and celebrated within the Aboriginal community. With such strong support from the community, the organization has the potential for truly organic, participatory growth. A caveat to this point, however, should be noted. Specifically, the importance of improving service and perhaps working relationships with currently underserved groups.
Moreover, external observers and experts (government officials, senior officials at other Aboriginal organizations, non-government sponsors) are highly supportive of and satisfied with the achievements of NAAF work. Nevertheless, from the perspective of the federal government, there is a need that NAAF draw robust, tangible links between the services it provides and the outcomes it claims to pursue and achieve. The TP and BFF activity area are, in some cases, prone to insufficiency on this important funding criterion. For instance, it is very difficult to postulate a strong degree of positive association between participation in a BFF or TP event and increased rates of educational participation and attainment. In essence, the problem is not in providing a legitimate argument for correlation but in actually tracking what BFF and TP participants are doing prior to participation in these programs. The simplest solution to this issue would be monitoring participant flow from one activity area to another. In this case, flow from BFF and TP to the Education Program. Furthermore, to improve the likelihood that this specific link occurs, NAAF should dynamically promote the opportunities available through the Education Program at BFF and TP events.
Recommendation
4. NAAF should scale-up the promotion of the Education Program to participants in TP and BFF. This will support the degree to which intermediate and final outcomes are achieved for both the TP and BFF activity areas. Moreover, in order to improve monitoring and measurement of program success, NAAF should track participant flow from one activity area to another.
Management Response: Accepted
The Director of International Relations, INAC will work with other government departments and NAAF through regular multi-party meetings to support NAAF in developing more effective monitoring systems that can better track participant activity prior to and following participation in TP or BFF activities by, for example, establishing participant profiles which can be entered into a database and tracking interactive website use.
Implementation date: April 1, 2010
The Director of APP at PCH will support this recommendation by contributing expertise in the development of NAAF's monitoring systems.
Implementation date: April 1, 2010
5.4 Cost-Effectiveness
The level of financial resources delivered to NAAF by the federal government has not changed significantly over the past five years. The minimal financial analysis performed in this evaluation indicates that the mix of private and public sector funding, and the mix of contribution payments and trust funds that NAAF receives from the federal government, is suited to the objectives of NAAF and is received with reasonable expectations of results. The level of funding currently provided is sufficient to maintain NAAF's program mix and scale of operations. It will likely not favour program expansion.
To a certain extent, the funding provided to NAAF by the federal government is a source of stability for operations. Government funding alone could not support NAAF's current operations; however, the funding provided by the government plays a critical role in attracting private sector donors. To these donors, a financial support-base from the federal government is an indicator of organizational sustainability and accountability.
One important lesson of this evaluation, which arose from the document and literature review, was that the federal government could do better to maintain accessible records of contribution payments made to NAAF. An improved record of financial documentation, or at least the improved accessibility of financial documentation, will definitely better senior managers' understanding of the cost-effectiveness of NAAF funding on an ongoing basis.
Another key lesson learned is that any future studies or evaluation work performed should be supported by the analysis of performance measurement data, which is collected on a regular basis and made available for analysis without complication. NAAF should make more accessible and up-to-date records of performance indicators available to all relevant stakeholders, including federal funders. By increasing the accessibility of such information, NAAF will not only allow for clearer communication with key funders, but will also facilitate the preparation of consequential, meaningful, and accurate promotional materials. Useful performance measurement data is already being collected by NAAF, and NAAF should take further steps to improve the accessibility and dissemination of this data.
The extent to which government funding has been allocated internally between activity areas is largely dependent on the objectives NAAF is expected to pursue. In this context, investigation indicates NAAF has been efficient in the allocation of funding to its four distinct activity areas. However, it also indicates that if the federal government wishes to pursue the growth or expansion of certain activity areas or outcomes, funding considerations and targeted analysis will be necessary.
More generally, NAAF provides the federal government with a highly cost-effective means of achieving many departmental and government-wide priorities and commitments regarding Aboriginal Peoples. Several factors contribute to this experience including NAAF's strong reputation with Aboriginal and non-Aboriginal peoples alike, NAAF's network of Aboriginal stakeholders, NAAF's strong relationships with appropriate private sector partners, NAAF's history of success in the area of Aboriginal social development, and, finally, NAAF's current skills base and capacity for performing high-quality work in the field. For the federal government to have a mechanism of such capability and proficiency available for use in pursuing its own objectives represents a considerable cost-savings over providing the services on its own or even through an alternative organization. This is a considerable advantage provided to the federal government in terms of value-for-money.