DESCRIPTIVE SUMMARY

Institutional Research and Evaluation

Between 1995 and 1997, a team of faculty, staff, and administrators managed institutional planning, research, and evaluation at the college under a system known as ICAN (Improvement: Continuous and Never-ending) (3.1), a participatory governance effort. A previous master planning effort dating from 1990 had been primarily the work of the Sage Institute, a consulting firm that the district employed. By 1997, however, the college dissolved ICAN (3.2) and reassigned the responsibility for planning to three new sub-committees of the newly revised Mission President’s Coordinating Council (3.3): the Master Planning Committee (3.4), the Accreditation Committee (3.5), and the Program Review Committee (3.6). By 1998, the endeavor to produce a systematic planning process had entirely become the responsibility of the Master Planning Committee. As of this writing, the college has again revised the governance structure with regard to the planning process to better suit the proposed campus-wide assessment process (see Standard 10).

These efforts resulted in the development of two formal master plans: the Sage Institute Comprehensive Master Plan 1990-1995 (3.7), written before the college moved to its permanent location, and Los Angeles Mission College: Odyssey 2001 (3.8), written in 1996 by ICAN. In addition, two other incomplete attempts to form a master plan using outside consultants did not result in formal documents. The first effort took place under the auspices of the Maas, Rao, and Taylor consulting firm in 1994 (3.9); the second effort was an effort by the college with the assistance of two consultants from the Executive Service Corporation that occurred in 1998. Overall, few members of the college community were familiar with any of these plans, and they have had limited effect in guiding the direction of the college as a whole.

Historically, research at the college had been supported principally through special funds, which has included the hiring of research personnel for particular programs. However, in November 1999, the college hired an associate dean for campus development and institutional research, and most of the infrastructure necessary to perform institutional research and reporting has been developed since then. This has included both data acquisition and the development of appropriate delivery mechanisms and reporting formats. The college has acquired and made accessible specialized research databases of demographic and student characteristics, course enrollments, degrees and certificates awarded, and basic skills assessment and placement (3.10). It has also made accessible other data sources, such as previous district and college student surveys (3.11), recent U.S. Census data, and data from the Los Angeles Community College’s principal administrative systems. In addition, the college introduced in February 2000 the Instructional Measures System (IMS) (3.12), a reporting system designed to provide up-to-date information about instructional staffing and enrollment. A Website (3.13) devoted to institutional research was developed in the spring of 2000 to more effectively display and disseminate research information. And a comprehensive, regular survey program, covering students, faculty and staff, community, and alumni is under development.

The college has used these data sources to investigate a variety of empirical questions stemming from both accountability requirements and the desire to improve institutional quality. Trends and comparative information on a variety of student characteristics and performance indicators (such as access, success, retention, transfers, degrees and certificates) have begun to allow the college to describe the student population and evaluate progress toward Partnership for Excellence and other institutional goals (3.14).

Currently, the college provides physical space, computer equipment, software, and supplies to support ongoing research. In addition, the college supports the participation of research personnel in district-wide committees and funds conference attendance related to institutional research and assessment. The college also has an institutional membership in the Research and Planning Group of the California Community Colleges.

Until recently, most planning and decision-making at the college has been driven by the administration, which has considered influences from community groups and has been held accountable for the performance of the college by the district chancellor and Board of Trustees. Although the administration has guided planning, it has distributed oversight and consent for plans and resource allocation among several participatory governance committees.

For example, the Budget Committee (financial resource allocation) and the Campus Development Committee (facilities use, maintenance, and construction) have submitted policies and plans directly to the Mission President’s Coordinating Council. The Academic Senate Faculty Hiring Committee (3.15) has established hiring priorities and forwarded its recommendations to the president. Department chairs have submitted requests for classified workers to one of the three vice presidents, who have presented the request to the Budget Committee for consideration. One of the stated functions of the Mission President’s Coordinating Council has been to coordinate all of these plans toward achievement of the stated mission and goals.

Since 1997, the Master Planning Committee has been specifically responsible for the development of a comprehensive planning policy. The primary charge of the committee has been to develop a planning process that would integrate educational, financial, facilities, and human resource planning toward the achievement of the college mission. Though the committee has proposed and discussed various methods over a three-year period, it has only recently agreed upon a course of action. In 1999, the committee voted to implement the following strategies:

The college will clearly define, review, and assess institutional goals;

The college will establish, review, and assess several practical strategies for each goal as well as institutional-level outcome measures (performance indicators) and criteria;

All college units will define their own realistic objectives, outcome measures, and criteria based on college goals and strategies on a prescribed timetable;

The appropriate participatory governance body will prioritize annual operational plans involving resource allocation according to their support for the college mission and goals;

The associate dean for campus development and institutional research and the appropriate new participatory governance bodies will assess outcomes for both annual operational plans and the systematic unit institutional effectiveness review;

The Master Planning Committee will serve as the "oversight" body, providing constructive criticism and guidance to the units and a bridge to the participatory governance coordinating body.

Attendant to the effort, the administration and the committee facilitated the creation of a mission statement and seven college goals, which the Faculty Senate, the Community Advisory Committee, and the Mission President’s Coordinating Council formally adopted in October 1999. Subsequently, as discussed in Standard 1, the new associate dean of research and a subcommittee of the planning committee revisited the mission statement and goals and again revised them to make them more measurable. The planning committee approved this revision in July of 2000 (3.16). The revised mission statement and goals are published in the 2000-2001 Catalog (3.17).

In pursuing the strategies above, the same planning committee proposed a new comprehensive assessment and planning process that is designed to be a continuous, systematic procedure using research to identify needs, establishing measurable outcomes, evaluating those outcomes, tracking outcomes over time, and providing feedback to improve the process and the institution. (This process is described in detail in the self-evaluation section of this standard.)

Institutional Outcomes Assessment

Two principal factors have guided institutional outcomes assessment: those related to the financial health of the college (budgetary) and those motivated by the accreditation standards and the stated mission of the college (student learning outcomes). District-wide fiscal constraints dictated that the budget-driven outcomes were the highest priority for improvement efforts from 1996 to 1999. During that period, the college made attempts to improve weekly student contact hours (WSCH), full-time equivalent students (FTES), full-time equivalent faculty (FTE), and the ratio of WSCH to FTE (WSCH/FTE).

The district budget with its allocation formula (3.18) has determined the framework within which the college administration establishes goals for internal efficiency and enrollment growth. College administrators have communicated these goals in meetings with individual department chairs, occasionally presenting disciplines with historical data and other factors considered in allocation of FTE. During the period of budget constraints, an overarching mandate in college planning and resource allocation was to increase WSCH/FTE so that college income would match or exceed expenses. FTE allocation to each discipline on a semester basis has been the responsibility of the vice president of Academic Affairs, who discusses department course offering needs with each chair. Chairs have been given discretion to apply their allotted FTE in a manner that would balance the educational needs of the students with the efficiency goals of the college and the district. Overall, this strategy has succeeded in that the college’s expenses did not exceed its district allocation for fiscal year 1999-2000.

The college has also collected data that responds to Partnership for Excellence and to other traditional outcome measures, but few members of the college community are aware of the specifics. Most are surprised to learn that the college had only 183 documented transfers to 4-year institutions and 306 AA/AS degree recipients (3.19) in the 1998-1999 academic year. Likewise, few know the college enrollment in a particular semester, much less the enrollment goal necessary for the college to meet its operating budget.

The college has specified and measured some outcomes, typically as the result of an administrative evaluation, external initiative, or ad hoc directive (e.g., discipline WSCH/FTEF, average class size, or Partnership for Excellence outcome targets). Because of the absence of a defined college-centered planning process, some of these measures and their associated goals have, by default, become the college’s indicators. These measures have not been completely understood by the college community, have not been well defined in relation to available or potential data sources, and have not necessarily been related to the college’s mission and goals. In 2000, however, the college has made efforts to publicize PFE measures and use them in preparing measurable objectives.

Since 1995, Program Review has been the vehicle for oversight of the assessment of student learning outcomes and for overall program evaluation (3.20). In 1999, the college reorganized this process to include the following components (in addition to the fiscal considerations mentioned above): review of student surveys on effectiveness of programs and courses; review of course outlines, certificate requirements, and degree requirements; analysis of instruments used to assess learning outcomes and skill acquisition; description of the unique characteristics of each discipline; list of major accomplishments; identification of major changes in the academic/vocational area; assessment of facilities, financial, and human resource needs; and establishment of specific goals and objectives. Another revision to the process in 2000 included the establishment of specific outcome measures (performance indicators) tied to the objectives and a mechanism for applying outcomes to improvement. With this revision, the college renamed the process Institutional Effectiveness and expanded it to evaluate all units in the college, including committees and offices. The proposal that the college move forward with an assessment and planning model that would incorporate the former program review process and coordinate with annual operational planning motivated this change.

An integral part of each program review has been the evaluation of expected student outcomes as revealed by achievement of learning objectives written in the course outline (3.21), a document that the Academic Senate Curriculum Committee must approve. For the most part, individual instructors and departments, using traditional methods such as evaluation of assignments and testing, have been measuring student achievement of these learning objectives. The English and math departments are currently discussing standardized evaluation devices that will assess "satisfactory" achievement of learning objectives in introductory courses.

Measurement and validation of student achievement beyond the level of the individual instructor or discipline has been limited, but Program Review has attempted to examine the outcomes of students who are no longer enrolled at the college. First, the district has consolidated statistics on certificates and degrees awarded by the college in a district report (3.22). Second, transfer student tracking has been performed primarily using reports from the state chancellor’s office and the National Student Loan Clearinghouse (3.23). However, other than anecdotal evidence (3.24) established by correspondence between faculty and former students, no campus-wide process exists for measuring student job placement or job advancement.

To evaluate student services and overall student satisfaction with the educational services that the college provides, the vice president of Student Services conducted Noel-Levitz Student Satisfaction Surveys in 1996 (3.25) and 1999 (3.26). Likewise, individuals report using their own surveys and district data to gather pertinent information in order to assist in program development, planning, and grant writing. However, until very recently, coordination of research with planning, program evaluation, and resource allocation at an institutional level has occurred in a non-systematic, ad hoc fashion as information needs arise.

SELF EVALUATION

Institutional Research and Evaluation

The history of planning at the college reveals that it has not had a consistent, ongoing institutional planning and evaluation process integrated with research. This has tended to reduce the reliance on research information.

Research activities have mostly focused on particular topics or programs (such as matriculation assessment and placement, instructional staffing, and enrollment), and the information generated has supported accountability and decision-making functions. Sometimes a specific issue or administrative initiative has spurred some research, but this has tended to be a one-time activity with little connection to the planning process.

The fact that the college has recently hired a full-time research officer at the associate dean level indicates a significant commitment of resources. However, a process does not yet exist to determine the effectiveness of research and evaluation. Consequently, the college cannot determine the adequacy of funding in relation to the contribution that research is expected to make in the areas of institutional planning, institutional effectiveness, and accountability mandates. The college has also purchased planning software, (3.27) representing another important commitment of resources. In addition, the new president has proposed the establishment of an assessment and planning office with sufficient staff to support the proposed process described below.

Until the proposed assessment and planning process is fully operational, the college does not have a way of determining how well it is accomplishing its mission and goals. The college does track a number of traditional performance and student outcome measures for community colleges (such as retention, persistence, WSCH/FTEF, enrollment, transfer, etc.) However, until assessment is under way, it does not have a continuous process to determine the extent to which these measures are appropriately related to the college’s mission and goals.

As discussed in Standard 1, the college has recently revised the mission statement and goals to make them more measurable. Although the college has not yet formalized the goals into measurable objectives for all units, it has established accompanying outcome measures and assessment strategies for the new assessment process. Until the college institutionalizes assessment and planning and develops a coordinated planning environment, progress toward the college’s goals cannot be effectively documented, tracked, and retained in institutional memory.

No clear evidence yet exists that program evaluations lead to the improvement of programs and services at the college because the new Institutional Effectiveness Process, which now includes measures of effectiveness, is in the process of being implemented.

Institutional Planning

Until the recent development of the assessment and planning process by the planning committee, the college had not defined or published its procedure. Rather, the planning process had to be inferred from the context of the entire governance structure of the college. As a result, the process has been unknown and poorly understood. In general, the college community has been aware that the individual committees that have been in existence for several years have been the deliberative bodies for their respective areas. However, because of a lack of a systematic planning process, committee members have made decisions in isolation and in an uncoordinated fashion.

The committees that have been responsible for planning and resource allocation have varied in composition and reflect different segments of the college community. Participatory governance committees have included the Budget Committee, the Campus Development Committee, and the Master Planning Committee. Academic Senate committees with similar planning responsibilities include the Faculty Hiring Committee and the Curriculum Committee.

The Classified Hiring Committee has not met for two years, and the three vice presidents primarily decide classified hiring as personnel needs arise. It should be noted that in the planning process adequate representation of appropriate segments of the college community is dependent upon active participation by all committee members. This requires appropriate announcements, scheduling of committee meetings, and administrative support for classified staff participation. Recently, classified personnel and students have been underrepresented in the planning process. However, classified staff employees are represented on the planning committee that has been designing the new assessment and planning process.

Until the development of the new assessment and planning model is completed, evaluation and planning will not be integrated at the institutional level. Although evaluation of educational programs and student services has been taking place, no strategy for linking the information-gathering processes with institutional planning and resource allocation has existed. For example, no formal policy or procedure has existed for committees involved in the planning processes to receive and analyze program/discipline reviews. A major difficulty in identifying priorities for improvement has been a lack of specific and measurable performance indicators. During the development of the assessment and planning process, the committee developed performance indicators for every strategy attached to all college goals. This is important because until the college establishes a baseline for performance, it will not be able to identify where to apply resources in order to achieve maximum benefit toward achievement of its mission.

Institutional Outcomes Assessment

The college has not explicitly defined or published intended institutional outcomes. Thus, no organized documentation for their achievement exists. Several categories of institutional outcomes have implicitly operated at the college by default: ongoing budget-driven outcomes and state-driven outcomes such as Partnership for Excellence criteria. However, the college has developed nascent outcomes (performance indicators) matched with appropriate strategies from the most recent college mission statement and goals. Thus, each unit of the college can establish its own achievable objectives with measurable outcomes. Without measurable college goals, unit objectives, and performance indicators, it is not possible to systematically assess the progress of the college or its units.

It will take considerable institutional commitment and organization for the latter to replace the former as the primary criteria by which the college measures achievement. Previous college master plans have included a mission statement and goals attached to numerous impractical and difficult to measure objectives lacking performance indicators. Therefore, the college has never evaluated achievement in a systematic fashion.

Until the 1999 revision of program review, the process had not been effective in evaluating and improving the programs and disciplines undergoing review. Failure of the process was due partly to an unhappy relationship between the faculty and the process itself. Instructors and administrators in the past viewed the process primarily as an intrusion and an imposition on their time. This perception was valid because program evaluations were not implemented, and they provided no consequences for the management of programs and disciplines.

The current redefined Institutional Effectiveness Process not only includes outcome measures but also more actively involves the disciplines, helps faculty define realistic objectives appropriate to the goals and resources of the college, and in general engages them in a discussion of how they work with students. However, this new process will take several years before the college can collect and use much evidence to effect improvement. The Institutional Effectiveness Process is an inextricable part of the larger assessment and planning process described below.

Program review, in its prior incarnations, focused solely on instructional disciplines. Because of its role in determining the effectiveness of the entire college, Institutional Effectiveness will include non-instructional units of the college as well.

Without defined outcomes and a coordinated system for their assessment, the college has not had a formal vehicle, such as an annual report, to communicate quality assurance information to the public.

The college communicates the quality of the education and services to the community chiefly by anecdote through active and former students. The federal Student Right to Know Act mandates that the college publish specific information about institutional quality (e.g. transfer rate, graduation rate, crime statistics) on a regular basis. A primary responsibility of the new associate dean for campus development and institutional research is to consolidate this information and make it available to the public (3.28). The college does include the official WASC accreditation statement in several of its official publications (3.29).

Members of the college serve on various community service groups such as Kiwanis and local chambers of commerce, and the college president meets several times each year with a community advisory group. These activities enable college leadership to demonstrate to key community members the value and quality of the college and its programs to the community at large.

To differing degrees, vocational programs of the college use community advisory committees to assist in program planning and development (3.30). These groups allow for the exchange of information relevant to the professional expertise of faculty and the student skills that employers desire. Academic programs do not have such advisory committees.

The absence of a systematic means for assuring quality to the public is a major weakness of the college. The college’s humble beginnings in downtown San Fernando storefronts continues to influence the collective perception of Mission College. Although enrollment has grown dramatically since its move to the permanent location in 1991, lack of communication has led many in the region to maintain misperceptions of the quality and types of programs the college offers.

Integrated institutional research, evaluation, and planning have not existed at the college. Therefore, a reason to assess these functions has never existed. College planning, evaluation, and research have historically reflected the management style of continuously changing college and district administrations. Implementation of and commitment to a long-term, stable, and integrated strategy based upon the needs of the institution has yet to be accomplished.

Assessment and Planning at Los Angeles Mission College

In the 1999-2000 academic year, the convergence of three events caused a significant change in the college’s approach to outcomes assessment and planning: a number of faculty and administrators attended several California Assessment Institutes (3.31); the college hired an associate dean of research with expertise in assessment and planning; and the district appointed a new permanent college president who has an interest in institutional effectiveness and quality assurance.

Because of these influences, the college began to implement an assessment and planning process in spring of 2000. At that time, a subcommittee of the Master Planning Committee set up a process to use the college goals to develop measurable objectives for each unit of the college. (As part of this review, the committee also revised the goals, as discussed in Standard 1.)

The subcommittee developed a detailed document that made clear basic assumptions about the assessment and planning process, described and expanded on the college goals, outlined its intention, suggested strategies for achieving these goals, and described performance indicators associated with each goal. It presented this document, along with the recommendation that the college begin assessment, to the Master Planning Committee in fall 2000 (3.32). This process, which would subsume and expand on the previous program review processes, would require each unit of the college to set measurable objectives, implement these objectives, and evaluate outcomes in terms of performance indicators the college had established with the goal of continuing improvement. In addition, the subcommittee suggested an initial set of college units to undergo assessment.

It is important to understand the basic assumptions the committee made about assessment to understand the fundamental change that will occur in the culture of the college as it implements the process. Those assumptions are contained in the document Assessment and Planning at Los Angeles Mission College (3.33).

"Los Angeles Mission College desires to assure the public of the quality of education at the institution. It further recognizes the mandate from the Western Association of Schools and Colleges that institutions assess themselves in a systematic manner designed to improve the effectiveness of teaching and learning. The college views the assessment process, conducted within the context of a "culture of evidence," as a means of achieving both institutional improvement and public accountability.

Assessment involves three principal activities: Planning, Implementation, and Evaluation. Planning involves developing measurable objectives, establishing associated core measures of effectiveness, deciding actions to be taken, determining responsibility for these actions, and allocating sufficient resources to carry out the actions. Implementation involves carrying out the actions and revising them, where appropriate. Evaluation involves measuring the effect of these actions on the institution through established performance indicators, analysis and interpretation of outcomes to determine effectiveness, and, if necessary, recommending changes to improve institutional effectiveness.

The faculty, administration, and staff commit to institutional assessment. The effectiveness of every unit of the college will be measured against clearly stated objectives that stem directly from the institutional mission and goals, and resources will flow to the units that are most effectively advancing college goals. The college pledges to make the assessment process as inclusive as possible to disseminate the results of assessment to all interested constituencies and to use these results to improve the institution and meet the public’s need for accountability."

Further, the guidelines from the same document explicitly define the parameters of the process as it is implemented:

1. The focus of assessment and planning is first, institutional improvement and second, institutional accountability.

2. Assessment will extend to all aspects of the college and, therefore, will require the participation of all college costituencies.

3. Only established performance indicators as well as data and information that represent important and enduring values the college strives to attain form the basis of the "culture of evidence" and will be used to determine institutional effectiveness.

4. All college resources (financial, physical, and human) will be allocated to the maximum extent permitted by institutional constraints through the assessment process.

5. Resource allocation will be based on plan objectives and outcomes, as derived from the assessment process and will conform to principles of participatory governance as established at the college.

In the spring of 2000, the college also purchased a software system to assist with the management of the assessment and planning process. This system will allow the college to organize more effectively the objectives that each unit develops, track the progress that units make toward their objectives, prioritize budget requests associated with unit objectives, and develop evaluative reports focused on improving institutional performance.

In the fall of 2000, the college began to consider major changes in its governance model that would explicitly link assessment and planning to college-decision making (3.34). At that time, the new college president initiated discussion on this topic by introducing a proposal to consolidate several of the college’s standing participatory governance committees as outlined in Standard 10. Under the president’s proposal as modified during deliberation in several campus committees, the former Master Planning Committee and Accreditation Committee would be combined and renamed the Assessment and Planning Committee.

The functions of the committee are currently being refined, as are operational procedures for implementing unit operational planning and the evaluation of unit effectiveness. The Mission President’s Coordinating Council passed the proposed changes to the governance structure on November 13, 2000 (3.35).

Once the assessment and planning process is underway, the college will publish a yearly report that describes annual progress toward each of the college’s goals, particularly describing trends in performance indicators for each goal. Such a process would also be a small, but significant, step toward the integration of college planning, evaluation, and research.

PANNING AGENDA

5. In the 2000-2001 academic year, the college will institutionalize the assessment and planning process that the planning committee developed in 1999-2000 so that it will be fully implemented during the 2001-2002 academic year. The college will establish the enabling governance model associated with the process along with an assessment office under the direction of the associate dean for campus development and institutional research. The college will also develop the operational plan using the process, and selected units of the college will undergo evaluation in the revised Institutional Effectiveness Process.

6. The college will publish the results of the assessment and planning process on the Website and in print annually.