Chapter 4: SMSU as a Learning-Focused Organization
Assessing Student Learning
Criteria and Core Components supported in this section include 2c, 4c.
Assessment of student learning at Missouri State is embedded throughout all levels of the University. In general, curriculum development and review are strengthened through a decentralized but systematic process of program assessment at the department, college, and university levels. Assessing whether existing curriculum is rigorous, responsive to student needs, and meets the expectations of external constituents is carried out in a variety of ways. Representative examples of assessment strategies include
This section of the report discusses the connections between assessment and long-range planning, the work of the Center for Assessment and Instructional Support (CAIS), the assessment processes of academic units, and the guidance of the Assessment Council (AC). In sum, each university unit collects the information it needs to assess its programs and make decisions about how to improve student learning. The CAIS, through the guidance of the AC and the long-range plans, advises units, collects unit reports, and assists in compiling the university-level Annual Performance Measures that monitor the progress of learning within the University. The method of data collection for the performance measures makes the reported information useful at the level of origination (units) as well as at the administrative levels as they assess student learning and pursue continuous improvement.
Assessment and Long-Range Planning: A Brief History
Criteria and Core Components supported in this section include 2d, 4c, 5a.
Since just prior to the 1995 NCA/HLC Self-Study, the CAIS has been immersed in the University’s long-range planning and assessment strategies. The CAIS worked with the Assessment Council, Academic Affairs, and the Administrative Council to design the Plan for Improved Student Learning and Assessment, which delineates a decentralized approach. This approach assumes that the academic and non-academic units being evaluated are the most knowledgeable about their disciplines, about how to assess student learning in their disciplines, and about how changes could be implemented to improve their programs. Periodic reporting by units, described in sections that follow, documents the improvement processes.
The 2000-06 long-range planning process involved the CAIS, which revised the Measuring Progress section of the previous plan and added Chart Q, Assessing University Outcomes. The former section describes system effectiveness measures in order to track the implementation of state and institutional goals. In addition to performance measures, annual program outcome measures provide academic review statistics for use in developing longitudinal descriptions of the academic units that are associated with the University’s themes. The regular collection of measurement statistics maintains a continuous data set to support a variety of reporting and accountability requirements as well as reviews and decisions. These measures link assessments within academic units to the University as a whole through an ongoing system. Chart Q illustrates a comprehensive system of assessments coordinated by CAIS to provide a pool of outcome data that supports campus decisions.
Since the 1995 Self-Study, departments have received requests for Assessment Reports on Student Learning Improvement and Validation periodically from the CAIS. These reports include descriptions of the processes for assessment and program objectives and how they are assessed, changes made as a result of assessments, and results supporting improvements and validating current practices. In response to suggestions expressed in the last NCA Site Team Report, the CAIS designed several worksheets to assist department heads and committees in identifying which of their regular activities provide assessment information and helping them link the outcomes of those activities with program objectives. The worksheets include
The University’s approach to assessment assumes that the academic and non-academic units being evaluated are the most knowledgeable about their disciplines.
The resulting Student Learning Improvement and Validation Reports from units, which were first incorporated into the University’s planning ten years ago, provide information to faculty for use in improving classroom techniques and curricular decisions. Administrators and committees use them as an information base in reviews and reports, and as indicators of needed program revisions and resources. Colleges and departments have interpreted data and performed specific evaluations of students’ progress through their major programs. The CAIS regularly has summarized information from unit=level plans into tables for each college. The assessment tables indicate the types of assessment and the points in the students’ educational career at which the assessments occurred. Unit reports indicate the use of assessment information in discussions and decisions among the faculty, departments, and colleges, and with Academic Affairs.
Embedding assessment within the University’s long-range plans has supported Missouri State’s goal of increasing standards in teaching and learning
The embedding of assessment within the University’s long-range plans has supported its goal of increasing standards in teaching and learning and its campus-wide emphasis on student-centered and professional obligations of faculty and staff. The 1995 long-range plan (Welcoming) included several assessment goals: 1) assign responsibilities for measuring student learning achievement and reviewing the curriculum, 2) adopt a host of measurement and outcomes assessment activities, and 3) establish performance measures to evaluate progress toward accomplishing the plan itself. The plan further stated that “it is imperative that departments and colleges critically examine the curriculum they offer—both individual courses and major programs of study.” The plan mandated the “review of offerings in light of providing efficient, effective, learning experiences” for SMSU students. The success of the assessment aspects of the long-range plan are measured by 1) comparing revised offerings with the previous courses and programs, 2) surveying faculty and departments, and 3) surveying employers to assess the value of instructional programs and provide information for annual reporting. Some examples of the results of these processes are included in the description of unit assessment processes that follows.
Welcoming also required detailed processes for assessing outcomes of the revised general education program, of the Public affairs mission, and of distance learning courses. Assessments that continue to be used, but are separate from the academic unit assessment processes, include assessments of General Education, Public Affairs, and Distance Learning.
Assessment of General Education
Table 4.1: Baccalaureate Degree Recipient Data Reported to CBHE for FY 04
Assessment of Public Affairs
Public Affairs outcomes are evaluated in part through the UCLA Cooperative Institutional Research Program (CIRP) survey and the Higher Education Research Institute (HERI) College Student Survey of Seniors, locally developed questionnaires, volunteer activities in the community, service-learning coursework, and general education courses. Since implementing the CIRP survey in 1997 and the HERI survey four years later, a considerable amount of trend data has become available to consider when making decisions. A Citizenship and Civic Issues questionnaire accompanies these surveys. Results of the latter are reported in Performance Measure 60, Student Citizenship.
Assessment of Distance Learning
An assessment task force reviewed distance-learning courses in 1999-2000 to evaluate and improve the assessment of courses delivered by a variety of media: television, internet, or recordings. The primary purpose was to ensure that distance-learning courses maintain the same level of intellectual rigor and learning outcomes as on-campus courses.
Coordinating Assessment: The Center for Assessment and Instructional Support (CAIS)
Criteria and Core Components supported in this section include 1e, 4b.
Much of the University’s assessment coordination occurs through the work of the CAIS, established in 1987. Since its inception, the CAIS has coordinated the collection and reporting of outcome information that is incorporated into system, institutional, and department plans; self-studies, five-year campus reviews; Missouri Department of Higher Education reviews; the Higher Learning Commission Accreditation Self-Study; and other accreditation self-studies. The intent of the reported outcomes is to represent the priorities and commitments of state and institutional goals and to provide evidence that the University is making progress toward its educational and public affairs mission. This information also provides support for decisions designed to improve student learning outcomes.
The CAIS maintains integrity by ensuring the confidentiality of all information. All tests and surveys administered by the CAIS are submitted for review by the Institutional Review Board for the Protection of Human Subjects. All CAIS personnel, including graduate assistants, complete the Human Participant Protections Education for Research Completion Certificate. Resource information concerning the required general education exit examination, national and local surveys, and major field tests are provided on the Assessment Web site. Standard procedures for administration of assessments ensure that all participants are treated fairly and consistently. (See Figure 4.1 on the following page.)
Figure 4.1: The CAIS Program
The CAIS conducts a comprehensive multidimensional program that:
Criterion and Core Components supported in this section include: 1e, 4b.
The Assessment Council (AC)
Founded in 1987, the Assessment Council acts in an advisory capacity to the CAIS director and as a communication vehicle for Missouri State University colleges and departments. The Council ensures that University performance measures provide a data pool of specific information on learning outcomes in general education, academic majors, employability, lifelong learning and public affairs. This information (collected through Chart Q) has included assessments in public affairs that incorporate items related to diversity, values, citizenship, and participation in society. The Council also supports assessment of student learning outcomes through plan documentation review, information collected and provided to the University community, and resources made available to internal and external constituents. A chronological overview of some of the Council’s work follows.
In addition to collecting the Student Learning Improvement and Validation Reports from academic units, the CAIS assists in undergraduate assessment through placement testing in the basic areas of writing and mathematics. The placement tests help to ensure that students are placed in learning environments best suited to their abilities.
Until the summer of 2005, the CAIS provided essay writing materials and funding for English Department faculty members to supervise placement essay writing and evaluation. Entering students completed a one-hour essay that was scored on the City University of New York evaluation scale. Placement of entering students is now based on ACT English scores.
The Mathematics Department also has received funding from the CAIS for a faculty member to supervise a locally developed mathematics placement assessment during student orientation sessions. Since the fall of 1997, entering students have been allowed to use their ACT math scores for placement in the appropriate courses. Some students continue to take the one-hour placement test, which consists of fifty questions in three components: basic algebra, advanced algebra, and trigonometry.
Assessment within Academic Units
Criteria and Core Components supported in this section include 3a, 3b, 3c, 4a, 4b, 4c, 5a, 5c.
Some of Missouri State’s colleges and departments choose to use the Major Field Achievement Tests (MFAT) as part of their assessment processes. Since 1992 the CAIS has purchased these exams for departments that have requested them. All of the College of Business Administration’s departments and sixteen departments in other colleges now include these standardized examinations in their assessment plans. In 2004 the CAIS reported summary data to the CBHE for 1,098 of 2411 graduating students who took standardized examinations in their major fields. According to national normative data, of those students, 559 scored above the 50th percentile (50.91%) and 193 (17.6%) scored at or above the 80th percentile. Data associated with these scores provide information used to adjust the curriculum and teaching techniques within units and by individual faculty. Some of these changes are mentioned within the following descriptions of the varied assessment processes of Missouri State’s academic units.
Table 4.2 lists all departments that administered the MFATS in 2003-04, while Table 4.3 reflects the major field testing results for FY04 Baccalaureate recipients, as reported to the CBHE.
Table 4.1: MFATS Given by Departments
Departments Giving MFATS in 2003-04
Table 4.2: Baccalaureate Degree Recipient Data for FY 04: Major Field Testing
Graduate programs and courses are assessed through a variety of processes. Colleges review departments and programs on a three-year cycle. In several cases external accreditation involves an additional review. Departments structure and evaluate their graduate programs in ways appropriate to their discipline, usually with some combination of
The Graduate Council
The Graduate Council ensures the rigor and appropriateness of coursework (and other associated graduate experiences) by approving only qualified instructors to teach 600-level courses, direct research, and oversee clinical experiences and other scholarly endeavors. During the past ten years, Graduate Faculty status has been redefined to fall into three categories: Research, Clinical, and Performance. All departments have reviewed their criteria for admission to the Graduate Faculty and submitted these to the Graduate Council for approval. These reviews resulted in higher standards. Each faculty member wishing to have Graduate Faculty status must be recommended by his or her department and approved by the Graduate Council. The quality of faculty is assessed via criteria such as
The Council also reviews and acts on all proposed new graduate courses, changes in existing graduate classes, and changes in graduate policies. The quality of graduate classes is the foremost consideration in evaluating a new graduate program, or assessing an existing one. Graduate courses numbered 600+ inherently involve more depth of information, require more participation and commitment, and require more extensive collection, synthesis, and analysis of information than do undergraduate courses. Classes with numbers 500 – 599 are open to both graduate and upper-division undergraduate students. They either incorporate higher standards for all class members or have higher standards for the graduate students than for the undergraduates.
The Graduate Student Council conducts a biennial graduate student satisfaction survey with the results broadly shared, including direct discussion of results with the Graduate Council (results available in the resource room). In addition, each semester the Graduate College surveys graduating students to determine their satisfaction with their experiences, including programs and courses (results available in the resource room).
Graduate students in the Master of Arts in Teaching program take the Praxis II as a part of the admissions process… the pass rate students in the current cohort is 98%.
All education programs within departments and colleges throughout the University are linked through the Professional Education Unit (PEU) and are assessed in accordance with the PEU governance system and its comprehensive assessment processes. An independent assessment committee evaluates each program on a three-year cycle and provides feedback for consideration in curricular decisions. The data are shared with internal and external constituents through publication on the PEU Web site, internal and external review processes, and the decision making process of the PEU. Assessment plans and results are available for each unit on the COE Web site.
Course matrices for certification, Missouri Standards for Teacher Education Program (MoSTEP), and specialty area competencies are maintained on the COE Web site for the National Council for the Accreditation of Teacher Education (NCATE) and are tied directly to course syllabi. The Educational Testing Service Praxis II examination is mandated for all candidates at the initial level prior to student teaching. Graduate students in the Master of Arts in Teaching program take the Praxis II as a part of the graduate admissions process. The Praxis II pass rate for the current cohort is 98%. Candidate data are collected at three points in the graduate and undergraduate programs for use in ongoing candidate advisement and program development and change. A portfolio process has evolved over the past five years. The Conceptual Framework learning outcomes provide the vision for this assessment while individual programs developed specific criteria for portfolio artifacts, available in electronic form. Portfolio requirements adhere to the PEU three point check system and are specific to the individual program assessment plans.
College of Business Administration (COBA)
The College of Business Administration is involved in a substantial college-wide assessment project. The initial phase targeted the identification and assessment of competencies for every COBA undergraduate. The second phase shifted to departments where competencies for majors and concentrations were revised. The COBA Assessment Task Force is now similarly focusing on the graduate programs. Graduate faculty are identifying competencies, mapping them to core courses, and identifying appropriate assessment measures. In addition, the ETS Major Field Achievement Test for Business has been administered as an overall program evaluation since 1995. To date, more than 6,400 students have been tested. This approach captures more than 90% of the graduating seniors, and the exam results consistently have been above the national average.
Undergraduate and graduate degree programs in business and accounting are accredited by the Association to Advance Collegiate Schools of Business (AACSB) with assessment plans designed to evaluate and improve student learning through those accreditations. Assessment of degree programs offered in the Department of Industrial Management are associated with the accreditation by the National Association of Industrial Technology. The School of Accountancy maintains assessment activities to support quality learning that prepares students for a variety of nationally recognized professional accounting certifications, including Certified Public Accountant, Certified Management Accountant, Certified Internal Auditor, Certified Government Financial Manager, and Certified Fraud Examiner.
College of Humanities and Public Affairs (CHPA)
CHPA has a three-year cycle of program review that requires each department and program to do a self-study resulting in recommendations for the next three years. The college plan is developed by the dean, in consultation with department heads, to guide efforts in curricular development, program change, and facilities management. The dean hosts an annual retreat with department heads each fall to discuss the college plan and the needs of each department.
Feedback from graduating students and core faculty has resulted in a number of substantive and procedural changes that streamline and raise assessment standards in targeted areas.
Each department in the college has its own assessment program, designed and managed by a Faculty Advisory Committee. Examples of assessment procedures include exit interviews, assessment reports by faculty on student progress, alumni surveys, a college-wide faculty evaluation instrument, post-graduation assessment of degree programs, classroom evaluation of teaching by peers, accreditation reviews, and a survey of graduates every five years to collect information on such things as salary levels and career placement. Each department uses this information for discussion about and creation of curricular changes.
For example, the History Department continuously reexamines its programs in light of the accreditation requirements of the Department of Elementary and Secondary Education (DESE), NCATE and the National Council for the Social Studies (NCSS) to ensure that they are making progress toward program goals and achieving success for its students when they take the MFAT. The program received the rating of “Exceptionally Strong” from the NCSS in its last accreditation review.
Graduate program assessments include a comprehensive examination, mandatory seminar paper or thesis, and exit surveys that track career choices and opportunities and evaluate the program. The Master in Public Administration program bases its evaluation on knowledge-based assessment and performance-based assessment, class research requirements, testing and writing components, internships, exit interviews, and the career paths of graduates.
Based on feedback from graduating students and core faculty, the Master of International Affairs and Administration program introduced a number of substantive and procedural changes that streamline and raise assessment standards in targeted areas. Revisions include a four-track program, a thesis option that prepares students for entry into Ph.D. programs, and comprehensive examination questions that are limited to the required core classes. Students with a GPA of 3.75 or higher are eligible for the thesis option and are also exempt from taking the comprehensive exam.
Recent revisions in the Sociology, Anthropology, and Criminology Department as a result of assessment information include addition of a course to address a deficiency found in understanding the juvenile justice system, elimination of a course because students indicated it was not serving its intended purpose, and revision of the structure of the Senior Seminar.
College of Health and Human Services (CHHS)
Examples of the variety of assessment activities used to review and revise relevant course materials and teaching methods in CHHS include the following:
Examples of assessment activities by specific CHHS departments include
College of Arts and Letters (COAL)
The COAL five-year plan has goals and objectives that are assessed yearly by the dean, associate dean, and department heads. Specific steps are taken to align student learning outcomes with the college mission. The plan is flexible enough to allow for changing priorities. Departments have either three- or five-year plans that coordinate with the college plan. COAL also has an external advisory committee and departmental advisory committees that give input for continuous improvement in academic and extra-curricular programs to maintain consistency with professional standards.
Departments have assessment plans designed to evaluate program goals and objectives annually. Some departments in COAL hold faculty retreats that incorporate information from assessment in the planning process. Departments consider assessment feedback from courses and exit evaluations in meetings of assessment committees, curriculum committees, and program review activities. The faculty evaluates the success of students in courses and programs to determine appropriate changes in curriculum and teaching approaches in an effort to improve student learning.
Evaluations include student portfolios reviewed by faculty members and external advisors, exhibitions, and performance evaluations. The recently developed Media, Journalism, and Film program, for example, has clearly specified goals that are evaluated through creative products assessed by internal and external reviewers. Most of the departments publish undergraduate and graduate handbooks for majors that state educational goals and standards for student outcomes. In graduate programs, theses and seminar papers are reviewed by a committee of graduate faculty to ensure that program objectives are met.
The Music Department and Art and Design Department are leaders on campus in conducting assessment programs that evaluated student success at predetermined points along a student’s educational career. These programs have explicit learning goals that are evaluated by internal and external experts. Examples from the Music Department include
The Music Department and the Art and Design Department were leaders on campus in conducting assessment programs…
Changes as a result of the Music Departmental Assessment Committee’s analysis include
College of Natural and Applied Sciences (CNAS )
Each department follows an assessment plan and submits an annual report to the CAIS. The college’s programs also seek accreditation when it is available.
University College (UC)
University College provides the oversight of the General Education program, whose assessment processes have been described previously. In addition to the three-year cyclical review of each course within the program, the CAIS administers an exit exam. One example of a significant curricular change that was made within the General Education program as a result of a thorough assessment process is the recent revision of the “Introduction to University Life” course (IDS 110). Growing concerns voiced in surveys by faculty and students over the structure and effectiveness of IDS 110 led an Advisory Committee in 2003-04 to take a series of steps designed to restructure the course so that it better reflects the General Education goals and public affairs mission of the University. These steps included
Also within University College, the Honors Program assesses the structure and content of its courses in consultation with its students, instructors, advisory committee, and staff. Faculty selected to teach in the program must first be nominated by departments for their records as scholars and excellent teachers; they must also be committed to the concept of excellence in education. Substitutions of faculty assignments may be made only with the advice and consent of the Dean of the Honors College. To ensure the continuing excellence of faculty teaching Honors College courses, the Dean of the Honors College, in consultation with the appropriate department head and dean, annually reviews all faculty appointments in light of students’ evaluations of their courses.
In addition, instructors of each Honors class provide a written evaluation of each student, that is kept on file and is available to the student as a means of tracking strengths and areas where improvement may be necessary throughout his or her participation in the program. Planning for the future is both an ongoing and evolutionary process, with close oversight by the Honors College Dean. In 2006, the Advisory Committee will begin implementing a more systematic assessment of Honors College students and alumni.
College of Continuing Education and the Extended University (CCE)
The majority of the for-credit classes offered through the College of Continuing Education and the Extended University (CCE) are monitored and assessed by the specific academic departments who provide faculty for and sponsor the courses. One exception is the Missouri State Online Program; in this case, CCE has a more direct role in acquiring assessment information from online students. Another is the online Master of Science in Administrative Studies (MSAS) degree program. The program director has worked in cooperation with CCE staff to design and administer an assessment tool that is used to evaluate all online classes comprising the MSAS Program. This process has resulted in several program improvements. Departments offering intersession and special credit classes are required to submit a class syllabus that outlines the educational goals of the course, anticipated student learning outcomes, and ways in which this learning will be assessed.
Before a class may be offered in one of these non-traditional formats, the proposal and syllabus must be developed by a faculty member and approved by the academic department head and appropriate dean, and in some instances by the Associate Vice President for Academic Affairs. This procedure helps monitor the quality of these classes.
Through the use of student evaluations separate from those that go to the sponsoring departments, the Center for Continuing and Professional Education assesses all non-credit courses. This unit also utilizes community advisory boards to assist in program assessment and evaluation to make sure the services provided address market demands and needs.
Processes for Assessing Student Learning: A Summary
Criteria and Core Components supported in this section include 2c, 3c, 4c, 5a.
At Missouri State, program curricula are assessed in the context of student accomplishments. Assessment of the knowledge, skills, and abilities of students within their majors and in general education provide a basis for maintaining and improving curricular components. In sum, the Plan for Improved Student Learning and Assessment and the Assessment Council guide the coordination of university assessment activities through the CAIS. The Plan emphasizes unit documentation of goals and objectives, associated outcome measures, and the application of the resulting information to improve student learning at the academic and institutional unit levels. CAIS assessment information of all types provides support for improvement in the curriculum, teaching practices, and the university environment as well as support for planning and budgeting decisions. Several processes are used by internal and external groups to assess student learning. These include
The results of these reviews are used to make administrative decisions concerning the continuance of programs, additions of programs, or reallocation of resources. Results of self-studies assist faculty and administrators in making decisions about student experiences, courses, and programs to enhance programs in the future and to determine needed resources. Knowledge gained from assessments of student characteristics, progress, abilities, and employment assists in identifying areas of excellence in student learning and areas that need improvement. This information contributes to the identification of ways to improve teaching, learning, and the environment for learning. By linking this information with the activities of the Academic Development Center, the University provides instructors opportunities to enhance their teaching effectiveness and curriculum design.
The Self-Study Steering Committee notes the following challenges associated with assessing student learning: