Computer Science Assessment Process

for
Computer Science BA, Computer Science BS, and Software Engineering BS
degree programs

Computer Science Department
State University of New York at Oswego
Last update: October 2011

1. Educational Objectives

Our departmental mission statement (last updated in 1996) reads:
The Mission of the Computer Science Department at Oswego is to provide students with a learning environment which will nurture their growth into competent software developers, computer scientists, and information system designers. We strive to maintain modern computing resources, modern software systems and languages, and a diverse faculty capable of engaging students in meaningful learning activities steeped in classic computer science yet applicable to contemporary problems associated with computer and information science.

Our expectations of students graduating from our programs match those of the ACM in its current Curricula Recommendations (section 4.2): Graduates possess an appreciation of the interplay between theory and practice, are adaptable, attend to rigorous thinking, are familiar with common themes and principles, have significant project experience, and can adopt a system-level perspective. Our main differences from typical programs and common curriculum recommendations stem from a greater focus on project-based teaching and learning. Additional information about the structure of our programs may be found in the CS department's 2009 Self-Study.

2. Learning Outcomes

We expect graduates to demonstrate knowledge and abilities within each of the six principal outcomes listed below. Each category defines core competencies, with details that may vary over time to reflect the continually evolving fields of computing. The three degree programs, Computer Science BA, Computer Science BS, and Software Engineering BS share core outcomes, but vary in depth within categories, as discussed below.

  1. Programming. Constructing software that meets its requirements; employing commonly applicable data structures and algorithms; using and creating APIs, components, services, and applications; using programming tools, operating systems, and networked services in the construction of programs.
  2. Formalisms. Manipulating formal systems including logic, analytic and discrete mathematics, and models of computation, that may be used to represent properties of problems and solutions encountered in computing.
  3. Systems. Understanding the structure and properties of common architectural elements of physical computing systems (including arithmetic units, memory, persistent storage, communication, and networks), along with those of layered software systems (including instruction sets, operating systems, runtime systems, and virtual machines) that impact the structure, functionality, and performance of programs executing on them.
  4. Design. Knowing of, selecting and employing the underlying algorithms, patterns, styles, and design techniques applicable in the construction of software solutions to common problems; developing software in a variety of programming languages, computational styles (e.g., object-oriented, functional) and modes (e.g., parallel, event-based, networked); using abstraction, decomposition and composition techniques (e.g., modules, frameworks) to create larger software components and programs from smaller ones; adapting designs to target systems with varying architectures, resources, features, and limitations.
  5. Analysis. Using formalisms to understand software systems and evaluate their correctness, performance, and resource (memory, processors, communication, etc) requirements; using empirical analytic methods (including testing, simulation, and measurement) to augment formal methods; evaluating the impact of software on its users and society, including matters of usability, security and privacy based on an understanding of the ethical standards of the profession.
  6. Process. Planning, developing, and delivering software projects, especially those developed by teams of developers; performing scheduling, quality assurance, and requirements collection; applying analytic techniques to software verification, testing, and maintenance; communicating knowledge to others in the course of collaboration, tutoring, presentations, and documentation; engaging in ongoing professional development, life-long learning and process improvement.

Depth Requirements

The CS BS and SE BS programs impose further depth requirements in outcomes, corresponding to their greater numbers of courses required (rather than allowed as electives) compared to the CS BA program:

  • CS BS Depth. Designing, constructing, and analyzing solutions based on algorithms, protocols, representations, and related design techniques, including new or experimental ideas, applicable to problems encountered in a signficant area of computing such as artificial intelligence, middleware, embedded systems, or gaming.
  • SE BS Depth. Managing software development lifecycles, assigning development tasks and roles, performing requirements analysis, estimation, configuration, and system maintenance and evolution; systematically applying quality assurance; understanding problems and solutions encountered in the development of large scale systems entailing networking, concurrency, and/or persistent storage, that are developed and maintained by multiple groups.

Relationship to ABET and ACM

While sometimes expressed differently, program outcomes are consistent with the "a-k" criteria used by ABET for computing and engineering programs, as noted below in brackets. These are in turn consistent with current ACM curricular recommendations. The ACM guidelines include more detailed outcomes that we use to guide assessment of component learning objectives (see Section 3).

  1. an ability to apply knowledge of mathematics, science, and engineering [design, analysis]
  2. an ability to design and conduct experiments, as well as to analyze and interpret data [analysis]
  3. an ability to design a system, component, or process to meet desired needs within realistic constraints [design]
  4. an ability to function on multi-disciplinary teams [process]
  5. an ability to identify, formulate, and solve engineering problems[programming, design, analysis]
  6. an understanding of professional and ethical responsibility [analysis]
  7. an ability to communicate effectively [process]
  8. the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context [analysis]
  9. a recognition of the need for, and an ability to engage in life-long learning [process]
  10. a knowledge of contemporary issues [process]
  11. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. [programming]

3. Assessment

Our program outcomes cover knowledge and abilities that span multiple courses. Appendix I lists more specific per-course learning objectives that together are used to assess the more general outcomes. Our primary assessment relies on relevant student work products (programming projects and associated artifacts) and exams, quizes, or test items from these courses. When applicable, these rubrics match detailed objectives recommended in current ACM guideline documents (mainly the 2008 update). In keeping with the project-based focus of our curriculum, we rely on work products when possible, to better assess that students can effectively use knowlegde in developing software.

In addition to course-based assessment, our upcoming SE BS program evaluation will also entail surveys of graduates and their employers, as well as consultation with the Engineering Advisory Committee.

Historical note: Our department initially used a single comprehensive capstone exam. However, the results were too contaminated by sampling and recency effects to serve as a reliable instrument. The department does not yet have a full evaluation using the current process.

4. Evaluation

Our assessment process is designed to be consistent with ABET recommendations, and to be useful in the continuous improvement of our programs. We plan to perform a full ABET-compliant assessment of at least the SE BS program within the next two years. This effort will then be used to further improve curriculum and assessment.

Appendix I : Per-Course Objectives

The following (in-progress) lists specific learning objectives from each course contributing to associated program outcomes. The listed objectives do not necessarily cover all requirements of any given course, only those that contribute to program outcomes. Some sets of objectives overlap because not all courses are required of all students, yet any set of electives must together provide minimal compliance. In our project-based curriculum, the entries for most electives are similar because they entail development of significant software projects, collaboration, written and/or oral reports, and they cover current trends in their fields and their impact. Graduates across all programs take approximately ten courses involving programming; each contributes about 10% of the total coverage, but most of the contributions of earlier courses are prerequisite to those in later courses where they are measured. We do not include here include cognate requirements in Math and Science or courses colisted with other programs,

Key for bracketed course annotations:
C: Required for CS BA and BS programs
S: Required for SE program
A: One of two choices for requirements (applies only to CSC222/322)
T: Required for a CS or SE BS concentrate
E: free elective

Courses in common core for all programs

CSC212 CS1 [C,S}
1. Programming
  • Produce programs entailing elementary constructs including: variables, expressions, conditionals, loops, functions, methods, objects, and arrays
  • Produce correct conditional statements for a collection of related conditions
  • Produce an iterative structure that makes progress and correctly terminates in the presence of the required condition
  • Implement a program that can traverse a recursive data structure
  • Use an integrated development environment (IDE) to build and run a program
2. Formalisms
  • Simplify a given boolean expression
3. Systems
4. Design
  • Design a collection of classes with associated attributes and methods that collectively contribute to a solution
  • Identify the appropriate parameters and return types necessary to support method operation
  • Correctly specify bounds on user input
5. Analysis
  • Understand the impact of infinite loops on available computing resources
6. Process
  • Package a solution as a deployable program.
  • Cite the work of others that informs/contributes to their finished product
CSC221 Foundations of CS [C, S]
1. Programming
2. Formalisms
  • Formulate and manipulate logic expressions for a variety of applications
  • Describe and recognize basic proof techniques
  • Understand, define and manipulate discrete structures such as functions, relations, and sets
  • Understand recursion as a basic paradigm for computing with functions
  • Perform proofs by mathematical induction
  • Understand formal definitions of machine models.
  • Prove the equivalence of languages described by finite state machines and regular expressions.
  • Construct pushdown automata and the equivalent context free grammars.
  • Prove the equivalence of languages described by pushdown automata and context free grammars
  • Use the pumping lemma and closure properties to prove particular problems cannot be solved by finite automata.
  • Describe characteristics of a Turing machine.
  • Explain the theoretical limits on computational solutions of undecidable and inherently complex problems.
3. Systems
4. Design
  • Design finite automata to recognize string patterns
  • Construct finite automata with specific properties and the equivalent regular expressions.
  • Describe simple recursive data structure and algorithms
5. Analysis
  • Understand induction as a basic paradigm for proving properties of recursive functions
6. Process
CSC241 CS2 [C, S]
1. Programming
  • Implement programs that entail common data types including lists, stacks, queues, trees.
  • Implement programs that entail sorting and searching algorithms.
2. Formalisms
  • Explain why big-O analysis ignores constant factors
  • Explain the relation between nested loops and polynomial complexity.
3. Systems
4. Design
  • Design classes and APIs for collections and aggregates
  • Use existing APIs in the design of application programs
  • Design a recursive solution to a given problem
  • Use an event-driven framework to design a program supporting user interaction
5. Analysis
  • Characterize the time complexity of common data structures using big-O notation.
  • Emprirically compare the performance of multiple solutions to a common problem
6. Process
  • Develop at least one program in multiple stages, each with added requirements, functionality, and tests.
CSC365 Data Structures and Algorithms [C, S]
1. Programming
  • Produce programs that entail the construction and/or use of balanced trees, hash tables, persistent data structures, and graph algorithms.
2. Formalisms
  • Characterize asymptotic properties of algorithms with constant, linear, polynomial, logarithmic, and exponential complexity.
3. Systems
  • Describe the properties of file systems and storage media supporting persistent data structures
4. Design
  • Design APIs for representing aggregated data.
  • Describe alternative representation strategies for trees, graphs, and tables, the algorithms they support, and their suitability for various applications
  • Describe general categories of algorithms including brute-force combinatorial, greedy, divide-and-conquer, and stream (or String) processing.
  • Describe common graph algorithms, including those for traversal, path, and spanning tree computations
5. Analysis
  • Analyze the time and space complexity, and empirical tradeoffs of alternative data structures with similar functionality.
  • Identify differences among best, average, and worst case behaviors of an algorithm.
6. Process
  • Develop at least one layered program that entails construction and testing of a component providing a data representation, and an application that uses it.
CSC344 Programming Languages [C, S]
1. Programming
  • Process some representation of code for some purpose, such as an interpreter, a serializer, an expression optimizer, or a documentation-generator.
  • Write programs using multiple encapsulation mechanisms, including function closures and object-oriented interfaces, in multiple programming languages.
  • Implement components that avoid assigning to mutable state or considering object identity.
  • Write event handlers for use in reactive systems, such as GUIs.
2. Formalisms
  • Describe the syntax of a fragment of a programming language using a context free grammar
  • Describe a fragment of a type system in terms of Set operations.
3. Systems
  • Explain benefits and limitations of language support for automatic memory management (garbage collection), and reason about memory-management errors including memory leaks and dangling-pointer dereferences.
4. Design
  • Define and use operations on aggregates (loops, iterators, mapping operators, etc) using idioms supported in multiple programming languages
  • Contrast object-oriented inheritance (code-sharing and overriding) from subtyping (the idea of a subtype behaving like a supertype).
5. Analysis
  • Rigorously determine that a version of a component (a) without mutable state (b) with mutable state, ensures a given property or result.
  • Distinguish a language definition (what constructs mean) from a particular language implementation (compiler vs. interpreter, run-time representation of data objects, etc.).
  • Resolve a question about an unfamiliar situation (for example, about naming scopes) by reference to a language specification document.
6. Process
  • Explain the benefits and limitations of static typing in different modes of software development, including prototyping and quality assurance.
  • Describe criteria for choosing a programming language for a given software development project.
CSC380 Software Engineering [C, S]
1. Programming
  • Effectively use the API of a target platform to deliver required functionality
  • Effectively use collaboration tools such as wikis and code repositories
2. Formalisms
  • Characterize the different models supported by UML or related modeling notations.
3. Systems
4. Design
  • Use the Unified Modeling Language (UML) to produce diagrams of classes and object interaction in their systems
  • Generate Use Cases that describe user interaction with the system
  • Describe the challenges associated with scaling the team project if the users are increased by several orders of magnitude
5. Analysis
  • Create a test plan to effectively isolate defects and verify correct operation of the final project
  • Produce a correct UML interaction (State, Sequence, Communication, etc.) diagram given a problem specification
  • Identify and use accepted practices for code testing and inspection
  • Identify and describe the stakeholders associated with software projects and their motivations
  • Identify the security technologies (encryption, access controls, etc.) necessary for their projects
  • Prepare a usability testing protocol that appropriately involves human subjects
6. Process
  • Produce software working in a group of developers
  • Produce a requirements document that articulates verifiable system attributes
  • Accurately record efforts contributing to a project
  • Produce a Build Plan that outlines a series of incremental deliverables
  • Articulate the differences among accepted development paradigms such as Top down, Bottom up, Agile
  • Provide regular updates to project sponsor on the status of the development
  • Prepare and present regular builds demostrating incremental functionality
  • Present a final oral report that demonstrates functionality of the project
  • Produce a written narrative describing the project and reflecting on the process.
CSC222 Computer Organization [A]
1. Programming
  • Produce programs written in an assembly language for a given instruction set that entail elementary constructs including: variables, conditionals, loops, and subroutines.
2. Formalisms
  • Describe given computer arithmetic operations using boolean algebra
3. Systems
  • Describe the basic units and architecture of Von Neumann machines and the steps entailed in executing programs
  • Describe the design choices surrounding pipelining, superscalar CPUs and related CPU design features and their impact on program execution
  • Describe the design choices surrounding caches and memory hierarchies and their impact on program execution
  • Describe the design choices surrounding DMA, interrupts and related IO control
  • Describe the tradeoffs and interactions between hardware and software control of system functionality such as memory address management
  • Describe the differences between non-virtualized and virtualized operationing systems, and their impact on program execution.
4. Design
5. Analysis
6. Process
CSC322 Systems Programming [A]
1. Programming
  • Produce programs written in a low-level systems programming language (such as C) entailing elementary constructs including: variables, expression on bits, conditionals, loops, and operating system calls.
  • Implement a program that manages its own memory and/or IO processing with minimal reliance on system or run-time support.
  • Implement a program that entails control flow (for example, interrupts, traps, non-local jumps) not typically supported in higher-level languages.
2. Formalisms
3. Systems
  • Describe the underlying representation choices for given data types (including pointers)
  • Describe the common components and structure of common run-time support systems including linkers, buffered IO, and shared libraries.
4. Design
  • Construct the API (for example as a C header file) for a component with given functionality.
5. Analysis
  • Measure the performance of programs that have similar functionality but rely on alternative system functions.
6. Process

Courses required for a CSBS concentrate or SEBS

CSC338 Robotics [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC366 Comp Models of Cognitive Process [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC375 Parallel Computing [T]
1. Programming
  • Use mutual exclusion to avoid a given race condition
  • Write a program that correctly terminates when all of a set of concurrent tasks have completed
  • Use a properly synchronized queue to buffer data passed among activities
2. Formalisms
3. Systems
4. Design
  • Distinguish using computational resources for a faster answer from managing efficient access to a shared resource
  • Explain when and why multicast or event-based messaging can be preferable to alternatives
  • Describe the relative merits of optimistic versus conservative concurrency control under different rates of contention among updates
  • Explain why checks for preconditions, and actions based on these checks, must share the same unit of atomicity to be effective
  • Describe at least one design technique for avoiding liveness failures in programs using multiple locks or semaphores
5. Analysis
  • Give an example of a scenario in which blocking message sends can deadlock
  • Give an example of an ordering of accesses among concurrent activities that is not sequentially consistent
  • Give an example of a scenario in which an attempted optimistic update may never complete
  • Distinguish data races from higher level races
  • Explain why synchronization is necessary in a specific multithreaded program
  • Describe how data distribution/layout can affect an algorithm’s communication costs
  • Write a test program that can reveal a concurrent programming error; for example, missing an update when two activities both try to increment a variable
6. Process
CSC416 AI Programming Languages [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC420 GUIs [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC435 Web Services [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC436 Networked Systems [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC444 Compiler Construction [T]
1. Programming
  • Implement a scanner and parser for a complete small language, and/or modify/extend one for a full production language.
  • Implement static analyses, including type-checking and symbol resolution to identify binding occurrences.
  • Implement code generation for a given target -- a machine, virtual machine, or a lower-level programming language.
2. Formalisms
3. Systems
4. Design
  • Design software accommodating the essential steps for compiling source code to a target representation.
  • Use formal grammars to specify the syntax of languages
  • Use declarative tools to design or generate parsers and scanners
  • Identify parsing issues introduced by ambiguity, associativity, precedence
  • Design an abstract syntax representation supporting a set of static analyses.
  • Identify static semantics issues introduced by various scoping rules and type systems.
  • Characterize target-platform characteristics including registers, instructions, bytecodes, dynamic memory management, layouts for objects and activation records
  • Represent program dynamics as data flow using basic blocks, control-flow graphs,static single assignment, and/or alternative approaches
  • Design data flow solutions for problems including available expression analysis and liveness analysis
  • Understand alternative approaches to implementing non-elementatry control flow (such as exceptions) and invocation (such as object-oriented method dispatching).
5. Analysis
  • Validate conformance of a given compiler action to a given property required in a language specification, using formal and/or empirical techniques.
6. Process
  • Deliver a working compiler in stages.
CSC445 Computer Networks [T]
1. Programming
  • Write a program that performs any required marshalling and conversion into message units such as packets to communicate interesting data among two hosts, and measure its performance when run on different hosts or networks.
  • Implement a simple server; for example a spell checking service
  • Implement a reliable protocol (for example, a subset of TCP) using an unreliable one (for example, UDP).
  • Implement a distributed program (for example, a shared whiteboard) entailing group communication.
2. Formalisms
3. Systems
4. Design
  • Explain why synchronization constructs such as simple locks are not useful in the presence of distributed faults
  • Give examples of problems for which consensus algorithms such as leader election are required
  • Explain why no distributed system can be simultaneously consistent, available, and partition tolerant
  • Explain the tradeoffs among overhead, scalability, and fault-tolerance when choosing a stateless design versus others for a given service
  • Describe the scalability challenges associated with a service growing to accommodate many clients, as well as those associated with a service only transiently having many clients
5. Analysis
  • Distinguish network faults from other kinds of failures
  • Measure the observed throughput and response latency across hosts in a given network
6. Process
CSC454 System Simulation and Virtual Worlds [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC455 Computer Game Programming [T]
1. Programming
  • Implement a computer game.
2. Formalisms
3. Systems
4. Design
  • Design a computer game.
  • Understand the historical origins of computer games and their impact on game design
  • Understand interaction mechanisms and their use in games.
5. Analysis
6. Process
  • Apply knowledge of game development practice in a team that includes non-programmers
CSC459 Database Systems [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC466 AI [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC470 Computer Graphics [T]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC480 Software Design [S]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC495 SE Project I [S]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC496 SE Project II [S]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process

Free Electives

CSC320 Numerical Methods [E]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC332 Cryptology [E]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC443 Operating Systems [E]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process
CSC465 Algorithms [E]
1. Programming
2. Formalisms
3. Systems
4. Design
5. Analysis
6. Process