Peer Online Course Review

Before diving into Peer Online Course Review (POCR), it helps to first understand the CVC-OEI, the Course Design Rubric, and the Course Finder / Exchange.

CVC-OEI stands for California Virtual Campus – Online Education Initiative.  The CVC-OEI is funded by the Chancellor's Office and consists of faculty and Distance Learning professionals throughout the State, working to ensure that students in California Community Colleges have access to high-quality online courses designed to support their learning and success.

The Course Design Rubric was developed by the CVC-OEI in 2014 and has become a statewide standard for quality online course design.  The rubric incorporates best practices along with regulatory and accreditation requirements; and is used as a “checklist” when conducting Peer Online Course Review.

The Course Design Rubric covers four areas:

1. Content Presentation
2. Interaction
3. Assessment
4. Accessibility

The CVC-OEI based these standards on work already done by @ONE, the CCC Distance Education Coordinator’s group, the Academic Senate for California Community Colleges, the Chancellor’s Office, and a variety of other standards established at local colleges and nationally.

In February 2018, the SJDC Academic Senate unanimously approved adoption and implementation of the CVC-OEI Course Design Rubric at Delta.

The Rubric was also endorsed by the Statewide Academic Senate in 2018.

The CVC Course Finder and Exchange are located at cvc.edu

The Course Finder is a searchable database of fully-online courses offered in the California Community College system.  Courses that go through a local college’s POCR process and fully align to the Course Design Rubric are awarded the CVC’s “Quality Reviewed” badge and appear higher in the Course Finder’s search results.

Courses from a student’s “Home College” will always display first in the Course Finder.  For example, if a student’s Home College is San Joaquin Delta, and they are searching for online courses in Early Childhood Education, Delta’s offerings will be displayed first, followed by online ECE courses at other CCCs.

The Exchange is a feature of the Course Finder that allows students currently enrolled in a California Community College to instantly enroll in online courses offered at eligible colleges without filling out a separate application.  The goal is to support timely completion by providing all students equitable access to online courses and program pathways.

Peer Online Course Review (POCR) was established in 2014 by the CVC’s Professional Development Workgroup – a group of faculty and Distance Learning professionals throughout the State.  It is a formal process through which a faculty member can voluntarily request to have their course reviewed by their peers. The primary goals are to:

1. Improve course design and accessibility
2. Improve student success and completion

In addition, POCR results in:

3. Aligning a course to the CVC-OEI Course Design Rubric
4. Obtaining a “Quality Reviewed” badge from the State
5. Highlighting the course to students throughout California who are using the Course Finder to locate high quality online classes

Currently, over 40 California Community Colleges have a certified local POCR program, with dozens more working toward the goal of certifying their programs and entering badged courses into the Exchange.

In April 2022, after 3 years of program development and a rigorous 1-year application process, Delta College became the 13^th CCC to receive State-certification as a POCR campus.

At Delta, POCR is voluntary.  However, if you wish to receive the State-issued “Quality Reviewed” badge and have your course featured in the Online Course Finder and Exchange, successful completion of Delta’s POCR process is required.

• A deeper understanding of widely practiced online course design standards
• A class that works better for you and for students, saving you both time
• A class that is more polished and professional looking
• An opportunity to learn new approaches to teaching online in a safe and supportive environment
• Membership in a community of practice in which faculty share their challenges, develop new approaches, and support one another
• A class that is more equitable, and accessible to all students
• Improved quality in your other Past participants report that what they learned in POCR positively impacted their other online and in-person classes!

In addition to many of the same benefits faculty will see, such as a user-friendly class that is both accessible and more equitable, students will benefit from:

• A more engaging and inclusive learning experience
• Clear expectations, instructions, and supports
• More varied and authentic assessments
• A course that meets the highest quality standards in the State

Expect a thorough, in-depth review of every aspect of your course. No course will align with every standard in the first review. This can be challenging. It’s hard to hear that “your baby” is not perfect given all the work you’ve put in. Even if your course is close to "alignment," reviewers may still give suggestions to make it even better.  For example: This element is in alignment. Well done! To move into the "exemplary" category, you might consider adding...

Feedback from Delta Faculty who’ve been through POCR:

I really appreciate you helping me get this course up to the highest standards possible. Anything that helps me ultimately helps our students, and that's what this is all about.

Thank you so much for the meticulous review! We learned a lot from the process! Even though the review is over, we are working on bringing aligned areas up to exemplary using the suggestions from the review team.

Click to Meet a Reviewee

POCR will help you build a better class, but it will NOT...

• Be mandatory
• Be shared with your supervisor
• Be part of your faculty evaluation

Per the SJDCTA 2021-24 contract, Article 16.1.2.7:  The peer online course review process (POCR) evaluation shall not be used in a faculty member’s administrative evaluation and is for the purpose of identifying professional development opportunities.

Delta’s dedicated team of POCR reviewers includes the Lead Instructional Designer, two Accessibility experts, and experienced online faculty from all TrACs, the Academic Senate, CTA, and Equity Task Forces.

All reviewers are faculty.

All have completed an intensive 45-hour POCR training course and attend two statewide POCR meetings per year, to ensure a consistent review approach with peers across the CCC system.

The Lead Instructional Designer also serves as Delta’s POCR Lead and has regular interaction with the CVC-OEI to ensure process currency.

Many reviewers have been involved in POCR for 3+ years and have reviewed multiple courses.

They are here to help and support you.

• Interested in becoming a certified POCR Reviewer? See Requirements.

Click to Meet a Reviewer

It takes roughly six months to take a single online class through POCR.

The course author will be assigned a team of three reviewers.  Reviewers follow the four sections of the CVC-OEI Course Design Rubric (summarized below).

Two reviewers will cover Sections A-C, and the third reviewer will always be an Accessibility expert to review Section D.

Each of the four sections of the rubric contain a number of standards that are rated as either Incomplete, Aligned, or Exemplary.  Reviewers are looking for alignment but may offer guidance on how to bring some of the standards to exemplary.

SECTION A: CONTENT PRESENTATION

The 14 elements in this section address how content is organized and accessed in Canvas. Key elements include course navigation, learning objectives, and access to student support information.

SECTION B: INTERACTION

The six elements in this section address both Instructor-to-Student and Student-to-Student interaction.  It's important that instructors initiate interaction with students and encourage interaction among students.

SECTION C: ASSESSMENT

The eight elements in this section address the variety and effectiveness of assessments within the course. Key elements include the alignment of objectives and assessments, the clarity of instructions for completing activities, and evidence of timely and regular feedback.

SECTION D: ACCESSIBILITY

The 16 elements in this section are reviewed to determine if a student using assistive technologies will be able to access the instructor’s course content as required by Section 508 of the Rehabilitation Act of 1973 (also known as “508 Compliance”). The accessibility elements focus on instructor-generated content that is primarily under the control of faculty when developing a course.

EXAMPLES OF REVIEWER FEEDBACK

A11: Learner Feedback – Exemplary: We were happy to see three anonymous "Self-Reflection" assignments for students to consider their understanding of course concepts – as well as two anonymous surveys in Weeks 8 and 17 asking for student feedback on the course itself. Feedback collected early and throughout the semester allows you make changes to the course that current students can still benefit from.  This also gives students extra incentive to give you constructive feedback.  Excellent work here!

Example of issues and comments from a review team:

Issue: Instructor has embedded a 20-minute YouTube video on a Canvas page and given instructions to, “Watch the following video.”

Review team comment: A9: Instructions – Incomplete. Great job embedding the video on your course page. This will help students stay focused on your content! We could not find instructions for students as to what to do with the video. Will students be assessed on this material? What areas/ideas in the video should the students focus on? We recommend giving students instructions such as, “Please watch the following video, taking notes on the topics of ..... as these will be covered in the unit discussion.”

Issue: Instructor has sporadic use of rubrics, and when used, criteria for meeting assessment expectations is unclear or minimal in detail.

Review team comment: C5: Rubrics/Scoring Guide – Incomplete. Students will appreciate the criteria for grading outlined in the Syllabus under "Criteria for Grading Assignments and Assessments," as well as in your discussion rubrics. We only found rubrics for about half the discussions, and suggest including rubrics for all of them, as well as instructions for how students can view the rubrics. For example, "Select the three-dot icon in the top-right corner and then select 'Show Rubric'." To align with this standard, we suggest creating rubrics for all assignments, too, such as Assignment 4. Students find it most helpful when rubrics are fully developed, with descriptions for each rating that clearly state the criteria for meeting competencies.

At the onset of the POCR project, authors must sign an agreement for this "work for hire" and permit other Delta faculty to adopt their master couse shell. See  Content Usage MOU signed by SJDC and CTA.

At the onset of the project, authors are eligible for the first $2,500 payment (see Course Author MOU).

• Sign Faculty Author Agreement

POCR requires a lot of work from both the course author and the review team – but the results for both faculty and students are transformative!  The following section covers the work you can expect before and during review.

Before the process begins, course authors are expected to successfully complete a free, 4-week online class that covers the CVC-OEI Course Design Rubric – Canvas 2.0: Online Course Design.

Once a review has been scheduled, course authors can expect it to take around 6 months from the initial kick-off meeting with their review team, to receiving the “Quality Reviewed” badge from the CVC.  This process assumes a course is being developed from scratch.  Preexisting courses that are being re-designed to align to the rubric may be completed in less time, but it’s not unusual for an instructor’s first time through POCR to take a full semester or more.

If an instructor is accepted into the Peer Online Course Review process, they agree to the following:

• The course must use Open Educational Resources (OER) or free materials and will be flagged at Zero-Textbook Cost (ZTC).
• Course Author successfully completes Canvas 2.0 training on the Course Design Rubric.
• Course Author will develop a “Master Shell” that is not highly personalized and can be reasonably adopted and used by other Delta faculty.
• Course Author aligns the course to all standards in the CVC Course Design Rubric, with support from the office of Regional & Distance Ed.
• The course is peer reviewed by a team of three Delta faculty who serve on the College’s local POCR team; author will follow a schedule and meet agreed upon deliverables.
• For each faculty member who creates a Master Canvas Shell for the CVC Exchange for compensation, their Canvas shell will have an Attribution Noncommercial Creative Commons License, as per MOU Canvas Master Shell Content Usage Agreement.

Phase 1 will last up to 6 weeks and will include:

1. A “Kick-off” meeting will be held between the course author(s), POCR Lead and Accessibility Reviewer to discuss the process phases, expectations, and to answer questions.
2. The course author will be given a new Canvas shell. The POCR Lead and Accessibility Reviewer will already be added with the “Teacher” role, for the purpose of supporting course authors throughout the full duration of the review process.
3. The course author will be provided with resources needed to start the POCR process, including:
   1. A current copy of the CVC-OEI Course Design Rubric
   2. The SJDC Local POCR Checklist, which gives further guidance on what reviewers are looking for
   3. The CVC-OEI’s Course Review Prep Form
   4. A draft 6-month schedule
   5. A follow-up meeting date for 3 weeks out
4. Within 3 weeks of the course shell being made available, the course author will use guidance from the resources provided and Canvas 2.0 training, if complete, to load/develop the following initial content:
   1. 1. Homepage
      2. Pre-Course Welcome Letter/Announcement
      3. Syllabus
      4. Introductory module (also called a “Welcome” or “Getting Started” Module)
      5. ONE (1) full content module, complete with:
         1. An introduction of the topic(s) to be covered
         2. Module-level objectives
         3. Instructor-created lecture material
         4. A variety of multimedia (text, audio, video, and/or images)
         5. Assessment(s)
5. The POCR Lead and Accessibility Reviewer will be available to the course author during this 3-week period to answer any questions or offer guidance.
6. At the end of the 3 weeks, the course author will meet with the POCR Lead and Accessibility Reviewer to give a brief overview of the material. This team will take up to one week to conduct a preliminary review of the initial content and prepare a list of additions/adjustments based on the CVC-OEI Rubric standards. This preliminary “change report” will be given to the course author, and a follow-up meeting will take place as soon as possible to discuss findings and answer questions.
7. If the course author continues with the review process, they will have one week to make the changes to the initial course content outlined in the change report. The POCR Lead and Accessibility Reviewer will be available to answer any questions. At the end of the week, the course author will provide written evidence of how recommended revisions were made in the course. The POCR Lead and Accessibility Reviewer will confirm whether initial content is aligned with CVC-OEI Rubric standards.
8. If this initial content is aligned, a review team of up to three faculty will be assigned and given access the course shell with the “Teacher” role. The course author agrees to meet regularly with reviewers and respond to their inquiries in a timely manner throughout the course of POCR.  “Reviewers” may include the POCR Lead.

Phase 2 will last up to 6 weeks and will include:

1. Developing or re-designing the first half of the course following the recommendations of the review team, within 6 weeks.
2. The instructor agrees to meet with one or more members of the review team (which may include the POCR Lead and Accessibility Reviewer), every 3 weeks to chart progress, ask questions, and review course design. A meeting schedule will be set with the following expectations:
   • At 3 weeks into Phase 2, approximately 1/3 of the course should be complete, including several fully-developed Modules which include: topic introductions, unit-level objectives, instructor-created lecture content, a variety of multimedia, and assessments with detailed instructions and/or rubrics. All content developed by the instructor should be in the Canvas shell, and/or the ”Files” area of the shell (as opposed to a personal Google account or document repository that the reviewers may not be able to access). Any copywritten material should show attribution.
   • At 6 weeks into Phase 2, approximately 1/2 of the course should be complete, including several additional, fully-developed, Modules which include: topic introductions, unit-level objectives, instructor-created lecture content, a variety of multimedia, and assessments with detailed instructions and/or rubrics. All content developed by the instructor should be in the Canvas shell and copywritten material should show attribution.
3. The POCR Lead and Accessibility Reviewer will conduct a brief review to verify whether the above deliverables are met before any further stipend payments (if applicable) are made.

Phase 3 will last 8-12 weeks and will include:

1. Developing or re-designing the second half of the course following the recommendations of the review team, within 6 weeks.
2. The instructor agrees to meet with one or more members of the review team (which may include the POCR Lead and Accessibility Reviewer), every 3 weeks to chart progress, ask questions, and review course design. A meeting schedule will be set.
   • At 6 weeks into Phase 3, the remainder of the course should be complete. A full semester-length course should include all content modules (as detailed in Phase 2), assessments, discussions, rubrics, surveys, resources, syllabus, homepage, and a welcome letter/announcement.
3. Once the course is fully developed, the course author will prepare the course for detailed review by:
   • Removing any unused pages or content.
   • Running the Canvas Link Validator (Course Settings >> Validate Links in Content) and fixing any broken links.
   • Filling out the CVC-OEI’s Course Review Prep Form, including for each rubric standard: a specific example and location of that example.
   • Sending the completed Course Review Prep Form to the POCR Lead.
4. The course author will meet with the full review team to give a brief overview of the course. After this meeting, the course author’s editing access to the shell will be temporarily suspended, so that no changes are taking place while reviewers are looking at the course and drafting feedback.

   The review team will have two weeks to conduct a detailed review and complete their report. The team will meet with the course author(s) as soon as possible to discuss findings.

   The course author has two weeks to make the recommended changes to bring “Incomplete” areas into alignment and provide written evidence of how recommended revisions were made in the course.  The review team has one week to re-review those areas.  From this point in the process, there may be one final iteration of making changes to the course and reviewers checking for alignment.

5. Once the local POCR team signs off on the course being fully aligned to all areas of the CVC-OEI Course Design Rubric, the course will be submitted to the CVC for “Quality Reviewed” badging. Any final stipend payments, if applicable, will be made after the course is badged.
6. The course author agrees to make further revisions as required by the CVC for ongoing alignment.

• Review the suggested 6-month timeline - for POCR projects scheduled for January - June 2023.

• No more than one, two-week extension can be given to any of the three phases. An extension in Phase 3 may not be possible if too close to the project completion date.
• While every effort will be made to work with a faculty member in the POCR process, failure to meet the deliverables outlined in any of the three phases may result in the POCR process for a course being terminated by the Dean of Regional and Distance Education and the Vice President of Instruction.
• If an instructor cannot finish the review process due to illness or unexpected personal or professional concern, they can request one of the following, contingent upon the approval of the Dean of Regional and Distance Education and the Vice President of Instruction:
• Another instructor in the discipline who has completed the necessary training assumes the remainder of the course development/design and receives any remaining stipend payments (if applicable).
• Reschedule the course development/design and review to the subsequent semester.

• The course author’s editing access will be temporarily suspended while any member of the review team is actively reviewing content, including the POCR Lead and Accessibility Reviewer. Editing access will be restored either when the course author is asked to make changes to bring a standard into alignment, or at the conclusion of the POCR process.  This minimizes the need to re-review pages that are being edited.
• The completed course will be submitted to the CVC for badging and will be made available for other Delta faculty to adopt and teach with. The course author should keep this in mind and develop content that is not highly personalized, and can be reasonably adopted and used by others.
• If a course is being developed by two or more course authors, additional meetings may be required with review team members to determine who will work on which parts of the course. Payments, if applicable, may be divided between course authors.

The following amounts and schedule are a sample only and dependent on MOU(s) agreed upon by the District and CTA.

1 Course - $7,500

• $2,500: Course author agrees to the above deliverables and accepts the terms of the MOUs between SJDCCD and SJDCTA: Faculty Grant Stipends – POCR Course Authors and Canvas Master Shell Content Usage Agreement.
•  $2,500: Course author has developed the first half of the course and the POCR Lead and Accessibility Reviewer have singed-off on Phase 2 deliverables being met.
• $2,500: Course author has developed a full semester-length course that has been signed-off by the local POCR Team and badged by the CVC.

As per the Content Usage MOU signed by SJDC and CTA, Delta College faculty who are interested in adopting master course shells must sign an agreement.

• Faculty Adopter Agreement
• Process for Adopting a Master Course

Jennifer Azzaro
Instructional Designer and POCR Lead
Jennifer.Azzaro@deltacollege.edu

Vivie Sinou
Dean of Regional and Distance Education
Vivie.Sinou@deltacollege.edu