POCR Online Course Reviewers

Experience

POCR reviewers must have successfully taken at least one of their fully online courses through POCR and received a “Quality Reviewed” badge from the California Virtual Campus (CVC).

Training & Certification

Reviewers must have successfully completed the following training courses. Contact Jennifer.Azzaro@deltacollege.edu for registration details.

♦ Canvas 2.0: Online Course Design (20 hours)

♦ Peer Online Course Review (45 hours)

Delta’s dedicated team of POCR reviewers includes the Lead Instructional Designer, two Accessibility experts, and experienced online faculty from a variety of TrACs, the Academic Senate, CTA, and Equity Task Forces.

All reviewers are faculty.

All have completed an intensive training course and attend two statewide POCR meetings per year, to ensure a consistent review approach with peers across the CCC system.

The Lead Instructional Designer also serves as Delta’s POCR Lead and has regular interaction with the CVC-OEI to ensure process currency.

Many reviewers have been involved in POCR for 3+ years and have reviewed multiple courses.

You will be paired with an experienced reviewer to help guide you through your first review!

 Click to Meet a Reviewer

Confidentiality

Peer Online Course Reviews will NOT be:

♦ Mandatory

♦ Shared outside the assigned review team

♦ Part of the faculty evaluation process

Per the SJDCTA 2021-24 contract, Article 16.1.2.7:  The peer online course review process (POCR) evaluation shall not be used in a faculty member’s administrative evaluation and is for the purpose of identifying professional development opportunities.

Reviewer Pool and Course Assignments

Individuals who meet the above requirements will be added to the pool of available Reviewers, however this does not guarantee a course review. Review assignments will come from the Office of Regional and Distance Education and may be accepted or declined.

Staying Current with Norming Meetings

In order to remain in the pool of available reviewers, both active and inactive reviewers must participate in a minimum number of State and Local “Norming” meetings per year (usually 2 but subject to change). Regular norming helps to ensure currency in the CVC-OEI Rubric standards and understanding of what to look for when reviewing courses.

Course Teams, Timeline, and Workload

Each review team consists of 4 reviewers: the POCR Lead, Accessibility Reviewer, and two additional faculty. It takes roughly six months to take a single online class through POCR, using a 3-phase process:

Phase 1

(6 weeks): Kick-Off Meeting, Develop Initial Content, and Preliminary Review

♦ Course Authors work closely with the POCR Lead and Accessibility Reviewer, only.

Phase 2

(6 weeks): Develop the First Half of the Course and Informal Review

♦ Two additional faculty reviewers are assigned and required to attend ~2 check-in meetings with course author and review team. Informal review is done by the POCR Lead and Accessibility Reviewer.

Phase 3

(8-12 weeks): Develop the Second Half of the Course and Formal Review

♦ About 4 meetings, plus formal review (which may necessitate additional meetings). Formal review includes significant time looking over every part of the course (Modules, Assignments, Assessments, Discussions, Homepage, Syllabus, Announcements, Rubrics, Resources, and multimedia).

♦ Reviewers must fill out the CVC-OEI Rubric document and include substantive feedback that is both collegial and evidence-based.  Reviewers confer with members of the team assigned to that course and ensure review is completed by the timeline set by the POCR Lead.

♦ Reviewers join the project with the understanding that each course may require several iterations of reviews and feedback.

For more detail, see our sample POCR Timeline.

Reviewers should expect to conduct a thorough, in-depth review of every aspect of the course. Review requires care. No course will align with every standard the first time through. This can be challenging for both course authors who take pride in their work, and reviewers who need to remain constructive, supportive, and encouraging. It’s not easy for authors to hear that multiple areas of a course are “Incomplete”.  Even if a course is close to "Alignment," reviewers should still give suggestions to make it even better.  For example: This element is in alignment. Well done! To move into the "exemplary" category, you might consider adding...

Feedback should always be:

♦ Substantive: Even if a rubric element is “Aligned,” the CVC requires more than simply re-phrasing the rubric language – and certainly more than, “Aligned: great work!” Feedback should have substance and speak to the uniqueness of the course.

♦ Collegial: Most feedback statements should attempt to start with something positive, even if the element is found to be “Incomplete” and in need of revision. Acknowledge the course author’s effort and intent.

♦ Evidence-based: Reviewers must cite the places they checked – and evidence they found – regardless of whether an area is Incomplete / Aligned / Exemplary.

♦ Supportive: Offer encouragement and suggestions for how the course author might bring an area into alignment.

♦ Confidential: Reviewer feedback should always be kept between the course author, review team, and POCR Lead.

See the next section for specific feedback examples.

Conferring With Teammates

Reviewers are expected to confer with their teammates and use various forms of State and Local guidance, paired with evidence from the course, to ensure accurate findings for each of the CVC-OEI Rubric standards.  When in doubt, team members may seek further guidance from the POCR Lead.

Tips

♦ The review team is just that, a team! Be sure you agree in your shared feedback and use We statements throughout the review. “We looked in Modules 2, 4, 6, and 8…”  “We were unable to find…”

♦ Critique the course, not the course author. Remember to ground your feedback in the rubric language and evidence.

Incomplete

Issue: A9 (Instructions) marked “Incomplete” because the instructor has embedded several 10–20-minute YouTube videos on Canvas pages and given instructions to, “Watch the following video.”

Substantive review team comment:

A9: Instructions – Incomplete.

[Collegial Acknowledgement] Great job embedding videos on your course pages. This will help students stay focused on your content! [Evidence-based] We looked on the pages that include videos (below) as well as the Module Introduction pages, but could not find instructions for students as to what to do with the videos.

♦ Module 2: Intro page and Emotional Development page

♦ Module 3: Intro page and Behavioral Development page

♦ Module 4: Intro page and Social Development page

[Supportive Suggestions] Will students be assessed on the videos? What topics/ideas in each video should students focus on? We recommend giving instructions such as, “Please watch the following video, taking notes on the topics of .... as these will be covered in the unit discussion.”  Adding context to the videos will help students get the most out of viewing them.

Aligned

Substantive review team comment:

A1: Placement of Objectives – Aligned.

[Evidence-based] We found Unit-level outcomes in each of the 5 course Modules, at the top of each page that specifically addresses that objective. [Collegial Acknowledgement] Students will find this direct mapping of content to objectives very helpful!  [Supportive Suggestion] To bring this area to “Exemplary,” consider listing the unit-level objectives on the first page of each module, titled: "Unit X Introduction." Clear and consistent placement will help students anticipate what to expect as they begin each new Module.

Exemplary

Substantive review team comment:

A3: Alignment of Objectives – Exemplary.

[Evidence-based] We checked each module and found that all contain a consistent set of resources that support unit objectives (including video lectures, slides, notes, and learning resources). [Collegial Acknowledgement] Students will benefit from the explicitly clear connections on the Module Introduction pages, where unit objectives are mapped to both course objectives AND assessments within the unit. [Further Evidence] For example:

Unit 3, Objective #2: Calculate and interpret a Coefficient of Determination.

Mapping: Supports Course Objectives 2, 4, and 5.  Assessed in the Unit 3 Worksheet and Exam.

Both the Unit 3 Worksheet and Exam require calculation of a Coefficient of Determination.  Students will learn how to do this in the Unit 3 videos, slides, and notes.

Prior to formal review, the course author will provide the assigned review team with a completed Course Review Prep Form, citing examples of where the team can find alignment.  This document serves as a starting point for reviewers to begin, but is not an exhaustive checklist.

The POCR Lead will provide each team member with a shared  CVC-OEI Rubric document for the course, which can be edited and saved in a shared document repository such as Box.

A stipend of $1,500 per course will be approved upon completion of review, meaning all areas of the Rubric were found to be either “Aligned” or “Exemplary” and the assigned team has signed off on the course being ready to submit to the CVC for “Quality Reviewed” badging.

This stipend is effective through June 2025 (as per MOU).

If a Reviewer cannot finish the review process due to illness or unexpected personal or professional concern, they can request, in writing, replacement by another individual from the pool of available reviewers, contingent upon the approval of the POCR Lead and Dean of Regional and Distance Education.  Division of payment, if applicable, will be determined by the Office of Regional & Distance Education and CTA.

♦ Active Reviewers may simultaneously have their own course(s) undergo review.

Jennifer Azzaro
Instructional Designer and POCR Lead
Jennifer.Azzaro@deltacollege.edu

Vivie Sinou
Dean of Regional and Distance Education
Vivie.Sinou@deltacollege.edu