Executive Summary
This analysis presents a review and comparison of two instructional technologies for administering and digitally grading online and in-person assessments: Crowdmark and Gradescope. We tested both instructor and student workflows for creating, submitting, and grading assessments using Crowdmark and Gradescope integrated with a test course in D2L. Our evaluation criteria included ease of use, features available, accessibility, and flexibility.We found some key similarities:
- Remote and in person assessments are supported, with multiple question types.
- Grading is done by question rather than by student for more consistency.
- Multiple graders can grade assignments, such as co-instructors and teaching assistants.
- Grades are synced automatically with the gradebook in D2L Brightspace.
The primary differences between these two are:
- Crowdmark can assign assessments according to sections and a drag and drop functionality is available for rubric comments.
- Crowdmark emails students when assessments become available and can accept more file types as well as rotate files more easily.
- Gradescope allows for time extensions at the course level as well as for each assessment and allows for grading the assessments before the due date.
"The second-best way to grade exams" by ilmungo is licensed under CC BY-NC-SA 2.0
Methods
We tested both instructor and student workflows for creating and submitting assessments using Crowdmark and Gradescope integrated with a test course in D2L. Sample assignments were created for the remote assessments that included all of the available question types (i.e., upload file, enter text, multiple choice, etc.). Using separate accounts, we assigned the assessments as an instructor, submitted the assessments as a student, then returned to the instructor account to grade the assessments and sync the grades to our D2L test course.Findings
Key Similarities:
Both Crowdmark and Gradescope offer keyboard shortcuts for faster grading; allow late submissions, group submissions, and enforced time limits; and allow for grading by question instead of by student as well as multiple graders such as teaching assistants. Assignment submissions can include pdf or image upload, free response/short answer in a text box, or multiple choice/multi select type questions (with bubble sheets) for online assessments.For both tools, students can upload one PDF and then drag and drop each page to match each question for remote assessments, while instructors can scan and upload student submissions in batches for in person assessments. Both tools will also attempt to split a batch PDF into individual student submissions.
Key Differences:
Accessing Tools
Students have to login to Crowdmark through the Crowdmark website. This link can be added to D2L Brightspace and opened in a new, external web page. The Crowdmark sign-in prompts students to select their institution and then uses students’ Brightspace login. Gradescope can be added to D2L Brightspace as an External Tool in a D2L content module. This allows students to access Gradescope within D2L as an embedded website within the D2L page, instead of as an external page, and does not require any additional login.Creating Assessments
When creating assessments in Crowdmark, instructors choose between administered (in person) assessments that instructors will upload or assigned (remote) assessments that students will upload (Figure 1). Administered assessments can include bubble sheets for multiple choice questions. Assigned remote assessments can include file upload, text entry responses, or multiple-choice questions (which are automatically graded).When creating an assignment in Gradescope, the assignment type must be chosen first. Then, for the first three assignment types, the submission type is designated as either the instructor or the students (Figure 2). Although Exam/Quiz and Homework/Problem Set are offered as two different choices, they actually have the same options and essential functions. There are no further options if the instructor will be uploading the assessments, but other options are available if students will be uploading. Submissions can be variable length, where students submit any number of pages and indicate the pages where their question responses are, or fixed length where students submit work where answers are in fixed locations (like worksheets). Instructors can also allow students to view and download the assessment template if desired. Multiple choice assignments can be created with printable bubble sheets that either instructors or students can upload. Programming assignments are available, which Crowdmark does not support, and they can be automatically or manually graded.
Figure 1: Assessment types available in Crowdmark.
Figure 2: Assessment types available in Gradescope.
Both tools have the ability for students to take online quizzes. Both have multiple choice and multi select that are auto-graded, and both have free response and file upload that are NOT auto-graded. Gradescope supports short answer questions which are auto-graded, but Crowdmark only has free response questions.
For assignments that students will upload, instructors must input text or upload a document for each individual question in Crowdmark. It is possible for an instructor to upload one document in the instructions field which contains all of the assignment questions and then simply enter numbers in the text boxes for each question, rather than the text of each question. Gradescope only requires one document to be uploaded. Each question is then identified by dragging a box around each question area on the page and a question title must be entered.
Assigning & Distributing Assessments
For courses with several sections, Crowdmark allows assessments to be assigned to specific sections rather than the entire course. To approximate this feature in Gradescope, an instructor would have to create separate Gradescope courses or duplicate assignments and direct students to the appropriate version for their section.Both tools allow instructors to set individual accommodations for each assignment to customize due date, lateness penalty, or time to complete. However, Gradescope also allows course-wide extensions for students, where extensions can be added for all assignments to customize time limits (multiply time by x or add x minutes) and due dates. Crowdmark requires accommodations to be made in the submission area for each assignment. It does not support course-wide accommodations.
When an assessment is assigned and released to students, Crowdmark sends a notification email to students, where Gradescope only sends an in-platform notification. Gradescope does send a confirmation email when students successfully submit an assignment. Both tools give instructors the option to send a notification email when returning student work.
Submitting Assessments
For in-person assessments, Crowdmark can include a QR code on assignments to ensure that every page of student work is correctly matched to the appropriate student for grading. The QR code can be manually scanned and matched to each student using an app as the assignment is turned in, or instructors can use automated matching (beta) to include a form field where students write their name and ID number for automated character recognition to identify the student and match them to that assignment’s QR code. Gradescope is developing a feature to create a unique label for each copy of an assignment and add that label to each page, but this is not currently available.Submitted file types are more flexible in Crowdmark, which can support PDF, JPEG, PNG, and iPhone photos, any of which can be rotated after submission. Gradescope accepts only PDFs or JPEGs and only PDF pages can be rotated. This means that Crowdmark offers much more flexibility in scanning software and orientation. Gradescope does have a built-in PDF scanner for iOS devices to circumvent format issues and allow seamless upload. Both tools assume that image submissions are of work associated with a single question. All work can be scanned into a single PDF for upload and each page then manually associated with each question in the assignment. In both tools, the student selects which question(s) are associated with each page(s), where multiple questions may be on a single page or multiple pages may be associated with a single question.
Crowdmark allows for group submissions when either the instructor or the students scan and upload the assessments. This ability to match multiple students to one assessment allows for two-stage exams, collaborative lab reports, or other group assignments. Gradescope only allows group submissions when students scan and upload assessments, although online assignments also allow group submissions.
Grading Assessments
Assignments can be graded immediately after students have submitted them in Gradescope. Crowdmark does not allow grading to be done until the due date has passed.In Crowdmark, all feedback comments created for each question are stored in a comment library which can be reordered easily by dragging a comment to the desired location. There is no limit on the number of comments that can be dragged and dropped onto each student’s submission. Crowdmark comments can have positive or negative points attached to them, but specifying points is not required. Gradescope does not allow for dragging and dropping multiple comments; however, text annotations are saved for each question and several can be applied to each submission. The separate rubric comments must be associated with positive or negative points for each question. The rubric type can be either negative scoring, where the points are subtracted from 1.0, or positive scoring, where the points are added to 0. Score bounds can also be set, with a maximum of 1.0 and a minimum of 0. While it is possible to select more than one rubric comment, only one comment can be added as part of a “submission specific adjustment” which can include an additional point adjustment.
Crowdmark sends grades to D2L and automatically creates the grade item in the gradebook. Gradescope requires that the grade item be created first, then associated with an assignment, before sending grades is possible.
|
Topic |
Crowdmark |
Advantage |
Gradescope |
|---|---|---|---|
|
Accessing Tools |
Must access through separate website; sign in to Crowdmark via Brightspace |
 |
Can add External Tool to D2L module and it can be accessed within D2L (embedded website into page) |
|
Creating Assessments |
Upload PDF and designate where questions are for administered assessments that instructors upload (drag question number to location on page) |
|
Upload PDF and designate where questions are by dragging boxes on the page for fixed length exam/homework that students upload or an administered exam/homework that instructors upload |
|
Must input or upload individual questions manually when creating remote assessments that students upload (but instructor can upload PDF in directions area and just enter Q1, Q2, etc. in text boxes) |
|
Must input question titles separately for variable length submissions that students upload, but questions are designated by dragging box over location on page (no need to enter text of question in Gradescope) |
|
|
Assigning & Distributing Assessments |
Can assign assessments to a section rather than entire course |
 |
Cannot assign assessments to a section; must create separate course or duplicate assignments and instruct students which one to submit |
|
Add time for accommodations for each assessment only (customize due date, lateness penalty, or time to complete) |
 |
Add extensions at course level and/or for each assessment (multiply time by x or add x minutes) |
|
|
Students always receive email when new assignments are ready to be completed |
|
Students are not notified when new assignments are ready; but students do receive email when they have submitted an assignment, and instructor has option to send email once the assignment is graded |
|
|
Submitting Assessments |
QR codes on printed work for in person administered assessments (can also use app to match assessments to students when scanning) |
|
Create printouts (beta) for in person assessments; give each student a copy of the assignment with a unique label on each page (this tool is NOT yet available) |
|
iPhone photos supported; can accept PDF, JPG, or PNG (and can rotate any file) for remote assignments submitted by students |
 |
iPhone photos not supported; accepts PDF or JPG only (can only rotate PDFs) for remote assignments submitted by students; multiple files and any file type accepted for online assignments |
|
|
Allows for group submissions whether students or instructors are uploading assessments (i.e. match multiple students to one assessment) |
|
Allows for group submissions only if students are uploading assessments, but also available for online assignments |
|
|
Grading Assignments |
Must wait until due date to begin grading remote assessments |
|
Online assignments can be graded immediately |
|
Drag and drop any number of comments from comment library for each question |
|
Can apply one previously used comment for each submission separate from rubric; cannot select or drag and drop multiple comments, but can add multiple previously used text annotations for each question |
|
|
Comments can have positive or negative points attached to them, but specifying points is not required |
|
Comments must have associated points (positive, negative, or 0) for each question; can change rubric type from negative scoring (points subtracted from 1.0) to positive scoring (points added to 0) as well as enable/disable score bounds (max of 1.0 and min of 0) |
|
|
Grades sent to D2L automatically with no need to create grade item first |
|
Grades sent to D2L automatically but must create grade item first |
MSU Usage Data
We explored the usage of each tool at MSU to determine if there was a perceptible trend towards one tool over the other. The total number of courses created in each tool is fairly similar (Table 2). Interestingly, the total number of students enrolled in those courses is much higher in Crowdmark, while the number of assessments administered is higher in Gradescope.|
Crowdmark |
Gradescope |
|
|---|---|---|
|
Courses |
322 |
292 |
|
Students |
25,322 |
14,398 |
|
Assessments |
3,308 |
4,494 |
Crowdmark has been used by MSU instructors since 2016. Gradescope has been used since 2018. More courses were created in Crowdmark until the 2020 calendar year (Figure 3). Usage of both tools spiked in 2020, presumably due to the COVID-19 induced shift to remote teaching, and was fairly equivalent that year. For the Spring 2021 semester, more courses have been created in Gradescope. It will be interesting to observe whether this trend towards Gradescope usage continues as 2021 progresses or if Crowdmark usage picks back up.
Given the disparity between number of students vs. number of classes & assessments, we explored the frequency of class sizes between the two tools (Figure 4). Both tools have been used for classes of all sizes, though the median class size is 37 for Gradescope and 63 for Crowdmark. We also explored the frequency of assessment numbers between the tools (Figure 5). We found that all but one course had 1-60 assessments created, with both tools most frequently having 2-20 assessments. Gradescope showed an interesting secondary peak of courses having 35-45 assessments. We do not have detailed information for either tool on what kinds of assessments were created or whether all of those assessments were actually used, not just created in the course for practice, or duplicates (e.g., available later, more accessible, or different versions for different class sections in Gradescope).
Figure 3. Number of courses created in each tool that had at least one student and
at least one assessment for each calendar year since 2016.
Figure 4. Number of courses having a given class size and at least one assessment.
Figure 5. Number of classes having a given number of assessments and at least one student.
Discussion:
Our analysis showed significant functional overlap between Crowdmark and Gradescope, where either tool could be chosen with little to no impact on instructor capability. However, there are a few advantages to the way that Crowdmark handles assignment tracking, submission, and grade syncing to D2L. In particular, Crowdmark already offers a fast QR-code method for matching every page of in-person assessments to the appropriate student enrolled in the course when scanning the assessments in batches. We expect this feature will become a strong asset in the Fall 2021 semester as more classes will be on campus.If we were to choose between Crowdmark and Gradescope for continued support, we would recommend Crowdmark. Gradescope is a competitive technology, but it is still developing and refining capabilities that are already available through Crowdmark or D2L. If an instructor were to need to switch from Gradescope to Crowdmark, they should refer to the D2L self-enroll course “MSU Tools and Technologies” for detailed information and resources on using Crowdmark at MSU and closely review Table 1 to understand the key differences they may encounter. The Assessment Services team and/or Instructional Technology & Development team in the IT department are also available for one-on-one consultation on using either technology (request a consultation via the MSU Help Desk).
