Students created a stable P.S. Press site, that houses or links to content from previous P.S. Press sites; they also created social media accounts that can continue from year to year. Students created four books, poempostcards, broadsides, and videos as a part of the publication process (photos below).

broadside

Broadside for Light Passes Through

Light Passes Through Front Matter

the front matter of *Light Passes Through*

poem postcards

Poem postcards, designed by student publishers in ENGL 1820.

Light Passes Through, finished copies

Light Passes Through, finished copies

finalist edition

Finalist edition

Student publishers, student authors at chapbook launch

Student publishers, student authors at chapbook launch

 

Tarfia Faizullah broadsides

Tarfia Faizullah broadsides

 

 

Preliminary Report: Assessment of ENGL 1010 & 2010 |

Spring 2016

Background of the assessment; assessment design.

The English Department has offered ENGL 1010 and ENGL 2010 in the online plus modality, first as an ongoing pilot, then as a regular offering on a small-scale. We’ve done ongoing pre- and post-surveys of students, in which we inquired about preparation, experience with online and hybrid modalities, aspects of the course design that favorably or unfavorably influenced the conditions of their learning, etc. In these surveys we found enough positive feedback to both revise aspects of the curriculum and to warrant pressing forward with the model.

In the Fall of 2015, we intentionally scaled up the number of sections and the number of personnel teaching the course, with the aim of using this larger population to do a learning outcomes assessment, in addition to another set of surveys.

The design of the learning outcomes assessment had three key features:

  • Sample from across sections in the online plus modality, as well as across sections of other modalities taught in the department (i.e., sections not taught in the online plus modality). We used a sample size of 50 ePortfolios for each course, randomly sampled (by Institutional Research), 25 from online plus designated sections, 25 from other modalities. We oversampled in case of unusable ePortfolios.
  • Use of an adapted Written Communication rubric, from the College’s adapted VALUE rubric.
  • Assessment across modalities, for purposes of comparison.

First we normed, with the 1010 and 2010 teams meeting separately. Our method for reading was double grading with adjudication. Pairs of readers, which changed at regular intervals, read the same ePortfolio submission, then used the scoring grid to assign values to the various parts of the rubric. Where there were significant differences—more than one point—we discussed what we saw, with the aim of arriving at a consensus. In no case did we need a third reader because of this discussion/negotiation protocol.

Faculty who participated in the reading and scoring part of the assessment included: Chris Blankenship, Rebecca Miner, Daniel Baird, Steve Harrison, Beth Bailey, Lisa Bickmore, Justin Jory, Benjamin Solomon, Kati Lewis, Brittany Stephenson, Ron Christiansen, Charlotte Howe, Allison Fernley, Nkenna Onwuzuruoha, Clint Johnson, Sue Briggs, and Stacie Weatbrook.

Limitations of the design

Because of the relatively small sample size, we found a number of ePortfolios without a submission for ENGL 1010 or ENGL 2010, even with our attempt to oversample.

Key observations/findings (contrast 1010 and 2010 outcomes):

The findings indicated that we should focus on stronger instructional materials and a more consistent approach to the following traits, since the scores overall indicated greater weakness in these traits across the samples regardless of instructional modality:

  • Genre/discipline specific conventions
  • Research/credible evidence
  • Content development

The overall scores are broken down in the following tables. The scale we used was

1 = Well below expectations                        2 = Below expectations
3 = Meets expectations                                4 = Exceeds expecatations

 

Average for Each Criterion

1010 online plus 1010

other

1010 overall 2010 online plus 2010

other

2010 overall
1. Context and Purpose for Writing 2.73 2.28 2.50 2.78 2.47 2.65
2. Content Development.

 

2.60 2.34 2.46 2.74 2.35 2.58
3. Credible Evidence

 

2.56 2.37 2.46 2.72 2.50 2.63
4. Organization

 

2.86 2.50 2.67 2.78 2.61 2.71
5. Genre and Discipline-Specific Conventions 2.33 2.25 2.29 2.70 2.47 2.60
6. Control of Syntax and Mechanics 2.46 2.31 2.38 2.84 2.85 2.84
7. Overall Impact

 

2.63 2.15 2.38 2.76 2.44 2.63

Number of portfolios which received each score on the overall impact criterion

1010 online plus 1010 other 1010 overall (total)
  Out of 15

Portfolios

Out of 16

Portfolios

Out of 31

Portfolios

4 0 1 1
3.5 1 0 1
3 4 1 5
2.5 8 4 12
2 2 7 9
1 or 1.5 0 3 3 
2010 online plus 2010 other 2010 overall (total)
Out of 25

Portfolios

Out of 17

Portfolios

Out of 42

Portfolios

4 1 1 2
3.5 4 0 4
3 9 5 14
2.5 5 3 8
2 5 6 11
1 or 1.5 1 2 3

Percentage of portfolios in each category on the overall impact criterion

1010 online plus 1010 other 1010 overall
Exceeds Expectations

(3.5 or 4)

6% 6% 6%
Meets Expectations

 (3)

26% 6% 16%
Close to Meeting Expectations (2.5) 53% 25% 38%
Below Expectations

 (2)

13% 43% 29%
Well Below Expectations

(1 or 1.5)

0% 18% 9%
2010 online plus 2010 other 2010 overall
Exceeds Expectations

(3.5 or 4)

20% 5% 14%
Meets Expectations

 (3)

36% 29% 33%
Close to Meeting Expectations (2.5) 20% 17% 19%
Below Expectations

 (2)

20% 35% 26%
Well Below Expectations

(1 or 1.5)

4% 11% 7%

We note that the scores suggest a fairly decided trend toward better performance for students in online plus designated sections, as compared to those sampled from other instructional modalities. We also noted better performance in students simply submitting the signature assignment to the ePortfolio for online plus designated sections, as compared to sections in other instructional modalities. We think this can probably be attributed to the stronger norming of instructional expectations in the team setting that is one of the characteristics of the online plus instructional model.

Recommendations:

  1. Signature assignment. The outcomes of this assessment could help us within the department in thinking about the signature assignment—how can the assignment be variable, adaptable, and still allow us to look consistently at outcomes across sections?
  2. Further investigation of the online plus effect. The marked better performance of students in online plus sections bears further investigation. Aspects of the online plus model that may affect the positive outcomes include (a) emphasis on one-on-one or small group instruction, focused on student writing; (b) team approach, with better support of part-time faculty and potentially better engagement in curriculum, as well as stronger follow-through on things like the ePortfolio and the signature assignment.
  3. Develop additional models of instructional teamwork. As a department, we may want to develop models for instructional teamwork aside from the online plus model, if we agree that such teams offer at least the potential of stronger support for and consistency in student outcomes.
  4. Develop additional curricular materials. We should work on developing curricular materials to strengthen the outcomes where students’ work exhibits the greatest weaknesses (genre and discipline-specific conventions; research and credible evidence; content development).
  5. Strengthen the ePortfolio message. We need stronger messaging about, and support for, key outcomes and strategies of ENGL 1010 and 2010, and specifically the ePortfolio and signature assignment. This was a strong theme of the discussion among assessment participants after the reading and scoring session, with new faculty members and some part-time faculty members underscoring the lack of clarity on this point.

Attachment A: Outline of Assessment Plan for ENGL 1010/2010 Online Plus

Attachment B: Rubric for Written Communication, Online Plus Assessment 2015 Fall ePortfolio

Attachment C: Norming Guidelines

Attachment D: Scoring Record

Attachement E: Rejected ePortfolios

ATTACHMENT A: Outline of Assessment Plan for ENGL 1010/2010 Online Plus

We have offered ENGL 1010 and 2010 in the online plus modality (online course + intensive small group or one-on-one interactions with students—a kind of flipped classroom, and a kind of hybrid course) for about two years. We have conducted various student surveys, with the cooperation of Institutional Research, to discover what student experience in the modality is.  The results of the surveys have been encouraging enough to continue refining the courses and continue teaching, along with a modest scaling up of the overall number of sections offered in both courses.

This fall, we have the greatest number of both personnel and sections in the model. ENGL 1010 is currently offering twenty-two mini sections (= eleven full sections), with two FT and two PT faculty teaching. ENGL 2010 is offering fifteen full sections, with three FT and three PT faculty teaching.

We planned, along with this intentional scale-up, to conduct a multifaceted assessment of both courses. The features of the assessment include:

  • Early semester student survey, to measure student attitudes and responses about their past experiences in writing courses, when they registered for the course, what attributes of a writing course they value, their understanding of what a hybrid course is, and a few demographic questions. We will follow this up with
  • A late semester student survey, that will measure the same items, mostly, as above. Working through IR allows us to key student data to an S#, which means we can track changes, if any, in their responses. We’ll also be looking deep into Banner data in order to find out as much as we can about retention/attrition data.
  • Finally, we plan a learning outcomes assessment, using student ePortfolio. We will sample from across the sections in the Online Plus modality, and will also sample from sections in other modalities, for comparative purposes. We are especially interested in other members of the department, both full- and part-time, participating in the reading and scoring of the ePortfolios for this part of the assessment.

We may also, down the line, do some sort of qualitative assessment of the team effect on participating instructors, particularly for part-time instructors.

Proposed Method and Sample Size:

We plan to use the Written Communication rubric. We plan to look only at English 1010/2010 signature assignments.

We propose looking at 25 portfolios each, randomly selected from Online Plus sections of 1010 and 2010.

We propose looking at 25 portfolios each, randomly selected from NON Online Plus sections of 1010 and 2010, for comparative purposes.

We propose to do two independent readings of each portfolio, with a third reader where the scores are at significant variance with each other (more than a one point difference).

ATTACHMENT B: Written Communications Quality Scoring Rubric

Criteria Well Below Expectations

1

Below

Expectations

2

Meets

Expectations

3

Exceeds

Expectations

4

1. Context and Purpose for Writing

Includes considerations of communicating with a particular audience, purpose of the work, and the circumstances surrounding the writing situations.

 

The student did not respond to the demands of the writing situations. The student attempted to respond to some of the demands of the writing situations. The student adequately responded to the demands of the writing situations. The student was highly responsive to the demands of the writing situations.
2. Content Development

Uses appropriate, relevant, and compelling content to illustrate an excellent grasp of the subject, conveying the writer’s understanding, and shaping the whole portfolio.

 

The student attempted to use content to describe the subjects suggesting incomplete knowledge of the subject. The student used content to describe the subjects but left the reader unsure whether the student fully understood it. The student explored the content suggesting a strong understanding of the subjects addressed. The student fully structured/developed the content throughout the portfolio making a compelling statement about their mastery of the various subjects.
3. Credible Evidence

Supports claims/themes /arguments /ideas /thesis /etc. (pick one).

The student did not include evidence. The student provided some, but insufficient evidence. The student supported his/her writing with evidence. The student consistently supported his/her writing with precise, relevant, and/or compelling evidence.
4. Organization

A specific and recognizable method for arranging and connecting the components of the writing.

 

The ideas are presented randomly and/or are not logically linked to each other thus making it nearly impossible for the reader to follow. The ideas are present within the writing but are not all connected to each other in a logical way thus making it difficult for the reader to follow.

 

The ideas are arranged logically and support each other making it possible for the reader to follow. The ideas flow smoothly from one to another and are clearly linked to each other making it very easy for the reader to follow.
5. Genre and Discipline-Specific Conventions

The student shows awareness of genre conventions and adapts the conventions appropriately for the writing situation.

The student does not demonstrate awareness of genre conventions. The student attempts to use some genre conventions. The student generally follows most of the genre conventions The student uses and adapts genre conventions fluently.
6. Control of Syntax and Mechanics

Follows the conventions of standard edited English or other language suitable to the genre/purpose.

The student did not follow many of the conventions of written English making the submission difficult for the reader to understand. The student attempted to follow the conventions of written English but the reader noticed several distracting errors. The student generally followed most of the conventions of written English and at times exceeded expectations for control of grammar and style. The student adeptly used the conventions of English to communicate strongly and clearly.
7. Overall Impact

A holistic judgment of the portfolio’s impact or quality.

The portfolio needs major revision and/or rewrite before resubmission. The portfolio has a few significant challenges (see other parts of this rubric) that require revision. The portfolio could be improved with a few revisions, but the work passes as is. The portfolio was very well done and could be placed as-is into the Gen Ed portfolio.

ATTACHMENT C: Norming Guidelines

General philosophy

The idea is not to grade but to evaluate the overall writing and the program design. That is we are not evaluating how well the student met the assignment criteria. If there are weaknesses in assignment design, then this will affect the quality of writing which is something we want to know about. Therefore we do not need to have access to the assignment sheet or teacher expectations. Instead evaluate the portfolio on a continuum: what does this writing represent at this moment in a student’s development as a writer?

Process

  1. Discuss rubric:
    1. History: first developed in Tom Zane’s assessment office for Written Communication in general; then revised (especially performance descriptors for #5 & #6) for the English 2010 assessment in 2014-15; finally revised again for this assessment (we removed or modified criteria which focused on argumentative writing; also we removed the creativity criteria).
    2. What are the key words that distinguish between performance descriptors? (these are sometimes bolded but not necessarily always)
    3. Have a brief discussion but recognize that all rubrics continually evolve. For this assessment we will be using this imperfect rubric. (in past experience people want to discuss at length or change the rubric for certain perceived weakness)
  1. Read and score first portfolio for the 7 traits
    1. Call out scores and put on the board
    2. Where in the sample can you demonstrate that criteria…..Have 2 or 3 instructors describe why they chose that score
    3. After hearing the other scores would anyone change their score? Why?
    4. Does this process say anything more globally about how we evaluate or teach this skill?
    5. Repeat for each criteria—where there is overall agreement move on quickly
  1. Read and score second portfolio for the 7 traits (repeat steps above)
  1. Overall conversation
    1. What did you notice about your individual process? Were you more worried about being perceived as an easy or tough scorer? Keep these insights in mind during the assessment.
    2. What from this process will inform how we evaluate writing for this assessment?
  1. Working in pairs
    1. Discuss scores at the end of the hour
    2. If there is a _____________ point disparity which can’t be resolved then this will trigger a third reading.

ATTACHMENT D: Scoring Record

 

S___________________

 

Criterion Reader 1 Reader 2 Average
1
2
3
4
5
6
7
S_______________________

 

Criterion Reader 1 Reader 2 Average
1
2
3
4
5
6
7
S_______________________

 

Criterion Reader 1 Reader 2 Average
1
2
3
4
5
6
7
S_______________________

 

Criterion Reader 1 Reader 2 Average
1
2
3
4
5
6
7

APPENDIX E: Rejected ePortfolios

 

Rejected eportfolios

1010 online plus 1010

other

2010 online plus 2010

other

Desired number of portfolios 25 25 25 25
Actual number of portfolios 15 16 25 17
Portfolios rejected for missing link in MyPage 14 8 1 7
Portfolios rejected for broken URL 0 1 2 0
Portfolios rejected for no relevant English work 1 5 0 6