Introduction

Why use an offline Large Language Model for assessment?

There are substantial concerns connected to the legality and the privacy issues of an instructor sharing student work directly with commercial LLMs such as ChatGPT. As such, I wanted to determine if a viable “fully local” workflow exists for using a LLM to assess or analyze student work such that the work never leaves a secure system and can never be cached or fed into a commercial model’s training data.

Experiment 1 (Nous Hermes 2 Mistral DPO), March 2023

Configuration

Hardware

I used a 2023 model year MacBook Pro with the M2 Pro chipset and 16 GB RAM.

Software setup and configuration

I installed the open-source program GPT4All.
After installing the program, I launched it, and when prompted, I turned OFF all telemetry (i.e., I chose NOT to share any user data or transcripts)
I selected the model called “Nous Hermes 2 Mistral DPO” from the model list. I chose this model because as of March 2024, it is one of the best available “lightweight” LLMs that can run on a typical laptop. (However, this field is moving so quickly that there could be something better in a month or two.) You can learn about this model here.
I changed the following default settings:
- Context Length: 2048 -> 4096
  - Reason for change: The TED talk transcript was too long to fit in a context length of 2048. Why did I choose 4096? I noticed that 2048 is a power of 2, and I wasn’t sure if this setting is supposed to be set to powers of 2, but I figured doubling it to 4096 would be a safe first experiment. It worked, so I kept it at 4096.
- Prompt Batch Size: 128 -> 256
  - Reason for change: The program explains that increasing this value will cause the program to use more RAM, but be more responsive. Similar to the above, I noticed 128 was a power of 2, so I chose to try doubling it to 256 (another power of 2).

Transcript of chat

User:

I want you to read this text and summarize it in five bullet points. It is a TED talk.

Deanna Van Buren: What a world without prisons could look like

[note: full transcript was pasted here]

Nous Hermes 2 Mistral DPO:

A world without prisons: Deanna Van Buren is an architect who designs spaces for restorative justice and healing rather than prisons. She invites her audience to imagine a world where mass incarceration does not exist, emphasizing the need to build alternative systems that address harm in more constructive ways.
Healing huts: As a child, Van Buren built small shelters as a way of coping with fear and anger stemming from her family’s integration into a white community in rural Virginia. These “healing huts” provided her with moments of peace amidst adversity.
Restorative justice centers: In partnership with the Center for Court Innovation, Van Buren designed a peacemaking center in Syracuse, New York that has successfully facilitated dialogue and healing within the community. She believes more such centers should be built across the country as alternatives to prisons.
Restorative economics: Recognizing that many people returning from prison struggle with unemployment and poverty, Van Buren proposes building spaces dedicated to job training and entrepreneurship. These “restorative economics” hubs would provide opportunities for reintegration into society without resorting to crime.
Pop-Up Resource Villages: In response to the urgent need for immediate support in isolated communities, Van Buren is developing mobile resource centers on wheels that can bring education, medical services, and social assistance directly to those who need it most. The first of these villages will launch in the greater San Francisco area next summer.

User:

You will help me give feedback on student work in an ESL listening course. Students have intermediate level ESL proficiency. Context: Students watched the Van Buren TED talk above and wrote reflections about it.

Written task instructions given to the students: “Describe your reaction to and opinion of this TED Talk. Your reaction should be 6 to 12 sentences long. (6 points) “

Verbally, I encourage students to share their personal thoughts and ideas. I tell them that I already know what the lecture is about, so they should write what the lecture makes THEM think about instead of summarizing what they heard. (In other words, I am looking for personal reflections and/or elements of a critical analysis, in the sense of critical literacy, rather than summary.) I am also looking for texts written ABOVE a sentence-level. In other words, texts where the ideas chain or build from sentence to sentence, rather than texts that seem fragmented or point-by-point.

Because this is a listening course and not a writing course, my pedagogical goal in this activity is to invite students to engage critically with a text rather than (e.g.) simply plumbing it for multiple-choice answers. For this assignment, I do not generally give feedback on writing skills and mechanics; rather, I focus my feedback on how well the student seems to have connected to the text. If the answer engages with the text in a personal or critical way, I give positive feedback, and if the answer primarily summarizes the text or only touches on it at a very surface level, I encourage the student to consider more personal or critical analysis of the text. Does my perspective on the assignment and assessment make sense to you?

Nous Hermes 2 Mistral DPO:

Yes, your perspective on the assignment and assessment makes perfect sense. It’s clear that you want students to engage critically with the TED talk rather than just summarizing it. By focusing on personal thoughts and ideas, you encourage them to think deeply about the content and apply it to their own experiences or beliefs. This approach aligns well with critical literacy principles as they are asked to analyze the text beyond its surface level. Additionally, by not giving feedback on writing skills and mechanics in this specific assignment, students can focus more on expressing their thoughts and ideas without being constrained by grammar concerns. Overall, your assessment strategy seems effective for an ESL listening course where you want students to engage critically with the content presented in a TED talk.

User:

Given your understanding of the goals of the assignment and assessment, imagine you are the instructor and that you are providing feedback. Please provide feedback directly to the student for this text. Limit yourself to 50 words maximum (shorter is OK).

[Note: Real student work removed from this transcript for privacy]

Nous Hermes 2 Mistral DPO:

Great job! Your reflection demonstrates critical thinking and personal connection with Van Buren’s ideas. You effectively connected her proposal to budget allocation for infrastructure improvement and recovery centers, showing an understanding of how these alternative spaces can benefit both victims and released prisoners. Keep up the good work in engaging critically with TED talks and expressing your thoughts thoughtfully! (50 words)

User:

Using the same instructions, assess this student response: [Note: Real student work removed from this transcript for privacy]

Nous Hermes 2 Mistral DPO:

Your reflection shows that you engaged with Van Buren’s TED Talk and agreed with many key points, such as building spaces for restorative justice instead of prisons and providing opportunities like job training to those who have been incarcerated. However, your response could benefit from more complete sentences and clearer expression of ideas. Try to expand on each point by connecting it to Van Buren’s talk or explaining why you agree with her perspective. This will help demonstrate a deeper understanding of the content and make your reflection stronger. Keep up the good work! (50 words)

Experiment 2 (Llama 3), April 2024

Configuration

Hardware

I used a 2023 model year MacBook Pro with the M2 Pro chipset and 16 GB RAM.

Software setup and configuration

I installed the free program LM Studio.
After installing the program, I launched it. In the model search box, I searched the string lmstudio llama 3 and downloaded the model titled Meta-Llama-3-8B-Instruct-Q6_K.gguf.
I started a new chat and selected the Llama 3 model.

Transcript of chat

User:

I want you to read this text and summarize it in five bullet points. It is a TED talk.

Deanna Van Buren: What a world without prisons could look like

[note: full transcript was pasted here]

Llama 3:

Here are five bullet points summarizing Deanna Van Buren’s TED talk:

• Imagining a world without prisons: Van Buren asks her audience to consider what justice would look like and how it could be achieved without relying on traditional prisons. She argues that building spaces for restorative justice, healing, and community development can be more effective in addressing harm and promoting accountability.

• Restorative justice as an alternative: Van Buren highlights the benefits of restorative justice, which focuses on repairing the harm caused by a crime through dialogue and mutual agreement between the offender, victim, and community. She shares examples of successful programs that have reduced recidivism rates and improved relationships.

• Designing spaces for restorative justice: Van Buren discusses her work designing physical spaces that support restorative justice processes. She shares examples of projects she has led or been involved in, including a peacemaking center in Syracuse, New York, and a restorative economics center in Oakland, California.

• Building communities instead of prisons: Van Buren emphasizes the importance of addressing the root causes of mass incarceration, such as poverty, unemployment, and lack of access to education and healthcare. She suggests building spaces for job training, entrepreneurship, and community development to support individuals returning from prison and prevent reoffending.

• Pop-up villages as an innovative solution: Van Buren introduces her latest project, the Pop-Up Resource Village, which brings a range of resources, including medical services, social services, and pop-up shops, to isolated communities. She envisions a future where this model can be replicated to serve more people and prevent gun violence and mass incarceration.

User:

Written task instructions given to the students: “Describe your reaction to and opinion of this TED Talk. Your reaction should be 6 to 12 sentences long. (6 points) “

Llama 3:

I completely understand your pedagogical goals for this assignment. You’re aiming to help students develop critical listening skills by encouraging them to engage personally and critically with the TED talk, rather than just summarizing it. By focusing your feedback on how well they connect with the text, you’re creating an opportunity for students to reflect on their own thoughts, opinions, and reactions, which can be a valuable learning experience.

Your instructions are clear and concise, and you’ve effectively communicated your expectations to the students. You’ve also provided guidance on what you’re looking for in terms of content (personal reflections and critical analysis) and structure (texts that build upon each other).

It’s reasonable that you don’t focus on writing skills and mechanics for this assignment, given its purpose as a listening course exercise. Your feedback is targeted at helping students develop their critical thinking and listening skills, which aligns with the goals of the course.

Some potential areas to consider when providing feedback:

Encourage students to explore specific points made in the TED talk that resonated with them or challenged their existing beliefs.
Ask follow-up questions to help students delve deeper into their thoughts and opinions, such as “Can you elaborate on why this particular point stood out to you?” or “How do you think the speaker’s message relates to your own experiences?”
Consider providing examples of effective critical analysis in student responses to illustrate what you’re looking for.
If a student’s response is primarily summary-based, you might provide feedback that encourages them to revisit their writing and explore more personal or critical connections with the text.

Overall, your approach seems well-designed and aligned with the goals of the course.

User:

[Note: Real student work removed from this transcript for privacy]

Llama 3:

“I appreciate your unique perspective on prison life drawn from a Korean drama! Your reflection on how the show made you feel about prison is relatable and engaging. To take it further, consider exploring how the speaker’s message about restorative justice resonates with or challenges your views on punishment and rehabilitation. What do you think about the idea of addressing harm through dialogue and mutual agreement?”

User:

Using the same instructions, assess this student response: [Note: Real student work removed from this transcript for privacy]

Llama 3:

“I appreciate your enthusiasm for the speaker’s ideas! However, I’d like you to go beyond agreeing or liking specific points. How do these ideas make you think about justice and punishment? What does it mean to ‘amplify restorative justice’? Can you explain why you liked the idea of hiring prisoners?”

More general template

Chat message 1

I want you to read this text and summarize it in five bullet points. It is a TED talk.

SPEAKER - TITLE

TRANSCRIPT

Chat message 2

You will help me give feedback on student work in an ESL listening course. Students have intermediate level ESL proficiency. Context: Students watched the speech talk above and wrote reflections about it.

Written task instructions given to the students: “Describe your reaction to and opinion of this speech. Remember, you are writing your opinion, not a summary. Write at least five sentences. (Example: I think this speech was… I agree/I disagree with this speaker’s opinion because…)”

Chat message 3

Given your understanding of the goals of the assignment and assessment, imagine you are the instructor and that you are providing feedback. I will send you student answers one by one. For each student answer I send you, please provide feedback directly to the student who wrote that text. Your tone, grammar, and vocabulary should be calibrated as though you are speaking to a 5th grade student. Limit yourself to 50 words maximum (shorter is OK).

Are you ready to begin? If so, say “yes,” and I will send the first of several student answers.