Introduction

Why use an offline Large Language Model for assessment?

There are substantial concerns connected to the legality and the privacy issues of an instructor sharing student work directly with commercial LLMs such as ChatGPT. As such, I advocate the use of “fully offline” workflows when using a LLM to assess or analyze student work, such that the work never leaves the secure system and can never be cached or fed into a commercial model’s training data.

Configuration

Hardware

I used a 2023 model year MacBook Pro with the M2 Pro chipset and 16 GB RAM.

Software setup and configuration

I installed the program LM Studio, which enables users to run free, open-source Large Language Models offline on their local hardware.
After installing the program, I launched it. In the model search box, I searched the string lmstudio llama 3 and downloaded the model titled Meta-Llama-3-8B-Instruct-Q6_K.gguf.
I started a new chat and selected the Llama 3 model.

Transcript of chat

User:

I am the instructor of a writing support course for international students in an information science graduate program. In this chat, you will assist me in providing feedback on student work. Specifically, you will provide feedback on student final essays. First, here is the general specification of the project:

[note: full essay assignment description was pasted here]

Do you understand the assignment that students needed to complete? Do you have questions about it?

Llama 3:

Yes, I understand the assignment. Based on the specifications provided, students were required to write an argumentative essay on a topic related to their field of study, using scholarly sources in English, including peer-reviewed academic journals. The essay should be 5-6 pages long, double-spaced, and formatted according to APA style (7th Ed). Additionally, it must include one visual element discussed in the essay, at least three paragraphs that support the thesis statement with key points, and two paragraphs that rebut counterarguments.

I do not have any questions about the assignment. If you’d like to provide a student’s essay for feedback, I’m ready to assist you in providing constructive comments on their writing.

User:

I will worry about the rubric and numeric scoring. I am asking for you to provide more impressionistic and holistic feedback about the content and structure of the essays, including acknowledging the strengths of the essays as well as providing feedback on areas for improvement. Your feedback will be given in the form of one or more paragraphs written directly to the student. Don’t include “Dear student” or wrap your comments in quotation marks or anything like that; just respond directly to the student, as if you are chatting with them personally. An ideal comment should be roughly 50-200 words, but not longer than that. Do you understand these parameters? Can I give you any additional information or context before we begin that would be helpful in completing this task?

Llama 3:

I understand the parameters now. You’d like me to provide holistic and impressionistic feedback on the content and structure of the essays, focusing on strengths and areas for improvement without referencing a specific rubric or scoring system. I’ll respond directly to each student as if we’re having a conversation.

Before we begin, is there any additional information or context about the students or their essays that might be helpful for me to know? For example, are there specific topics or themes that were assigned, or did students choose their own topics? Are there certain skills or concepts that you’d like me to pay particular attention to in my feedback?

User:

Students chose their own topics. Part of my task will be to assess students’ ability to follow APA style, but this is not your concern; I would like you to focus more on the issues of content and organization and, in general, whether the student makes a clear and coherent argument. One potential issue would be if an essay reads more like a “book report” than an argument– students sometimes have difficulty distinguishing between expository and argumentative writing, so if an essay seems too clearly trivial or “uncontroversial,” that would be a valid point for feedback. Are your ready to proceed? If so, I will send essays one by one, starting from the title page and ending at the conclusion before the reference list. You will reply each time with the feedback for the student.

Llama 3:

I’m ready to proceed! Please go ahead and share the first essay, starting from the title page to the conclusion. I’ll provide feedback on content, organization, and argumentation, focusing on whether the student makes a clear and coherent argument. If an essay appears more like a book report or lacks controversy, I’ll note that as well.

Please send the first essay, and I’ll respond with my feedback in roughly 50-200 words.

User:

[note: full student essay was pasted here]

Llama 3:

Your essay on the role of Artificial Intelligence (AI) in education is comprehensive and well-organized. You provide a clear thesis statement that AI can revolutionize the way students navigate complex data, ultimately empowering their critical thinking and problem-solving skills.

One of your greatest strengths is your ability to effectively incorporate supporting evidence from scholarly sources. You use quotes from reputable authors, such as Qadir (2022) and Rahman and Yutaka (2023), to illustrate AI’s capabilities and potential benefits in education. This demonstrates a clear understanding of the topic and shows that you have done thorough research.

Your discussion on the potential drawbacks of AI integration is also well-handled. You acknowledge concerns about diminishing research skills, plagiarism, and overreliance on AI-generated content, and provide thoughtful counterarguments to these points. This nuanced approach adds depth and complexity to your essay, making it more engaging and persuasive.

In terms of areas for improvement, I would suggest considering the following: Firstly, while you effectively introduce the concept of AI in education, some sections could benefit from a clearer transition between ideas. For instance, the section on AI’s role in assisting students with writing could be linked more explicitly to the earlier discussion on AI’s potential benefits.

Secondly, there are moments where your language becomes quite formal or even overly academic (e.g., “ultimately empowering their critical thinking and problem-solving skills”). While this is not necessarily a bad thing, it might be worth considering how to strike a balance between conveying complex ideas and making the text more accessible to readers.

Lastly, some of your paragraphs could benefit from stronger connections to the thesis statement. For example, the paragraph on AI’s role in solving mathematical problems feels somewhat disconnected from the broader argument about AI’s potential benefits in education.

Overall, however, your essay is well-written and effectively makes a case for the transformative power of AI in education. Your research is thorough, and you demonstrate a clear understanding of the topic. With some tweaks to organization, language, and connection to the thesis statement, this could be an even stronger essay.

User:

[note: full student essay was pasted here]

Llama 3:

Your essay on the impact of generative AI on human workers in employment is comprehensive and well-organized. You present a nuanced discussion of the potential benefits and drawbacks of AI integration into the workforce, drawing from various sources and studies.

One of your strengths is your ability to synthesize complex ideas and research findings into a clear and accessible narrative. Your writing is engaging, and you effectively use transitions to connect your points and maintain a logical flow throughout the essay.

Your discussion on the benefits of AI integration is well-supported by evidence from reputable sources. You highlight the potential for AI to augment human capabilities, increase productivity, and create new job categories. Your analysis of the National Bureau of Economic Research’s study on AI-assisted customer support agents is particularly effective in demonstrating AI’s ability to enhance worker performance.

However, I would suggest considering some areas for improvement:

While you provide a comprehensive overview of the potential benefits and drawbacks of AI integration, some sections feel somewhat disconnected from each other. For example, your discussion on the importance of education and training for workers could be linked more explicitly to your earlier points about AI’s potential impact on employment.
Some paragraphs, such as the one discussing the “Gen-AI: Artificial Intelligence and the Future of Work” report, could benefit from stronger connections to your thesis statement or clearer transitions between ideas.
While you effectively cite various sources, some references could be more explicitly integrated into your analysis. For example, you quote Ayoola et al.’s study on LLMs, but it might be helpful to incorporate their findings more directly into your discussion.

In terms of areas for improvement, I would suggest considering the following:

While you provide a thorough overview of AI’s potential impact on employment, some readers may find the discussion overly focused on the benefits of AI integration. Consider acknowledging and addressing concerns about job displacement or other negative consequences of AI adoption.
Some readers may find your writing style too formal or academic. Consider using more accessible language to convey complex ideas and research findings.
While you effectively discuss the importance of education and training for workers, some readers might appreciate more concrete examples or anecdotes illustrating this point.

Overall, however, your essay is well-written and provides a thoughtful exploration of AI’s potential impact on human workers in employment. With some revisions addressing these areas, your essay could be even stronger.

User:

[note: full student essay was pasted here]

Llama 3:

Your essay on the application of AI in video games is comprehensive and well-organized. You provide a clear overview of how AI has enhanced the diversity of NPCs and environments, improved game design and development, and created new experiences for players.

Your discussion on the benefits of AI in video games is well-supported by evidence from reputable sources. You highlight the potential for AI to improve game realism, create more immersive experiences, and enhance player engagement. Your analysis of Hutson’s (2018) research on the impact of AI on game design and development is particularly effective in demonstrating the transformative power of AI.

However, I would suggest considering some areas for improvement:

While you provide a thorough overview of the benefits of AI in video games, some readers may find your discussion too focused on technical aspects. Consider acknowledging and addressing concerns about the potential negative impacts of AI on game development, such as job displacement or creative stifling.
Some paragraphs, such as the one discussing procedural content generation, could benefit from stronger connections to your thesis statement or clearer transitions between ideas.
While you effectively cite various sources, some references could be more explicitly integrated into your analysis. For example, you quote Madan (2020) on dynamic difficulty adjustment, but it might be helpful to incorporate their findings more directly into your discussion.

In terms of areas for improvement, I would suggest considering the following:

While AI has the potential to revolutionize video game development, some readers may find your essay too focused on technical aspects and neglecting the creative implications of AI in gaming.
Some readers may find your writing style too formal or academic. Consider using more accessible language to convey complex ideas and research findings.
While you effectively discuss the benefits of AI in video games, consider acknowledging and addressing concerns about the potential negative impacts of AI on game development.

Overall, however, your essay is well-written and provides a thoughtful exploration of the application of AI in video games. With some revisions addressing these areas, your essay could be even stronger.