Replace Grammarly with open source LLM

✍︎

I have been using the self-hosted llama 30B model for local grammar check and translation because most of the smaller llama models are not good at following instructions. However, the new Mistral 7B is incredibly good. This means that now we can have an in-device local LLM for those tasks.

In this post, let’s quickly go through how to set up the Mistral via llama.cpp and integrate it with Mac Automator so you can use keyboard shortcuts to quickly fix grammar and translate in any text editor.

First, you can copy the bash script from below and run it to launch a llama.cpp server that serves the model via HTTP API.

#!/bin/bash

# Define the repository and model details
REPO_URL="git@github.com:ggerganov/llama.cpp.git"
REPO_DIR="llama.cpp"
MODEL_URL="https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q5_K_M.gguf"
MODEL_FILE="mistral-7b-instruct-v0.1.Q5_K_M.gguf"

# Clone the repository if it doesn't already exist
if [ ! -d "$REPO_DIR" ]; then
  git clone "$REPO_URL"
fi

# Change directory to the cloned repository
cd "$REPO_DIR"

# Build the project using make
# Assume make is idempotent, as it should only rebuild changed files
make

# Change directory to the models folder
mkdir -p models
cd models

# Download the model if it doesn't already exist
if [ ! -f "$MODEL_FILE" ]; then
  wget "$MODEL_URL"
fi

# Change directory back to the project root
cd ../

# Launch the server
./server -m models/"$MODEL_FILE" -c 8192

Second, create an Automator quick action and select “Run JavaScript”. Make sure you select workflow receives text in any application, and check that output replaces selected text as shown in this screenshot:

✍︎

✍︎

In the run javascript section, add the following code:

function escapeShellArg(str) {
    return "'" + str.replace(/'/g, "'\\''") + "'";
}

const removeBackticks = (str) => {
    // remove leading backticks
    str = str.replace(/^(```\n|```)/g, "");

    // remove tailing backticks and everything after
    const index = str.lastIndexOf("```");
    if (index !== -1) {
        return str.slice(0, index);
    }
    return str;
};

function run(input, parameters) {
    const apiEndpoint = "http://localhost:8080/completion";
    const prompt = `[INST] Corrects and rephrase user text grammar errors delimited by triple backticks to standard English.
Text=\`\`\`she no went to market\`\`\` [/INST]
[INST] Output: She didn’t go the market. [/INST]
[INST] Text=\`\`\`${input}\`\`\` [/INST]
[INST] Output:`;

    const requestData = {
        prompt: prompt,
        temperature: 0.1,
        stop: ["[/INST]"],
    };

    const curlCommand = `curl ${apiEndpoint} -X POST -v -H 'Content-Type: application/json' -d ${escapeShellArg(
        JSON.stringify(requestData)
    )}`;

    // Instantiate the Application object
    const app = Application.currentApplication();
    app.includeStandardAdditions = true;

    const apiResultJSON = app.doShellScript(curlCommand);
    // Parse the JSON response and extract the result
    const apiResultObject = JSON.parse(apiResultJSON);
    const result = apiResultObject.content;

    return result.trim();
}

Finally, go to system settings -> keyboard -> keyboard shortcut. Select services section and assign a shortcut for the quick action that you created in step two. After that, simply select any text in a text editing field and use the shortcut to send it to your local LLM for grammar check or translation as shown in thscreenshot:

✍︎

That’s all for setup! Thanks to open source community who made these advancements possible. It’s truly amazing that we now have such powerful tools on our devices while maintaining control over privacy.

PS: I also fine-tuned a few Mistral 7b models for grammar correction using synthetic data generated via ChatGPT. It should have better performance than the original model. If you are interested, please check it out at https://huggingface.co/mzbac/mistral-grammar/tree/main.