(2023-08-01) Willison Run Llama2 On Your Own Mac Using Llm And Homebrew

Simon Willison: Run Llama 2 on your own Mac using LLM and Homebrew. I just released a new plugin for my LLM utility that adds support for Llama 2 and many other llama-cpp compatible models

First, you’ll need LLM—my CLI tool for interacting with language models. The easiest way to install that is with Homebrew:

You can also use pip or pipx—though be warned that the system installation of Python may not work correctly on macOS, hence my prefence for Homebrew’s version of Python

The next step is to download a model. I recommend running this command to fetch a 7GB file containing the Llama 2 Chat 7B model

Running prompts through Llama 2 Chat

*Llama 2 Chat is notorious for how quickly it moralizes at you instead of doing what you want!

It turns out that’s down to the default system prompt. With LLM we can replace that using the --system option*

Playing with system prompts is really fun:

The llm -c option can be used to continue a conversation by adding a new prompt to it:

Any GGML model should work

Here’s how to install the 13B chat model (a 13GB download):

Accessing logged responses

The llm tool logs all prompts and responses to a SQLite database. You can view recent logs by running the llm logs commands

If you have Datasette installed

Using the Python API

How to speed this up—right now my Llama prompts often take 20+ seconds to complete.

I’m not yet sure that this is using the GPU on my Mac

Open questions and potential improvements