Ted Wild's Blog

Running LLM on an M2 Macbook Air

I wanted to see if I could run an LLM on my personal Macbook Air. I don't expect magic, but thought it'd be interesting to play around with prompts and such without having to worry about cloud costs or internet connections.

Instructions are mostly from this blog, which appears to be a bit out of date as of this writing. The updated instructions on Github did work.

brew install llm
llm install llm-llama-cpp
llm install llama-cpp-python
llm llama-cpp download-model \
  https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q2_K.gguf \
  --alias llama2-chat --alias l2c --llama2-chat
llm -m l2c 'Tell me a joke about a llama' --system 'You are very funny'

One troubleshooting tip: adding -o verbose 1 helped me realize what was wrong with the first instructions I tried.

llm -m l2c 'Tell me a joke about a llama' -o verbose 1

Other articles

Anki settings after a long break
Published: Thu 18 June 2020
By Ted

In Studying.

tags: anki tips

Anki settings after a long break

The standard advice when returning to an Anki deck after a long break is
1. Create a filtered deck that contains cards that are recently due (say, within a week)
2. Review remaining backlog little by little until all reviews are cleared
The idea is that …
read more
Full tensors in Torchscript
Published: Sat 08 February 2020
By Ted

In Programming.

tags: tips torchscript

Full tensors in Torchscript

I was productizing some PyTorch models recently and ran in to an issue with the way torch.full_like behaves in Torchscript.
```
import torch
```
The original code use torch.full_like to create a tensor with a given shape. It also forced the type of the new tensor …
read more
gRPC layout for Python

Published: Sat 30 March 2019
By Ted Wild

In Programming.

tags: gRPC Python tips

Setting up a layout for gRPC files that works seamlessly with Python
read more

Other articles

Anki settings after a long break

Full tensors in Torchscript

links

social