site banner

Friday Fun Thread for November 8, 2024

Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.

1
Jump in the discussion.

No email address required.

I can run 7B models on a Macbook M2 with 8 GB of ram. This is because of how Macbooks handle VRAM.

It's pretty slow, and 7B models aren't great for general tasks. If you can use one that's fine tune for a specific thing, they're worth it.

Frankly, however, I'd just recommend using something like together(dot)AI or OpenRouter to run larger models elsewhere. Normal caveats about not pushing sensitive info out there, of course. $30-$50 worth of credits, even for monster models like Meta's 405B, will take you easily though a month of pretty heavy usage (unless you're running big automated workloads 24/7).

I think there's going to be a race between local AI specific hardware for consumers and just cloud based hyperscaling. I don't know which will win. Privacy definitely plays a part. I'm quite optimistic to see a new compute hardware paradigm emerge.

I'm using openrouter.ai daily. The credits last for a surprisingly long time. Sonnet 3.5 is my go-to model.

I'd like something offline and private for sensitive use though.