Unable to load image

Bought a used mining rig

Since I've now listened to one (1) entire talk on AI, I am the world's foremost leading expert.

As such I will be running a localized llama 13B instance. For the first time in over a decade I've bought a PC that wasn't an HP workstation.

Specs:

H110 motherboard w/celeron and 4GB ram alongside Nvidia quadro 5200 8gb

Do you guys think it'll actually run a quantized llama? Is 500W PSU enough?

16
Jump in the discussion.

No email address required.

lol you'll find out when you try to run it

Jump in the discussion.

No email address required.

I need ur thoughts and prayers

I feel fairly confident esp with quantizing. I'm not expecting fast response and I can always add a second gfx card for 16gb vram.

Or comfortably run the 7B model which apparently sucks tho

Jump in the discussion.

No email address required.

use koboldCPP and split over vram and ram

only works with GGML.bin models though :marseydepressed:

Jump in the discussion.

No email address required.

Oh yah one of the dope :marseytrain2: coooders dropped this gem yesterday: https://simonwillison.net/2023/Nov/29/llamafile

Looks like a pretty good way to run Shit tbh

Jump in the discussion.

No email address required.

Reported by:
  • X : transphobia

just download and run my binary

:marseydisagree:

Jump in the discussion.

No email address required.

transphobia

Jump in the discussion.

No email address required.

Now, theoretically, how would I get python to interact with the web interface

Actually what I'm gonna do is try to run llama cpp and python reps so I can just use python to ask it stuff while it runs in C++

Apparently this setup is super lightweight

Jump in the discussion.

No email address required.

lol idk just get it running first and start from there i guess

Jump in the discussion.

No email address required.

:marseydance:

Jump in the discussion.

No email address required.

i use a chromebook bro lol i'm not running anything locally

but even running remotely usually isn't as simple as it might seem so it might be a fair bit of trial and error when your parts arrive and you give it a go

Jump in the discussion.

No email address required.

I installed a github repo for the first time in my life today so i have high hopes

Jump in the discussion.

No email address required.

lol never a bad thing to start but i guess there's a charm in trying to figure out how to solve the frick ups that might happen

Jump in the discussion.

No email address required.

Link copied to clipboard
Action successful!
Error, please refresh the page and try again.