Since I've now listened to one (1) entire talk on AI, I am the world's foremost leading expert.
As such I will be running a localized llama 13B instance. For the first time in over a decade I've bought a PC that wasn't an HP workstation.
Specs:
H110 motherboard w/celeron and 4GB ram alongside Nvidia quadro 5200 8gb
Do you guys think it'll actually run a quantized llama? Is 500W PSU enough?
Jump in the discussion.
No email address required.
lol you'll find out when you try to run it
Jump in the discussion.
No email address required.
I need ur thoughts and prayers
I feel fairly confident esp with quantizing. I'm not expecting fast response and I can always add a second gfx card for 16gb vram.
Or comfortably run the 7B model which apparently sucks tho
Jump in the discussion.
No email address required.
Oh yah one of the dope coooders dropped this gem yesterday: https://simonwillison.net/2023/Nov/29/llamafile
Looks like a pretty good way to run Shit tbh
Jump in the discussion.
No email address required.
Jump in the discussion.
No email address required.
transphobia
Jump in the discussion.
No email address required.
More options
Context
More options
Context
Now, theoretically, how would I get python to interact with the web interface
Actually what I'm gonna do is try to run llama cpp and python reps so I can just use python to ask it stuff while it runs in C++
Apparently this setup is super lightweight
Jump in the discussion.
No email address required.
lol idk just get it running first and start from there i guess
Jump in the discussion.
No email address required.
More options
Context
More options
Context
Jump in the discussion.
No email address required.
More options
Context
More options
Context
use koboldCPP and split over vram and ram
only works with GGML.bin models though
Jump in the discussion.
No email address required.
More options
Context
i use a chromebook bro lol i'm not running anything locally
but even running remotely usually isn't as simple as it might seem so it might be a fair bit of trial and error when your parts arrive and you give it a go
Jump in the discussion.
No email address required.
I installed a github repo for the first time in my life today so i have high hopes
Jump in the discussion.
No email address required.
lol never a bad thing to start but i guess there's a charm in trying to figure out how to solve the frick ups that might happen
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
More options
Context
More options
Context