Running a Local LLM

[tezvikhyath]

hello, i've been trying to run a local llm (llama3:70b) for fabric and i seem to have a problem here. I dont understand how to even run it for fabric.
fabric --remoteOllamaServer i've tried this line of code in my terminal and i end up with fabric: error: argument --remoteOllamaServer: expected one argument. I'm starving to find a solution for this problem. Thank You.
(im running onna mac m3 )

[xssdoctor]

if you are running it locally on the same computer, it should already be in fabric. no need for --remoteOllamaServer. just run fabric --listmodels and the ollama models should be there. the --remoteOllamaServer is if you are running ollama on a non-default port or another computer

[carlleonard]

Any of the models shown with --listmodels can be used with the option

--model modelName

It works for me. Not been too impressed when using the tiny models like PHI3 mini or the smallest Llama3 with 4bit quantizations. I'm CPU only so not doing any of the bigger models where it would be measured in seconds per token. Got to get a GPU.

I'd love to hear if anyone gets decent results with reasonably sized models.

[secretgspot]

if you are running it locally on the same computer, it should already be in fabric. no need for --remoteOllamaServer. just run fabric --listmodels and the ollama models should be there. the --remoteOllamaServer is if you are running ollama on a non-default port or another computer

it is on a local computer except ollama is installed via windows installer and fabric inside of wsl. ollama models are not found in fabric because fabric is running on 127.0.1.1 while windows machine with ollama is on 127.0.0.1

[mfslena]

Noob question: I was able to run using the cli parameter, but it does not work just by configuring the environment variables. My ollama server is not the same of the one I'm running fabric.

In this case, do I need to use the cli parameter? I would like to be able to use fabric without the ollama cli parameter, just using the environment variables.

[Melaron]

@ozzyobr I am also very interested to know the answer to this question. I primarily use my laptop, but my desktop serves Ollama. My current workaround is to add --remoteOllamaServer <my_server_url> to all the aliases in ~/.config/fabric/fabric-bootstrap.inc. I would also rather have it specified as an env variable.

[mfslena]

I've not found a solution for this. :(

I'm waiting for the go version of fabric to test and see if it works. If it doesn't support that I'll make a bash script, the downside is that I'll need to drop it in every server I've.

[Samitch1]

I still have the same issue. No local models are showing up and it seems there is no communication between Ollama and Fabric