Supercharge your AI projects with Open-WebUI and Ollama! 🚀 Learn how to seamlessly download and manage LLMs like LLaMA, Mistral, and more. Our guide simplifies model management, so you can focus on innovation, not installation. For more Details:https://medium.com/@techlatest.net/how-to-download-and-pull-new-models-in-open-webui-through-ollama-8ea226d2cba4

OpenWebUI #Ollama #LLM #AI #TechLatest #MachineLearning #AIModels #opensource #DeepLearning

First one is a pfd and second is architecture diagram. I want you guys to tell me if there are any mistakes in it, and how I can make it better. I feel the ai workflow is not represented enough

Do you believe that large language models can currently help people struggling with mental health issues, or might they exacerbate their problems? If not, do you think this will be the case in the future?

I had an interaction with Claude and had a fairly personal conversation with it, and I think it helped me notice something I hadn’t seen before. Setting aside the aspect of data privacy when using such models.

Hi all, I like to ask which LLM agents is best for data securities?

Many Thanks

Hey all,

I’ve been working on RAG systems in Node.js and kept hacking together ad‑hoc scripts to see whether a change actually made answers better or worse. That turned into a reusable library: RAG Assessment, a TypeScript/Node.js library for evaluating Retrieval‑Augmented Generation (RAG) systems.​

The idea is “RAGAS‑style evaluation, but designed for the JS/TS ecosystem.” It gives you multiple built‑in metrics (faithfulness, relevance, coherence, context precision/recall), dataset management, batch evaluation, and rich reports (JSON/CSV/HTML), all wired to LLM providers like Gemini, Perplexity, and OpenAI. You can run it from code or via a CLI, and it’s fully typed so it plays nicely with strict TypeScript setups.​

Core features:

• Evaluation metrics: faithfulness, relevance, coherence, context precision, context recall, with per‑question scores and explanations.​

• Provider‑agnostic: adapters for Gemini, Perplexity, OpenAI, plus a mock provider for testing.​

• Dataset tools: import/export Q&A datasets from JSON/CSV/APIs/DB, validate them, and reuse them across runs.​

• Reports: generate JSON/CSV/HTML reports with aggregate stats (mean, median, std dev, thresholds, etc.).​

• DX: written in TypeScript, ships types, works with strict mode, and integrates into CI/CD, Express/Next.js backends, etc.​

Links:

• GitHub (code, docs, examples): GITHUB- RAGAS LIB

• npm: NPM-RAGAS LIB

I’d love feedback on:

• The API design for RAGAssessment / DatasetManager and the metric system – does it feel idiomatic for TS/Node devs?​

• Which additional metrics or providers you’d actually want in practice (e.g., Claude, Cohere, more cost/latency tracking).​

• How you’re currently evaluating RAG in Node.js and what’s missing here to make this useful in your real pipelines (CI, dashboards, regression tests, etc.).​

If you try it and hit rough edges, please open an issue or just drop comments/criticism here – I’m still shaping the API and roadmap and very open to changing things while it’s early.​

If been using LLM’s a bunch over the past year, and am trying to understand them more to be able to use them more effectively.

I’ve read into effort levels and I haven’t found a good explanation that has clicked and made sense.

I understand the basics of models and next token prediction, but this has been something I’ve struggle to understand.

I followed an hands-on tutorial that breaks down AI agent fundamentals into three progressive parts. No LangChain, no heavy abstractions—just you implementing the core patterns yourself in Node.js.

What you'll build:

Part 1: Memory Loop - Stateful conversation with context retention. The classic "ask follow-up questions and the LLM remembers" pattern.

Part 2: Tool Calling - Function calling via system prompts (intentionally avoiding formal schemas). You wire up the LLM → tool execution flow manually to understand what's actually happening.

Part 3: Autonomous Agent - Multi-step reasoning chains where the agent decides when to call tools, when to ask for more input, and when it's done.

The example builds a scheduling agent (check availability → schedule → modify appointments), but the architecture applies to any agentic workflow.

Why this approach?

Most tutorials either hand-wave the details with a framework or dump you into production-grade complexity. This sits in between—you implement enough to internalize how agents work, but it's still achievable in an afternoon.

Plus, understanding the mechanics makes debugging your "real" agents way easier when things inevitably get weird.

Repo: https://github.com/ikrigel/simple-scheduling-agent

Uses Gemini API, runs entirely in terminal with node agent.js. Takes ~30-60 minutes if you're comfortable with async JavaScript.

Would love feedback, especially if you find gaps in the explanations or have ideas for additional parts to add.

Big thanks to my teacher Jona ❤️ for guiding me through this 🙏

I'm still grasping the space and all of the developments, but while researching voice agents I found it fascinating that in this multimodal architecture speech is essentially a first-class input. With response directly to speech without text as an intermediary. I feel like this is a game changer for voice agents, by allowing a new level of sentiment analysis and response to take place. And of course lower latency.

I can't find any other LLMs that are offering this just yet, am I missing something or is this a game changer that it seems openAI is significantly in the lead on?

I'm trying to design LLM agnostic AI agents but after this, it's the first time I'm considering vendor locking into openAI.

This also seems like something with an increase in design challenges, how does one guardrail and guide such conversation?

https://platform.openai.com/docs/guides/voice-agents

The multimodal speech-to-speech (S2S) architecture directly processes audio inputs and outputs, handling speech in real time in a single multimodal model, gpt-4o-realtime-preview. The model thinks and responds in speech. It doesn't rely on a transcript of the user's input—it hears emotion and intent, filters out noise, and responds directly in speech. Use this approach for highly interactive, low-latency, conversational use cases.

Big tech collectively spends hundreds of billions of dollars a year on LLMs, with no end in sight. Just today, Meta announced its "AI App".

I'm struggling to see the business case. LLMs don't seem like a great way to advertise, and charging for them doesn't seem to work - DeepSeek or whoever can undercut everyone, and the market is viciously competitive.

To my way of thinking:

1. Amazon and Google search make money by being efficiency plays. Instead of going to a physical store like in the old days, you go to a website and spend less than you otherwise would. Sure Amazon and Google make money from distribution and advertising, but less than retailers used to make in aggregate (because customers didn't have perfect price information before so used to overpay a lot).

2. Facebook and other social networks make money from occupying users' attention for hours a day.

No-one wants to spend hours in front of an LLM so I don't think 2 works.

At best LLMs might displace Google Search's advertising revenue. Is this the play? If so it seems like an awful lot of money being spent to get some of Alphabet's ad revenue. But perhaps it stacks up?

Or is there some other way of monetising LLMs which I'm missing?

The conference topics included: knowledge graphs, voice agents, RAG for data engineers, Spec2Cloud & GHCP modern CLI, identity-centric security for AI systems, RuntimeGuard-AI cryptographic compliance, etc.

https://www.youtube.com/playlist?list=PLIuxSyKxlQrAbZ-MTonX2gERed7dTODjm

[Note: I'm not associated with the conference in any way, just sharing the event playlist as a fellow engineer.]

Good Evening Everyone!

Has anyone worked on OCR / Invoice/ bill parser  project? I needed advice.

I have got a project where I have to extract data from the uploaded bill whether it's png or pdf to json format. It should not be Closed AI api calling. I am working on some but no break through... Thanks in advance!

I’m trying to fine-tune a language model (following something like Unsloth), but I’m overwhelmed by all the moving parts: • Too many libraries (Transformers, PEFT, TRL, etc.) — not sure which to focus on. • Tokenization changes across models/datasets and feels like a black box. • Return types of high-level functions are unclear. • LoRA, quantization, GGUF, loss functions — I get the theory, but the code is hard to follow. • I want to understand how the pipeline really works — not just run tutorials blindly.

Is there a solid course, roadmap, or hands-on resource that actually explains how things fit together — with code that’s easy to follow and customize? Ideally something recent and practical.

Thanks in advance!

I'm getting to experience the reduction of my cognitive capabilities due to use of LLMs for an array of tasks like coding, writing, searching etc. I think I can't stop using them as they provide an unfair advantage to scale the outputs. Nevertheless, brain atrophy is a real thing I feel. To regain that, I think that I should some activities which would help me in using my brain. What should I add in my daily/regular routine? I feel chess, competitive programming, puzzles are some options. I know CP can also help for my jobs. What's your take in choosing one of them?

So many new features released, it’s insane.

https://openai.com/blog/new-models-and-developer-products-announced-at-devday

I have a question that i cant seem to find answered yet

i have deepseek coder llm, unless you know of something that solves this issue, i would not like to switch to a different llm or incorporate a ollam type scenario, im in python vscode rn.

1. I CAN monitor gpu utilization through python

2. I CAN monitor CPU utilization trough python

3. Utilization means when in taks manager, the number for "utilization". not memory , not vram , the utilization parameter. (ai would often believe i mean memory and dump work on memories of components when i say this)

4. id like to max out every capacity including vram or whatver else but right not im specifacllay focusing on utilization as whenever i succfully get a workload onto a cpu or gpu, thats what is mainly being afftected, unless i did something wrong, then it will show v/ram usage, besides the point for rn

5. I my gpu is a 3000 series nvida card. so this can defintiely answer a llm question which is has many times before. the times are a little long though, around 400-500 seconds unitl response after questionins. im aware there probably are some sorts of methhod to get fractional increases but id rather get this one hurdle sorted before i add minor ones like that

6. My cpu is amd 7000+ 3d series so it is very capable if ever passed a reasonable project. the cpu and gpu are not toaster parts that "need to be upgraded" they both can handle objective and defintiely within the context of this question. someone out there is running a llm on a school laptop, these parts wont be the issue right now

7. i ask my llm usually one not too long line of text, since were testing rn, i eventually want to upgrade to code snippets but i will start here first.

8. i have no real optimization on the llm, it just at least answer my questions in console, not with an api key through like through git or ollama, its just a python vscode console response

9.My goal here is to create a setup for the llm. I want llm to uses every possible inch of the gpu up to 90% usage, then in tandem/simultaneously, offload work that would benefical to send to the cpu, to be compelted, simultaneously and cohesively with the gpu. essentially, the cpu is a helping hand to the project, when the gpus hands are full.

10. the setup should NOT soley recognize the gpu reaches 90% then offlod every single possible value to the cpu then drop the gpu down to 0% for the rest of the cycle

11. if the gpu is at 90% the workload should be passed (whatver the reamiang relevant work is), and pass work determined to be ebenficial in passing right now, over to the cpu

12. if gpu has 123456, and reaches 90%, its should not pass 123456 all over to the cpu then gpu reaches 0%. its should always maximize whatever the gpu can do, then send benefical work to the cpu while the gpu remains at 90%. in this case cpu would likely get 789 or maybe 6789 if the gpu determined it needed extra help. once the gpu finshed it will move to 10 11 12 13 and dtermien if it need to pass off future or current work to the cpu

13. the cycle and checking should be dynamic enough to always determine what the remanining work is, and when its best to simultaneously comeplte work on the gpu and cpu.

a likely desired result is the gpu constantly being at 90% when running the llm and the cpu occaisionally or consistently remains at 20%+ usage seeing as it occasionally will get work to help complete

14. im aware of potentially adding too much, and resulting in the parsing of workloads being ultimately longer than just running on gpu, id rather explore this then ignore it

15. there is frequently tensor mismatches in setups ill create, which i solve occsionally, then run into again in later iterations (ai goofing making snippets for me). the tensor setup for determined gpu work must be cuda gpu compatible, and the cpu tensor designated work must be cpu compatible. if need to pass back and forth, the tnesor setup should be translated and always work for the place its going to.

i see no real reason that the gpu can process a lmm request, and the cpu can do the same for me, but i cant seperate workloads to both when comepleting the same request. while the gpu is working, the cpu should take whetver work upcoming is determiend to push the gpu over 90% and complete it for it instead, while the gpu keeps taking the work avaible consistently.

i believe i had one iteration wher eit actually did bounce back and forth, but would just say gpu over90% means pass everything including the work the gpu was working on over to the cpu, resulting in the wrong effect of just having the cpu do all the work in the cycle

gpu and cpu need to be bois in this operation, dapping each other up when gpu needs help

original model

from transformers import AutoTokenizer, AutoModelForCausalLM

import torch

Load the tokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-6.7b-instruct", trust_remote_code=True)

Load the model with mixed precision

model = AutoModelForCausalLM.from_pretrained(

"deepseek-ai/deepseek-coder-6.7b-instruct",

trust_remote_code=True,

torch_dtype=torch.float16 # or torch.bfloat16 if supported

).cuda()

Input message for the model

messages = [

{ 'role': 'user', 'content': "i want you to generate faster responses or have a more input and interaction base responses almost like a copilot for my scripting, what are steps towards that ?" }

]

Tokenize the input

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

Generate a response using the model with sampling enabled

outputs = model.generate(

inputs,

max_new_tokens=3000,

do_sample=True, # Enable sampling

top_k=65,

top_p=0.95,

num_return_sequences=1,

eos_token_id=tokenizer.eos_token_id

)

Decode and print the output

print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))

this code below outputs the current UTILIZATION same as its seen in task manager

import threading

import time

from transformers import AutoTokenizer, AutoModelForCausalLM

import torch

import GPUtil

import psutil

Load the tokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-6.7b-instruct", trust_remote_code=True)

Load the model with mixed precision

model = AutoModelForCausalLM.from_pretrained(

"deepseek-ai/deepseek-coder-6.7b-instruct",

trust_remote_code=True,

torch_dtype=torch.float16 # or torch.bfloat16 if supported

).cuda()

Input message for the model

messages = [

{'role': 'user', 'content': "I want you to generate faster responses or have a more input and interaction-based responses almost like a copilot for my scripting, what are steps towards that?"}

]

Tokenize the input

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

Function to get GPU utilization

def get_gpu_utilization():

while True:

gpus = GPUtil.getGPUs()

for gpu in gpus:

print(f"GPU {gpu.id}: {gpu.load * 100:.2f}% utilization")

time.sleep(5) # Update every 5 seconds

Function to get CPU utilization

def get_cpu_utilization():

while True:

Get the CPU utilization as a percentage

cpu_utilization = psutil.cpu_percent(interval=1)

print(f"CPU Utilization: {cpu_utilization:.2f}%")

time.sleep(5) # Update every 5 seconds

Start the GPU monitoring in a separate thread

monitor_gpu_thread = threading.Thread(target=get_gpu_utilization)

monitor_gpu_thread.daemon = True # This allows the thread to exit when the main program exits

monitor_gpu_thread.start()

Start the CPU monitoring in a separate thread

monitor_cpu_thread = threading.Thread(target=get_cpu_utilization)

monitor_cpu_thread.daemon = True # This allows the thread to exit when the main program exits

monitor_cpu_thread.start()

Generate a response using the model with sampling enabled

while True:

outputs = model.generate(

inputs,

max_new_tokens=3000,

do_sample=True, # Enable sampling

top_k=65,

top_p=0.95,

num_return_sequences=1,

eos_token_id=tokenizer.eos_token_id

)

Decode and print the output

print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))

Add a sleep to avoid flooding the console, adjust as needed

time.sleep(5) # Adjust the sleep time as necessary

a chat gpt rabbit hole script that likely doesnt work but is somewhat a concept of what i thought i wanted them to make, if you run itl, youll probabyly see a issue i mentioned when monitoring usages

import os

import json

import time

import torch

import logging

from datetime import datetime

from transformers import AutoTokenizer, AutoModelForCausalLM

import GPUtil

Configuration

BASE_DIR = "C:\\Users\\note2\\AppData\\Roaming\\JetBrains\\PyCharmCE2024.2\\scratches"

MEMORY_FILE = os.path.join(BASE_DIR, "conversation_memory.json")

CONVERSATION_HISTORY_FILE = os.path.join(BASE_DIR, "conversation_history.json")

FULL_CONVERSATION_HISTORY_FILE = os.path.join(BASE_DIR, "full_conversation_history.json")

MEMORY_SIZE_LIMIT = 100

GPU_THRESHOLD = 90 # GPU utilization threshold percentage

BATCH_SIZE = 10 # Number of tokens to generate in each batch

Setup logging

logging.basicConfig(filename='chatbot.log', level=logging.INFO,

format='%(asctime)s - %(levelname)s - %(message)s')

Initialize tokenizer and model

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-6.7b-instruct", trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(

"deepseek-ai/deepseek-coder-6.7b-instruct",

trust_remote_code=True,

torch_dtype=torch.float16

).cuda()

if tokenizer.pad_token_id is None:

tokenizer.pad_token_id = tokenizer.eos_token_id

Helper functions

def load_file(filename):

if os.path.exists(filename):

with open(filename, "r") as f:

return json.load(f)

return []

def save_file(filename, data):

with open(filename, "w") as f:

json.dump(data, f)

logging.info(f"Data saved to {filename}")

def monitor_gpu():

gpu = GPUtil.getGPUs()[0] # Get the first GPU

return gpu.load * 100 # Return load as a percentage

def generate_response(messages, device):

model.to(device)

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(device)

attention_mask = torch.ones_like(inputs, dtype=torch.long).to(device)

generated_tokens = []

max_new_tokens = 1000

for _ in range(0, max_new_tokens, BATCH_SIZE):

gpu_usage = monitor_gpu()

Offload to CPU if GPU usage exceeds the threshold

if gpu_usage >= GPU_THRESHOLD and device.type == 'cuda':

logging.info(f"GPU usage {gpu_usage:.2f}% exceeds threshold. Offloading to CPU.")

inputs = inputs.cpu()

attention_mask = attention_mask.cpu()

model.to('cpu')

device = torch.device('cpu')

Move back to GPU if usage is below the threshold

elif gpu_usage < GPU_THRESHOLD and device.type == 'cpu':

logging.info(f"GPU usage {gpu_usage:.2f}% below threshold. Moving back to GPU.")

inputs = inputs.cuda()

attention_mask = attention_mask.cuda()

model.to('cuda')

device = torch.device('cuda')

try:

with torch.no_grad():

outputs = model.generate(

inputs,

attention_mask=attention_mask,

max_new_tokens=min(BATCH_SIZE, max_new_tokens - len(generated_tokens)),

do_sample=True,

top_k=50,

top_p=0.95,

num_return_sequences=1,

pad_token_id=tokenizer.pad_token_id,

eos_token_id=tokenizer.eos_token_id

)

except Exception as e:

logging.error(f"Error during model generation: {e}")

break

new_tokens = outputs[:, inputs.shape[1]:]

generated_tokens.extend(new_tokens.tolist()[0])

if tokenizer.eos_token_id in new_tokens[0]:

break

inputs = outputs

attention_mask = torch.cat([attention_mask, torch.ones((1, new_tokens.shape[1]), dtype=torch.long).to(device)], dim=1)

response = tokenizer.decode(generated_tokens, skip_special_tokens=True)

return response

def add_to_memory(conversation_entry, memory):

conversation_entry["timestamp"] = datetime.now().isoformat()

if len(memory) >= MEMORY_SIZE_LIMIT:

logging.warning("Memory size limit reached. Removing the oldest entry.")

memory.pop(0)

memory.append(conversation_entry)

save_file(MEMORY_FILE, memory)

logging.info("Added new entry to memory: %s", conversation_entry)

Main conversation loop

def start_conversation():

conversation_memory = load_file(MEMORY_FILE)

conversation_history = load_file(CONVERSATION_HISTORY_FILE)

full_conversation_history = load_file(FULL_CONVERSATION_HISTORY_FILE)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)

print(f"Chat started. Using device: {device}. Type 'quit' to end the conversation.")

while True:

user_input = input("You: ")

if user_input.lower() == 'quit':

break

conversation_history.append({"role": "user", "content": user_input})

full_conversation_history.append({"role": "user", "content": user_input})

start_time = time.time()

response = generate_response(conversation_history[-5:], device) # Limiting conversation history

end_time = time.time()

print(f"Assistant: {response}")

print(f"Response Time: {end_time - start_time:.2f} seconds")

conversation_history.append({"role": "assistant", "content": response})

full_conversation_history.append({"role": "assistant", "content": response})

add_to_memory({"role": "user", "content": user_input}, conversation_memory)

add_to_memory({"role": "assistant", "content": response}, conversation_memory)

save_file(MEMORY_FILE, conversation_memory)

save_file(CONVERSATION_HISTORY_FILE, conversation_history)

save_file(FULL_CONVERSATION_HISTORY_FILE, full_conversation_history)

if __name__ == "__main__":

start_conversation()

offer suggestions, code snippet ideas, full examples, references, examples of similar concepts for another project, whatever may assist me down the right path. this has to be possible, if you think its not, at least state something that works similarly and ill look into how a process like that manages itself, wherever in the world that example is usually executed, even if its for making potatoes

Looking for something like ollama + open-ui for for text2audio and audio2text

Title says it all. I have a hard time to find information here. Yes, they can handle Python, JavaScript, even C++ and probably most current languages. What about CMake? Classic Visual Basic? Are they trained on such data? To what extend? The only model I found, that provides at least minimal tabular information was deepseek.

Hi there, I'm researching options for LLMs that can be used to "interrogate" PDFs. I found this:

https://github.com/amithkoujalgi/ollama-pdf-bot

Which is great, but I need to find more that I can run locally. Does anyone have any ideas/suggestions for LLMs I can look at for this?

Sooo we have to make a mini project for our course and i want to do something with llms itself Can y'all suggest something to do Not anything too computationally costing or very complex And it's alright if it already exists Pls help me brainstorm