[App] ToshLLM — local LLMs on Intel + AMD GPU (Metal, AMD‑patched llama.cpp, open source)

[Cyberdevs]

5 minutes ago, engeldlgado said:

Thanks for the feedback.

Sure thing man.

 

6 minutes ago, engeldlgado said:

Was the AI in the middle of generating a response when this error occurred?

No I just wanted to test and see how the app handles file attachment and attached several files and the error occurred, but it was able to analyze a single somewhat short text file without any errors. I have to say that the files I've attached were pretty large files so I guess that's what cause the error.

 

8 minutes ago, engeldlgado said:

I will note it down, but keep in mind that hardware combination might simply hit its limits when benchmarking a 4B model like Qwen3.

Thanks, yeah I didn't expect much from that rig but since you've asked for a benchmark on Polaris/Vega GPUs though I share my experience. 

10 minutes ago, engeldlgado said:

Also you tested the experimental engine? maybe work better because it has a custom kernel for AMD.

I will give it a try later and keep you posted.

[XanthraX]

15 hours ago, engeldlgado said:

Try to use the smallest one first to test, btw, what kind of system spec you have?

Qwen 4B

I have both systems in my signature. Both based on CoffeeLake CPU's. RX 560 and 580.

 

[engeldlgado]

17 minutes ago, XanthraX said:

I have both systems in my signature. Both based on CoffeeLake CPU's. RX 560 and 580.

 

 

Sorry, i didnt notice because i was on the phone when i replied to you..

 

Your RX-580 is GCN/Polaris, not RDNA+...

My AMD decode kernel is only instantiated for RDNA+ (RX-5000/6000 series) maybe others but needs further testing, so ToshLLM won't work atm on your RX-580 and 560

 

But I'm going to study integrating a GCN/Polaris-compatible patch. I'll need to rewrite the kernel to use 64-lane SIMD groups instead of RDNA's 32-lane simdgroups, which is more complex, but I'm interested in exploring it.

 

I'll also study llama-metal old repo that i saw searching for this issue... to see if I can port it, to my patch to it and optimize it better for GCN GPUs.

 

I'll update if I make progress on GCN support... would you be willing to test it when I get a working solution?

Edited 17 hours ago by engeldlgado

[XanthraX]

Don't worry, I figured out something like this. I will try if you succeed with that update.

[engeldlgado]

Hi @Cyberdevs

Quick update, that i've work today:

Update (v0.81.25): you can now attach files in chat — including PDFs (text is extracted automatically, and scanned PDFs are read with on-device OCR), plus more text formats. And image input for vision models is in: drop in a vision model with its mmproj (e.g. gemma-3-4b) and you can attach an image and ask about it. Vision is experimental and the image encoder runs partly on CPU on AMD GPUs (some Metal ops aren't supported), so it works but isn't fully GPU-accelerated yet. DMG is building now.

 

Also i've add a option to change the default location for models




Also may ask you for a new test on the RX Card... Update the app, and just load a model and start the server, no benchmark, anyting, just send me the logs, im researching about the VEGA/GCN Cards...

Edited 6 hours ago by engeldlgado

[Alpha22]

23 hours ago, engeldlgado said:

 

 

Can you test the experimental engine too in settings? it has a improvements about speed, etc

Settings?

[engeldlgado]

1 hour ago, Alpha22 said:

Settings?

 
Yeah in settings... theres is an option to change the Inference Engine (llama.ccp) bundle, the experimental one, has better improvements against the normal one

[Cyberdevs]

@engeldlgado

I will give it a try when I can and I'll post the log here.
Thanks for the updates and your efforts 👍

[Alpha22]

1 hour ago, engeldlgado said:

 
Yeah in settings... theres is an option to change the Inference Engine (llama.ccp) bundle, the experimental one, has better improvements against the normal one

[engeldlgado]

29 minutes ago, Alpha22 said:

Excellent, it improves the performance on your end... i will take notes about your build, thanks for testing!

[Cyberdevs]

On my AMD RX6800XT

Two top benchmarks are after enabling these settings in version Version 0.81.26 (0.81.26):

2 hours ago, engeldlgado said:

Yeah in settings... theres is an option to change the Inference Engine (llama.ccp) bundle, the experimental one, has better improvements against the normal one

 

I'll test my RX580 later and post the results.