The smallest edge AI device for local LLMs

Posted by alex-moon 18 hours ago

Comments

Comment by ByteOfWood 17 hours ago

This entire page (except for the footer) is made with images. Is there any reason they would do this? Are they making these images by hand or are they generated somehow? Just seems like an unusual choice.

Comment by dexterdog 17 hours ago

Maybe it's supposed to be less readable by AI but I really think it just makes it a little harder for humans to scrape it.

Comment by chedieck 17 hours ago

So weird. There is some compression going on too, which makes it all at least a little ugly.

Comment by tristanj 17 hours ago

I don't understand the value proposition here.

This device is selling for $1,999. In the product page on their website, they show the device plugged into a MacBook Pro.

A MacBook Pro already has excellent ability to run local LLMs, so why wouldn't someone just spend that $1,999 on maxing out their MacBook specs (M5 Max with extra RAM) instead?

Or why not buy a Mac Mini with 48GB of RAM for $2300?

Comment by pizza234 16 hours ago

> Or why not buy a Mac Mini with 48GB of RAM for $2300?

Because this device has 80 GB, which allows running larger models (which is the whole point of the device).

AFAIK, the next significant step in terms of quality above 32 GB is 256 GB, but 80 GB still buys convenience.

> A MacBook Pro already has excellent ability to run local LLMs, so why wouldn't someone just spend that $1,999 on maxing out their MacBook specs (M5 Max with extra RAM) instead?

80 GB extra (48 → 128) cost as much as this device, but the base MBP is very expensive already (4000+$). This device is essentially a "cheap" addon for an existing system.

Comment by RataNova 10 hours ago

[flagged]

Comment by majorchord 16 hours ago

> why wouldn't someone just spend that $1,999 on maxing out their MacBook specs

Perhaps they don't want a new/upgraded machine or prefer a more portable solution.

Comment by simonw 17 hours ago

I hope for their sake they bought all the RAM they needed to fulfill those Kickstarter orders at least six months ago.

Comment by makefu 17 hours ago

The kickstarter comments page of this product does not look so glamorous ( https://www.kickstarter.com/projects/tiinyai/tiiny-ai-pocket... )

Comment by alsetmusic 16 hours ago

Compatible with Mac and Windows, but no Linux support? That's an odd choice. I would have picked Mac and Linux and de-prioritized Windows as a nice-to-have when there's time to get around to it.

Comment by vkaku 13 hours ago

It's a good step in the right direction. Hardware reliability and testing needs to happen. Tell me how long it can run without heating up.

Comment by majorchord 17 hours ago

Ancient models, at < 30t/s, for $2,000? Neat device but I'll have to pass.

Comment by raajg 17 hours ago

Is it just me, or others are also weirded out with that website being all images including all the text

Comment by amazingamazing 17 hours ago

Pretty decent but needs more details. Overall I give it a thumbs down. You can buy a 64GB Macbook Pro M1 Max in pretty good condition right now for a good bit less than 2K.

In practice you are better off:

- if youre in the mac ecosystem using 2k and upgrading to the ultra class chip next time you want to buy

- if youre into windows wait for the next ryzen ai max chip slated for next year.

- if you want desktop consider gx10 given that you need a computer to use this anyway

In retrospect it is crazy how good the m1 max series is given the amount of ram and memory bandwidth.

Comment by ingen0s 17 hours ago

I am confused as to weather this is software or a product

Comment by majorchord 17 hours ago

I think it's basically an NPU hanging off a USB connection, so it can be used as a compute device for local LLM inference instead of your existing CPU/GPU.

Comment by simonw 17 hours ago

The Kickstarter page makes that a little more obvious - it's hardware: https://www.kickstarter.com/projects/tiinyai/tiiny-ai-pocket...

Comment by micromacrofoot 17 hours ago

both, but def hardware

> SoC (Armv9.2

> CPU + NPU) 30TOPS +

> dNPU 160TOPS

Comment by RataNova 10 hours ago

[dead]

Comment by 17 hours ago