Introducing LFM3: Every Modality for Specialized Intelligence

[flagged]

This was using llama-serve on my local which by default gave it current time & unhelpfully "get_info" which is specific to the browser

The full request was

``` {"messages":[{"content":"What animal lays the biggest egg?","role":"user"}],"return_progress":true,"sse_ping_interval":1,"stream":true,"tools":[{"function":{"description":"Returns the current local date and time in ISO 8601 format, with the IANA time zone name","name":"get_datetime","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"},{"function":{"description":"Get runtime info (OS name), may call when user asks about local files or shell commands","name":"get_info","parameters":{"properties":{},"required":[],"type":"object"}},"type":"function"}],"reasoning_format":"auto","reasoning_control":true,"backend_sampling":false,"timings_per_token":true} ```

Disabling tools removes the bit about mentioning instructions.

Just an interesting detail I noticed.

Do y'all have any penalty/regularization w.r.t. reasoning traces or is it pure SFT -> RL?

Thank you for an update. Single step does sound exciting, look forward to check it out once I have the full system and data in place. Would LFM3-Audio-2B provide timestamps and speaker labels for each action item? Did you guys train it for any of those?

No weekends. We need you to get back to work on the next 8B.

Is it a parameter count problem? Ie why are you not seeing the cross modality bump in performance?

Fantastic, I need to try it! Thanks!!

This tracks with my experience with their models. Nice work. It would be interesting to see what they could do if they optimized even more for moderation like this without going so far as to hurt the model performance overall.

I'm largely referring to the fundamental components that can be combined to arrive at a decision. Is this a real life image or text, or is it some sort of art or fiction? Is this in a context of historical events for education, etc? Is this sarcasm? Is this a real threat? Is this nudity? Does this contain violence and gore? Is this suggestive? How old is this character? Is this content sufficiently related to the content it's replying to, or is it spam? Does this response deviate too much from the input request? Is this a business logo?

There was this book called The Inquisitive Mood and it very much reminds me of the chain interrogation process you can do with models now.

This isn't exactly a single use case, it's more like there are use cases for image editing software that use generative models, there are use cases for local LLMs to get a sense for abstract user intent, or to check if a model's action is in line with policy and so on.

As I said, the Liquid AI models are already very good at a lot of this, but considering how good the models are at it, I'm curious how much y'all train on tasks that are moderation-adjacent. If y'all don't have any model that's really specifically trained for these kinds of tasks, it would be interesting to see how close y'all could get if you tried.

This is an actually important problem in the AI space especially for local development.

There are other moderation tasks that are harder, like "is this copyrighted?" or "is this a McDonald's logo?" because of how much copyrighted material there is and how many businesses there are, but it might be possible to get a good feel for "is this likely a trademark?", "is this a business logo?", "does this text seem to describe a popular character?"

To some extent you can combine a model like this with a good database to optimize the filters and identification. Then you can provide a few reference image or text examples that are closer to what you're trying to filter out and the model can say, "yeah, this content is pretty close to your examples."

Google and OpenAI have already released some of their open source safety models and they were really quite bad, from my testing. Yet, Liquid AI hasn't really leaned into promoting their models for moderation purposes despite being excellent. I know there is a lot of political intersection in moderation, especially in the past many years, but I think a model that is smart enough and excels at letting people apply their own rules with fairly good accuracy is important right now.

the playground is bf16 or fp8 depending on the cloud GPU for the playground. I didn't have any issues with cows saying moo but I suspect it's just LLM non-determinism :)