Apple Outclasses ChatGPT & Top Predictions for 2024

🏝 TheTechOasis 🏝

Breaking down the most advanced AI systems in the world to prepare you for your future.

5-minute weekly reads.

TLDR:

AI Research of the Week: Apple Outclasses GPT-4V

Leaders: My Top Predictions for 2024

🎅 AI Research of the week 🎅

First and foremost, Happy New Year!!

Out of all AI news at the end of 2023, the fact that Apple is finally putting itself out there is probably the most important of all.

Although released back in October, an Apple model named Ferret has been making headlines recently.

Not only is it a first of its kind, but also beats ChatGPT-Vision in several tasks while offering new cues to what Apple may be cooking for the world in 2024.

The Importance of Getting Things Right

Out of all the issues you can face with a Language Model, hallucinations usually come first.

As Large Language Models (LLMs) are essentially stochastic word predictors, there is always some certain level of unpredictability in every response they give.

Consequently, when using these models, you must provide the model with relevant context in the actual prompt so that its response isn’t made up.

For instance, if you are preparing a summary of a news event that took place recently and, thus, the model does not know about it, you embed the article directly into the prompt so that the model uses it.

This process is called “in-context learning”, providing the relevant context for the response in real time.

This works because LLMs have seen so much data during training that they are capable of using data they have never seen before effectively, a process we describe as ‘zero-shot learning’ or ‘few-shot learning’ when you provide many examples.

However, while grounding the model on text data is as simple as appending the required data into the prompt, grounding the model on images is a different story.

In other words, what if the necessary ‘cues’ the model needs to answer effectively are objects in an image?

For example, semantically speaking, the response to the question “What can I cook with this” while pointing at a frying pan will be different depending on the ingredients in the table.

As context matters, grounding is essential, yet extremely complex to achieve in multimodal situations.

Adding insult to injury, when pointing to objects in an image to help the MLLM perform better, objects most often have irregular shapes, are partially covered by other objects, or can be poorly visible due to resolution issues.

This makes MLLMs extremely hallucination-prone. But Apple’s Ferret MLLM has a solution for this.

Free-form Referring and Grounding

Put simply, Ferret is the first MLLM that successfully performs referring and grounding on any object in an image in a free-form style.

In other words, it can successfully refer to any object in an image or ground its responses on any object in that image.

All in one model.

As we can see in the image above, the user can point to objects in an…

Enjoyed the introduction? Subscribe or sign in to continue with this edition.