Grounding LLMs into the physical world

Prof. Dr. Elia Bruni
Serwan Jassim

Veranstaltungstyp: Blockseminar
TeilnehmerInnen:

Beschreibung:
In this block course, we explore the intriguing debate about whether Large Language Models (LLMS) , known for learning through next-word prediction, can develop an understanding of the physical world. The course highlights the recent emergence of multimodal LLMs, like GPT-4, which learn from both text and visual inputs, and examines the extent to which this multimodality might enhance language grounding and physical understanding.

A core part of the course involves analyzing the traditional research approach in this domain, which focuses on LLMs performing next-word prediction tasks. We critically assess whether this method truly tests a model's understanding of physical concepts,. The course also proposes a novel evaluation framework for assessing 'genuine' physical understanding in LLMs. This framework involves using realistic physical simulators as proxies for the real world, requiring LLMs to solve tasks through sensorimotor interaction with these simulators. This approach aims to test the models' ability to bridge natural language with simulated environments and their grasp of intuitive physics, offering a more comprehensive evaluation of their cognitive abilities.

Learning objectives:

- Understand the current state and potential of LLMs in simulating human-like cognition.
- Evaluate the effectiveness of multimodal learning in LLMs for physical understanding.
- Critically analyze traditional next-word prediction tasks in assessing physical comprehension.

Erstes Treffen:
Montag, 16.09.2024 12:00 - 18:00, Ort: 93/E44

Ort:
93/E44: Montag, 16.09.2024 12:00 - 18:00, Dienstag, 17.09.2024 09:00 - 12:00, Dienstag, 17.09.2024 13:00 - 16:00, Mittwoch, 18.09.2024 09:00 - 12:00, Mittwoch, 18.09.2024 13:00 - 16:00, Donnerstag, 19.09.2024 09:00 - 12:00, Donnerstag, 19.09.2024 13:00 - 16:00, Freitag, 20.09.2024 09:00 - 12:00, Freitag, 20.09.2024 13:00 - 16:00, Montag, 23.09.2024 12:00 - 18:00, Dienstag, 24.09.2024 09:00 - 12:00, Dienstag, 24.09.2024 13:00 - 16:00, Mittwoch, 25.09.2024 09:00 - 12:00, Mittwoch, 25.09.2024 13:00 - 16:00, Donnerstag, 26.09.2024 09:00 - 12:00, Donnerstag, 26.09.2024 13:00 - 16:00, Freitag, 27.09.2024 09:00 - 12:00, 93/E31: Freitag, 27.09.2024 13:00 - 16:00

Semester:
SoSe 2024

Zeiten:
Termine am Montag, 16.09.2024 12:00 - 18:00, Dienstag, 17.09.2024 09:00 - 12:00, Dienstag, 17.09.2024 13:00 - 16:00, Mittwoch, 18.09.2024 09:00 - 12:00, Mittwoch, 18.09.2024 13:00 - 16:00, Donnerstag, 19.09.2024 09:00 - 12:00, Donnerstag, 19.09.2024 13:00 - 16:00, Freitag, 20.09.2024 09:00 - 12:00, Freitag, 20.09.2024 13:00 - 16:00, Montag, 23.09.2024 12:00 - 18:00, Dienstag, 24.09.2024 09:00 - 12:00, Dienstag, 24.09.2024 13:00 - 16:00, Mittwoch, 25.09.2024 09:00 - 12:00, Mittwoch, 25.09.2024 13:00 - 16:00, Donnerstag, 26.09.2024 09:00 - 12:00, Donnerstag, 26.09.2024 13:00 - 16:00, Freitag, 27.09.2024 09:00 - 12:00, Freitag, 27.09.2024 13:00 - 16:00, Ort: 93/E44, 93/E31

Leistungsnachweis:
Students will be evaluated based on group presentations and a final in-class exam. The group presentations will involve presenting a paper and implementing related code developed during the group work sessions. The grading breakdown is as follows: Group Presentation: 70% In-Class Exam: 30%

Veranstaltungsnummer:
8.3600

ECTS-Kreditpunkte:
8

Bereichseinordnung:
Veranstaltungen > Cognitive Science > Bachelor-Programm Veranstaltungen > Cognitive Science > Master-Programm Courses in English > Human Sciences (e.g. Cognitive Science, Psychology)