Large Concept Models: Language Modeling in a Sentence Representation Space

This research paper introduces a new approach to language modeling called a Large Concept Model (LCM). Instead of predicting the next word in a sequence, the LCM predicts the next sentence, using a special code that represents the meaning of each sentence. The researchers experimented with different ways to train the LCM, including using a method called “diffusion” which gradually adds noise to the sentence codes and then trains the model to remove the noise. They found that the LCM performs well on tasks like summarizing text and expanding short summaries into longer texts. The LCM also shows promise for working with multiple languages, even languages it hasn’t been specifically trained on. The researchers believe that the LCM has the potential to be even more powerful in the future with further development.

https://arxiv.org/pdf/2412.08821