A few thoughts on LLMs

A few thoughts on LLMs

I need to sit down and combine some of these thoughts into a coherent piece but for now, I just want to dump it all here for reference later. I doubt any of these are original and I have certainly seen some elsewhere.

  1. Market structure for LLM makers may end up being like the airline market. High fixed cost to set up, hard to create a product differentiation that users care about (other than price), lots of competitors entering, at least in part due to the “prestige” of owning one. Lots of utility for consumers but hardly any profit for producers.

  2. An LLM with an infinite context window, one that can contain all my life, will be an entirely different product than an LLM with a limited context window. You can never have enough when it comes to context windows.

  3. Notwithstanding 1, “personality” makes a huge difference in the experience of working with an LLM and the ability to create the right one could determine whether a model can dominate a market or a niche. Claude Sonnet 3.5 absolutely had than special sauce in my experience. We need a lot more of it.

  4. This.:

    I don’t see how we’re going to avoid a situation where the internet become lousy with AI-created, pseudo academic writing filled with made up facts and quotes, which will then get cemented into “knowledge” as those articles become the training fodder for future models.

    But combine it with the fact that if we all start getting our answers from LLMs, the online content & ad based business model goes caput and then what is the incentive for people to put up good content on the web? None.

  5. But anyone who has good proprietary, verified, high-quality data & content will potentially control the value for the customer even as base LLMs become a commodity. Therefore does more data and content start going behind the paywall? If it doesn’t it becomes training data and cannot be monetized.