Claude Sonnet 3.5 Release: Token Prices and Jevons Paradox

June 25, 2024

•

5 min read

Chris Mann

Summary · TLDR

With the release of Claude 3.5 Sonnet, there has been a lot of press about Moore's law and the recent LLM price decreases. We put this question to our Co-Founder and Co-CEO Shriyash Upadhyay (Yash). We wanted to share his perspective on this and his view for the future as it relates to token price and token consumption. We think you will enjoy what Yash had to say.

‍

“There's a really interesting idea in economics called Jevons Paradox. Jevon was an economist back in the second industrial revolution. This was when they were dealing with steam engines, and he was studying how coal usage changed as steam engines became more efficient. And the very counterintuitive conclusion he came to is that the more efficient steam engines became, the more total coal ended up being consumed. The cheaper your use of a steam engine became, the more use cases ended up being opened up, which could actually profitably use the steam engine. Total usage increased far more quickly than the prices came down. And this is what led the total amount of coal usage to increase.

‍

This is going to be the case for any good where demand is elastic with respect to price. And this is going to be the case for anything, which is a general-purpose technology, which can be applied in a lot of different places. As the cost comes down, the number of places where you can now profitably apply that thing goes up.

‍

This is also the fundamental dynamic that drives Moore's law. Why does Moore's law keep on happening? Why do we continually see so much R&D put into making chips cheaper? Aren't chips cheap enough already? I mean, back, you know, in the 1960s, imagine the cost of chips then. It's already so much cheaper. Like, why do we need to make things even more efficient, even cheaper? Because the more efficient, the less expensive they become, the more things we find to do with these chips, right? And so that fundamental dynamic is actually the reason why Moore's law continues because chips are general-purpose technology, where the cheaper they get, the more places they can be applied, the more usage they end up getting.

‍

That usage balloons way more quickly than costs come down. This is also what we've seen, for example, with cloud spending. Any given instance on Amazon is way cheaper today than it was a couple of years ago. The total amount of spend on AWS is continuously increasing.

‍

The same dynamic is going to come into play with LLMs. We believe this because we believe that LLMs are general-purpose technology. The way in which this happens is that as prices come down people end up finding more and more expensive ways of using these LLMs. Today, we are just seeing the tip of the iceberg in terms of LLM usage.

‍

Here is another great example. We used to write all of our code in assembly language. In assembly languages, you're writing everything. It is very low level. We're just shifting around individual bits. Back when in the Apollo space program they had a book where they wrote all of their code and the book was taller than the average person, and that was because the code they were writing was so low level, they had to write everything super efficiently.

‍

Now we write code in Python. Python is way, way less efficient than assembly. A good assembly programmer can write a program that runs ten to 100 times faster in assembly than it would run in Python. But we still prefer to use Python because Python reduces development time. It ends up being much nicer to work with. It's much easier to debug. We can guarantee that the program actually works in the way that we want, and if we mess up one bit over there, it doesn't make the whole system collapse. We found increasingly, in some ways inefficient, but actually efficient for business use cases, ways of using that compute by changing how we use programming languages.

‍

If you look at our customer's AI and ML pipelines we see this expansion of LLM use cases happening already. We see our customers today expanding the use of LLMs each time we see prices drop. They can point a use case at an LLM and get results much faster in many cases as compared to other approaches. It's more expensive from a token standpoint to use a LLM but less expensive from a human resource perspective and time to value in many cases.

‍

This is why our customers see our router as an intelligent system for managing cost and performance of their LLM environment. They understand they will need an intelligent cost management capability because they see the expansive set of use cases their companies are asking them to address today and what they expect to address in the future.”

- Yash, Co-Founder and Co-CEO, Martian

‍

We hope you enjoyed hearing Yash’s perspective on this. To go with this discussion we created an estimate of token consumption growth that will accompany each reduction in token price which we share in the image attached to this post.

‍

We’d love to hear what you think! Thanks for reading!

‍

Thank you! Your submission has been received!

Oops! Something went wrong while submitting the form.

Chris Mann