Small Language Models

When I was 10, I thought my Commodore 64's ability to insult me was marvelous.

I started programming on a Commodore 64 in early 1989. One of the first programs I remember playing with, which was probably largely copied from a book borrowed from the public library, fascinated me a lot. It was called “insults”. And it effectively generated a mad-lib insult, by randomly selecting a random word from each of four word arrays.

You [adjective 1] [noun 1] of [adjective 1] [noun 1].

And I had a blast for weeks thinking of new, colorful words to add to these four arrays, and being awed by the number of possible insults my little program could generate.

For example, given the following four word lists, I’d have 81 possible unique insults:

Adjective 1

  • steaming
  • smelly
  • ugly

Noun 1

  • pile
  • tub
  • vat

Adjective 2

  • vile
  • repulsive
  • unbecoming

Noun 2

  • rubbish
  • garbage
  • vomit

You can see where this is going… And why a 10-year-old thought it was funny. I had thousands of possible insults, just waiting to be generated!

You steaming tub of repulsive garbage!

LOL. Right? Right? Why aren’t you laughing??

Playing with ChatGPT lately has reminded me of this early experience of mine. Not because ChatGPT is great at creating bathroom humor at the level of a 10-year-old. But because ChatGPT is just about as smart.

Don’t get me wrong. ChatGPT is impressive. It’s a great tool. Far more broadly useful than my early insults program, to be sure. But it’s not intelligent.

It still just re-organizes inputs into (possibly) interesting outputs, without any actual understanding of what it’s doing or saying. It’s a master of bullshit. Not in the sense that it outputs lies, but in the sense that it has no concern for whether it outputs lies. It’s concern is whether its output matches a language model. A very impressive, complex, and convincing language model. But still just a language model. GIGO.

Just as my insults program harbored no ill-will against me, or any of its other hapless users, ChatGPT posesses no intelligence. It’s built on a language model. Not a knowledge, intelligence, or wisdom model.

Share this