this post was submitted on 18 Jun 2023
106 points (100.0% liked)
Programmer Humor
37070 readers
39 users here now
Post funny things about programming here! (Or just rant about your favourite programming language.)
Rules:
- Posts must be relevant to programming, programmers, or computer science.
- No NSFW content.
- Jokes must be in good taste. No hate speech, bigotry, etc.
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Sure it can create new words. It can't create new tokens, would be more correct, I think. But a token is just a text fragment, and, as far as I know, they can range from being several words to being single characters.
I got it it say vilumplox. It doesn't return any Google search results.
Tokens are usually never multiple words. Think of them like "information units". If you have a plural word "hats", there are two tokens: the "hat" one and the "s' that adds more information about it being plural. Combinations of words only really occur for proper names.
If it was Windows 95 it could generate them
https://www.youtube.com/watch?v=cwyH59nACzQ&t=306s