I Fed 24 Years of My Blog Posts to a Markov Model
Key topics
The fascinating experiment of feeding 24 years of blog posts to a Markov model has sparked a lively debate about the capabilities of traditional statistical models versus modern AI transformers. Commenters are weighing in on whether Markov chains can hold their own against large language models (LLMs), with some arguing that LLMs are essentially sophisticated Markov chains, while others insist they're fundamentally different. The discussion is highlighting the nuances of how these models work, with some pointing out that LLMs' ability to efficiently compute probabilities is the real breakthrough. As one commenter quipped, the comparison is "the question today," with transformers emerging as a clear leap forward in AI capabilities.
Snapshot generated from the HN discussion
Discussion Activity
Very active discussionFirst comment
50m
Peak period
37
6-12h
Avg / period
10.8
Based on 119 loaded comments
Key moments
- 01Story posted
Dec 13, 2025 at 3:19 PM EST
20 days ago
Step 01 - 02First comment
Dec 13, 2025 at 4:10 PM EST
50m after posting
Step 02 - 03Peak activity
37 comments in 6-12h
Hottest window of the conversation
Step 03 - 04Latest activity
Dec 16, 2025 at 3:24 PM EST
17 days ago
Step 04
Generating AI Summary...
Analyzing up to 500 comments to identify key contributors and discussion patterns
Want the full context?
Jump to the original sources
Read the primary article or dive into the live Hacker News thread when you're ready.
Respectfully, absolutely nobody wants to read a copy-and-paste of a chat session with ChatGPT.
I was having a discussion about similarities between Markov Chains and LLMs and short after I found this topic on HN, when I wrote "I can share if you like" was as a proof about the coincidence.
An LLM could be implemented with a Markov chain, but the naïve matrix is ((vocab size)^(context length))^2, which is far too big to fit in this universe.
Like, the Bekenstein bound means writing the transition matrix for an LLM with just 4k context (and 50k vocabulary) at just one bit resolution, the first row (out of a bit more than 10^18795 rows) ends up with a black hole ~10^9800 times larger than the observable universe.
The case for brain states and ideas is similar to QM and massive objects. While certain metaphysical presuppositions might hold that everything must be physical and describable by models for physical things, science, which should eschew metaphysical assumptions, has not shown that to be the case.
Yes, technically you can frame an LLM as a Markov chain by defining the "state" as the entire sequence of previous tokens. But this is a vacuous observation under that definition, literally any deterministic or stochastic process becomes a Markov chain if you make the state space flexible enough. A chess game is a "Markov chain" if the state includes the full board position and move history. The weather is a "Markov chain" if the state includes all relevant atmospheric variables.
The problem is that this definition strips away what makes Markov models useful and interesting as a modeling framework. A “Markov text model” is a low-order Markov model (e.g., n-grams) with a fixed, tractable state and transitions based only on the last k tokens. LLMs aren’t that: they model using un-fixed long-range context (up to the window). k is not negotiable. It's a constant, not a variable. Once you make it a variable, any process can be described as markovian, and the word is useless.
Again, no they can't, unless you break the definition. K is not a variable. It's as simple as that. There's no bound to how big the state can be, but the state cannot be flexible. The markov model uses k tokens, not k tokens sometimes, n tokens other times and whatever you want it to be the rest of the time.
2. “Fixed-size block” is a padding detail, not a modeling assumption. Yes, implementations batch/pad to a maximum length. But the model is fundamentally conditioned on a variable-length prefix (up to the cap), and it treats position 37 differently from position 3,700 because the computation explicitly uses positional information. That means the conditional distribution is not a simple stationary “transition table” the way the n-gram picture suggests.
3. “Same as a lookup table” is exactly the part that breaks. A classic n-gram Markov model is literally a table (or smoothed table) from discrete contexts to next-token probabilities. A transformer is a learned function that computes a representation of the entire prefix and uses that to produce a distribution. Two contexts that were never seen verbatim in training can still yield sensible outputs because the model generalizes via shared parameters; that is categorically unlike n-gram lookup behavior.
I don't know how many times I have to spell this out for you. LLMs do not follow the markov property in any sense of the imagination. You are just plain wrong.
I don't necessarily agree with GP, but I also don't think that a markov chain and markov generator definitions include the word "small".
That constant can be as large as you need it to be.
And in classes, the very first trick you learn to skirt around history is to add Boolean variables to your "memory state". The issue obviously being that this is exponential.
I don't think LLMs embody the markov property at all, even if you can make everything eventually follow the markov property by just "considering every single possible state". Of which there are (size of token set)^(length) states at minimum because of the KV cache.
The markov property states that your state is a transition of probabilities entirely from the previous state.
These states, inhabit a state space. The way you encode "memory" if you need it, e.g. say you need to remember if it rained the last 3 days, is by expanding said state space.
If you define it that way, then yes, a LLM is a "markov chain" of state space size (# tokens)^(context length), at minimum. That's not a helpful abstraction and defeats the original purpose of the markov observation.
Are deliberately missing the point or what?
Okay, so we're agreed.
* LLMs don't use Markov chains, * LLMs don't predict words.
* The R package markovchain[1] may look like it's using Markov chains, but it's actually using the R programming language, zeros and ones.
[1] https://cran.r-project.org/web/packages/markovchain/index.ht...
If you use a syllable-level token in Markov models the model can't form real words beyond the second syllable, and you have no way of making it make more sense other than increasing the size the token size, which exponentially decreases originality. This is the simplest way I can explain it.
MC have constant finite context and their state is the most recent k tuple of emitted alphabets.
You can certainly feed k-grams one at a time to, estimate the the probability distribution over next token and use that to simulate a Markov Chain and reinitialize the LLM (drop context). In this process the LLM is just a look up table to simulate your MC.
But an LLM on its own doesn't drop context to generate, it's transition probabilities change depending on the tokens.
It's like saying "hey guys, want me to Google this and paste the results page here"? It contributes nothing to the conversation.
Hate to be that guy, but I remember this place being nicer.
Everyone has access to ChatGPT. If we wanted its input we could ask it ourselves. Your offer is akin to "Hey everyone, want me to Google this and paste the results page here?"
These posts are low-effort and add nothing to the conversation, yet the people who do it seem to expect everyone to be impressed by their contribution. If you can't understand why people bristle at this, I'm not sure what to tell you.
I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random page in an old 1950s Funk & Wagnalls dictionary and using whatever I found there as a writing seed.
The problem with that is either your n-gram level is too low in which case it can't maintain any kind of cohesion, or your n-gram level is too high and it's basically just spitting out your existing corpus verbatim.
For me, I was more interested in something that could potentially combine two or three highly disparate concepts found in my previous works into a single outputted sentence - and then I would ideate upon it.
So I haven't opened the program in a long time so I just spun it up and generated a few outputs:
I'm not sure what the original pieces of text were based on that particular sentence but it starts making me think about a kind of strange void harkonnen with heart plugs that lead to weird negatively pressurized areas. That's the idea behind the dream well.Very The Age of Wire and String.
I spend all of my time with image and video models and have very thin knowledge when it comes to running, fine tuning, etc. with language models.
How would one start with training an LLM on the entire corpus of one's writings? What model would you use? What scripts and tools?
Has anyone had good results with this?
Do you need to subsequently add system prompts, or does it just write like you out of the box?
How could you make it answer your phone, for instance? Or discord messages? Would that sound natural, or is that too far out of domain?
You could use a vector database.
You could train a model from scratch.
Probably easiest to use OpenAI tools. Upload documents. Make custom model.
How do you make it answer your phone? You could use twillio api + script + llm + voice model. Want natural use a service.
Wouldn't fine tuning produce better results so long as you don't catastrophically forget? You'd preserve more context window space, too, right? Especially if you wanted it to memorize years of facts?
Are LoRAs a thing with LLMs?
Could you train certain layers of the model?
https://docs.unsloth.ai/get-started/fine-tuning-llms-guide
[0]: https://nanogenmo.github.io/
The only thing I'm a bit wary of is the submission size - a minimum of 50,000 words. At that length, It'd be really difficult to maintain a cohesive story without manual oversight.
It was pretty fun!
[1] https://youtu.be/rMmXdiUGsr4
[0] https://archive.org/details/Babble_1020, https://vetusware.com/download/Babble%21%202.0/?id=11924
[1] https://en.wikipedia.org/wiki/Cut-up_technique
"Do me a favor, boy. This scam of yours, when it's over, you erase this god-damned thing."
Trained on TFA I got this:
> I Fed by the used few 200,000 words. All comments were executabove. This value large portive comment then onstring takended to enciece of base for the see marked fewer words in the...
Then I bumped up the order to 2
> I Fed 24 Years of My Blog Posts to a Markov Model > By Susam Pal on 13 Dec 2025 > > Yesterday I shared a little program calle...
It just reproduced the entire article verbatim. This makes sense as BPE removes any pair of repeated tokens, making order-2 Markov transitions fully deterministic.
I've heard that in NLP applications, it's very common to run BPE only up to a certain number of different tokens, so I tried that out next.
Before limiting, BPE was generating 894 tokens. Even adding a slight limit (800) stops it from being deterministic.
> I Fed 24 years of My Blog Postly coherent. We need to be careful about not increasing the order too much. In fact, if we increase the order of the model to 5, the generated text becomes very dry and factual
It's hard to judge how coherent the text is vs the author's trigram approach because the text I'm using to initialize my model has incoherent phrases in it anyways.
Anyways, Markov models are a lot of fun!
I'm considering just deleting all tokens that have only one possible descendant, from the db. I think that would solve that problem. Could increase that threshold to, e.g. a token needs to have at least 3 possible outputs.
However that's too heavy handed: there's a lot of phrases or grammatical structures that would get deleted by that. What I'm actually trying to avoid is long chains where there's only one next token. I haven't figured out how to solve that though.
npm package of the markov model if you just want to play with it on localhost/somewhere else: https://github.com/Aperocky/weighted-markov-generator
Iirc there was some research on "infini-gram", that is a very large ngram model, that allegedly got performance close to LLMs in some domains a couple years back
It achieved state-of-the-art performance at tasks like spelling correction at the time. However, unlike an LLM, it can't generalize at all; if an n-gram isn't in the training corpus it has no idea how to handle it.
https://research.google/blog/all-our-n-gram-are-belong-to-yo...
Ngrams are surprisingly powerful for how little computation they require. They can be trained in seconds even with tons of data.
Except instead we fine-tuned GPT-2 instead. (As was the fashion at the time!)
We used this one, I think https://github.com/minimaxir/gpt-2-simple
I think it took 2-3 hours on my friend's Nvidia something.
The result was absolutely hilarious. It was halfway between a markov chain and what you'd expect from a very small LLM these days. Completely absurd nonsense, yet eerily coherent.
Also, it picked up enough of our personality and speech patterns to shine a very low resolution mirror on our souls...
###
Andy: So here's how you get a girlfriend:
1. Start making silly faces
2. Hold out your hand for guys to swipe
3. Walk past them
4. Ask them if they can take their shirt off
5. Get them to take their shirt off
6. Keep walking until they drop their shirt
Andy: Can I state explicitly this is the optimal strategy
Here’s a link: https://botnik.org/content/harry-potter.html
It is hollow text. It has no properties of what I'd want to get out of even the worst book produced by human minds.
That said, it’s obviously not to everyone’s tastes!
I recommend you to read Bengio et al.’s 2003 paper which describes this issue in more detail and introduces distributional representations (embeddings) in an RNN to avoid this sparsity.
While we are using transformers and sentence pieces now, this paper aptly describes the motivation underpinning modern models.
https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
Distributional representations, not distributed.
https://en.wikipedia.org/wiki/Distributional_semantics#Distr...
Of course, that's because it is a probability along a single dimension with a chain-length along that one dimension while LLMs and NNs use multiple dimensions (They are meshed, not chained).
I really want to know what the result would look like with a few more dimensions resulting in a markov mesh type structure rather than a chain structure.
Markov Models have state and emit tokens based on its current state and undergoes a state transition. A statistical/probabilistic analogue of a state machine.
For a Markov Model to be non-vacuous, non-vapid discussion point, however, one needs to specify very precisely the relationships allowed between state and tokens/observations, whether it's hidden or visible, discrete or continuous ...
The simplest such model is that the state is a specified, computable function of the last k observations. One such function is the identity function.
One can make the state a specified, computable function of k previous states and k most recent tokens/observations.
The functions may be specified only upto a class of computable functions, finite or infinite in size.
You can make the context length a computable function of the k most recent observations (therefore of varying length), but you have to ensure that the contexts are always full for this model to be well defined. Context length can be a computable function of current state and k most recent observations.
On and on.
The most basic/naive one is where one can estimate the unknown parameters of the model given example token streams generated by the model.
Identifiability means that out of all possible models, you can learn the correct one given enough samples.causal identifiability has some other connotations
See here https://causalai.net/r80.pdf as a good start (a nose in a causal graph is Markov given its parents, and a k-step Markov chain is a k-layer causal dag)
It depends on whether the state is visible in the observations or not. Hidden or not is an orthogonal axis of variation compared to the other variations mentioned in the comment.
In a non-hidden model there is no ambiguity or uncertainty about what the current state is.
https://archive.org/details/Babble_1020
https://forum.winworldpc.com/discussion/12953/software-spotl...
Usage:
Where "corpus.txt" should be a file with one sentence per line. Easy to do under sed/awk/perl. This spawns the chatbot with your trained brain.Giving 24 years of your experience, thoughts and life time to us.
This is special in these times of wondering, baiting and consuming only.
https://vale.rocks/micros/20251214-0503
3 more comments available on Hacker News