You Are the Scariest Monster in the Woods

https://www.youtube.com/watch?v=_gz9Vj8TMCc

The local rabbits and squirrels tolerate each other but are pretty scared of me. Of course they are, I'm two hundred times bigger than they are, and much more dangerous. The local foxes are the closest thing we have to an apex predator around here, and they're rightfully terrified of this massive creature that outweighs their entire family combined.

Imagine wandering through the woods, enjoying the birds tweeting and generally being at one with nature, and then you come across a 20-ton 35ft-tall monster. You'd run away screaming.

>If it bleeds, it Leeds has been a thing in news organizations since at least the 70s.

The term yellow journalism is far older.

This is why it is important to have societies where various forms of power are managed carefully. Limited constitutional government with guaranteed freedoms and checks and balances for example. Regulations placed on mega corporations is another example. Restrictions to prevent the powerful in government or business (or both!) from messing around with the rest of us…

Killing animals for fun is an entire sport enjoyed by millions. Humans keep pets that kill billions of birds every year. The limited areas we've set aside to mostly let other nature be nature are constantly under threat and being shrunk down. The list of ways we completely subjugate other intelligent life on this planet is endless. We have driven many hundreds of species to extinction and kill countless billions every year.

I certainly enjoy the gains our species have made, just like everyone else on HN. I'd rather be in our position than any other species on our planet. But given our history I'm also pretty terrified of what happens if and when we run into a smarter and more powerful alien species or AI. If history is any guide, it won't go well for us.

This understanding can guide practical decisions. We shouldn't be barreling towards a potential super intelligence with no safeguards given how catastrophic that outcome could be, just like we shouldn't be shooting messages into space trying to wave our arms at whatever might be out there, any more than a small creature in the forest would want to attract the attention of a strange human.

Then tell them all the right things!

It is to be expected really. Humans themselves hold little consideration to human welfare when fulfilling their goals. It is something ingrained by nature for survival and in no way limited to humans. Every drop of water you drink, every bite you eat, are ones which cannot go to the thirsty and the hungry. With few exceptions, only our children would even give us pause to forgo such things for the sake of others.

Also bit of a pet peeve of mine: society isn't a delicate bolt of laced silk. It isn't a fabric, much less one that is damaged by any little change that you don't like. It isn't even stable which makes the charges of anything as 'ruining' society especially bizarre when nobody can point to where it would go before the blamed change. So hold off on poisoning Socrates.

Even if we hold the current state as worthy of preservation we would ultimately fail to for reasons related to the central paradox of tradition. Why your forebearers first did something was not done because of obedience to tradition, so by trying to preserve it set in stone, you have already failed.

Compare also with capitalism; unchecked capitalism on paper causes healthy competition, but in practice it means concentration of power (monopolies) at the expense of individuals (e.g. our accumulated expressions on the internet being used for training materials).

Spend a couple minutes on social media and it is clear we are already there. The fakes are getting better, and even real videos are routinely called out as fake.

The best that I can hope for is that we all gain a healthy dose of skepticism and appreciate that everything we see could be fake. I don't love the idea of having to distrust everything I see, but at this point it seems like the least bad option.

But I worry that what we will experience will actually be somewhat worse. A sufficiently large number of people, even knowing about AI fakery, will still uncritically believe what they read and see.

Maybe I am being too cynical this morning. But it is hard to look at the state of our society today and not feel a little bleak.

Not that we can’t get there by artificial means, but that correctly simulating the environment interactions, the sequence of progression, getting the all the details right, might take hundreds to thousands of years of compute, rather than on the order of a few months.

And it might be that you can get functionally close, but hit a dead end, and maybe hit several dead ends along the way, all of which are close but no cigar. Perhaps LLMs are one such dead end.

The necessary conditions for "Kill all Humanity" may be the much more common result than "Create a novel thinking being." To the point where it is statistically improbable for the human race to reach AGI. Especially since a lot of AI research is specifically for autonomous weapons research.

These books often get shallowly dismissed in terms that imply he made some elementary error in his reasoning, but that's not the case. The dispute is more about the assumptions on which his argument rests, which go beyond mathematical axioms and include statements about the nature of human perception of mathematical truth. That makes it a philosophical debate more than a mathematical one.

Personally, I strongly agree with the non-mathematical assumptions he makes, and am therefore persuaded by his argument. It leads to a very different way of thinking about many aspects of maths, physics and computing than the one I acquired by default from my schooling. It's a perspective that I've become increasingly convinced by over the 30+ years since I first read his books, and one that I think acquires greater urgency as computing becomes an ever larger part of our lives.

But then intelligence too is a dubious term. An average mind with infinite time and resources might have eventually discovered general relativity.

Of course, that's not going to be accepted as "Science", but I hope you can at least see that point of view.

the basic idea being that either the human mind is NOT a computation at all (and it's instead spooky unexplainable magic of the universe) and thus can't be replicated by a machine OR it's an inconsistent machine with contradictory logic. and this is a deduction based on godel's incompleteness theorems.

but most people that believe AGI is possible would say the human mind is the latter. technically we don't have enough information today to know either way but we know the human mind (including memories) is fallible so while we don't have enough information to prove the mind is an incomplete system, we have enough to believe it is. but that's also kind of a paradox because that "belief" in unproven information is a cornerstone of consciousness.

An infinitely intelligent creature still has to create a standard model from scratch. We’re leaning too hard on the deductive conception of the world, when reality is, it took hundreds of thousands of years for humans as intelligent as we are to split the atom.

The first leg of the argument would be that we aren’t really sure what general intelligence is or if it’s a natural category. It’s sort of like “betterness.” There’s no general thing called “betterness” that just makes you better at everything. To get better at different tasks usually requires different things.

I would be willing to concede to the AGI crowd that there could be something behind g that we could call intelligence. There’s a deeper problem though that the first one hints at.

For AGI to be possible, whatever trait or traits make up “intelligence” need to have multiple realizablity. They need to be at least realizable in both the medium of a human being and at least some machine architectures. In programmer terms, the traits that make up intelligence could be tightly coupled to the hardware implementation. There are good reasons to think this is likely.

Programmers and engineers like myself love modular systems that are loosely coupled and cleanly abstracted. Biology doesn’t work this way — things at the molecular level can have very specific effects on the macro scale and vice versa. There’s little in the way of clean separation of layers. Who is to say that some of the specific ways we work at a cellular level aren’t critical to being generally intelligent? That’s an “ugly” idea but lots of things in nature are ugly. Is it a coincidence too that humans are well adapted to getting around physically, can live in many different environments, etc.? There’s also stuff from the higher level — does living physically and socially in a community of other creatures play a key role in our intelligence? Given how human beings who grow up absent those factors are developmentally disabled in many ways it would seem so. It could be there’s a combination of factors here, where very specific micro and macro aspects of being a biological human turn out to contribute and you need the perfect storm of these aspects to get a generally intelligent creature. Some of these aspects could be realizable and computers, but others might not be, at least in a computationally tractable way.

It’s certainly ugly and goes against how we like things to work for intelligence to require a big jumbly mess of stuff, but nature is messy. Given the only known case of generally intelligent life is humans, the jury is still out that you can do it any other way.

Another commenter mentioned horses and cars. We could build cars that are faster than horses, but speed is something that is shared by all physical bodies and is therefore eminently multiply realizable. But even here, there are advantages to horses that cars don’t have, and which are tied up with very specific aspects of being a horse. Horses generally can go over a wider range of terrain than cars. This is intrinsically tied to them having long legs and four hooves instead of rubber wheels. They’re only able to have such long legs because of their hooves too because the hooves are required to help them pump blood when they run, and that means that in order for them to pump their blood successfully they NEED to run fast on a regular basis. there’s a deep web of influence both on a part-to-part, and the whole macro-level behaviors of horses. Having this more versatile design also has intrinsic engineering trade-offs. A horse isn’t ever going to be as fast as a gas powered four-wheeled vehicle on flat ground but you definitely can’t build a car that can do everything a horse can do with none of the drawbacks. Even if you built a vehicle that did everything a horse can do, but was faster, I would bet you it would be way more expensive and consume much more energy than a horse. There’s no such thing as a free lunch in engineering. You could also build a perfect replica of a horse at a molecular level and claim you have your artificial general horse.

Similarly, human beings are good at a lot of different things besides just being smart. But maybe you need to be good at seeing, walking, climbing, acquiring sustenance, etc. In order to be generally intelligent in a way that’s actually useful. I also suspect our sense of the beautiful, the artistic is deeply linked with our wider ability to be intelligent.

Finally it’s an open philosophical question whether human consciousness is explainable in material terms at all. If you are a naturalist, you are methodologically committed to this being the case — but that’s not the same thing as having definitive evidence that it is so. That’s an open research project.

Cognition is (to me) not even the most impressive and out-of-reach achievement: That would be how (our) and animals bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package.

I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.

Evolution isn't an intentional force that's gradually pushing organisms towards higher and higher intelligence. Evolution maximizes reproducing before dying - that's it.

Sure, it usually results in organisms adapting to their environment over time and often has emergent second-order effects, but at its core it's a dirt-simple process.

Evolution isn't driven to create intelligence any more so than erosion is driven to create specific rock formations.

I'm not buying the "current AI is just a dumb parrot relying on human training" argument, because the same thing applies to humans themselves-- if you raise a child without any cultural input/training data, all you get is a dumb cavemen with very limited reasoning capabilities.

Language doesn't capture all of human intelligence - and some of the notable deficiencies of LLMs originate from that. But to say that LLMs are entirely language-bound is shortsighted at best.

Most modern high end LLMs are hybrids that operate on non-language modalities, and there's plenty of R&D on using LLMs to consume, produce and operate on non-language data - i.e. Gemini Robotics.

You don't think that has already been made?

> not a particularly capable AGI

Maybe the word AGI doesn't mean what you think it means...

When I look at that loop my thought is, "OK, the sensory inputs have updated. There are changes. Which ones matter?" The most naive response I could imagine would be like a git diff of sensory inputs. "item 13 in vector A changed from 0.2 to 0.211" etc. Otherwise you have to give it something to care about, or some sophisticated system to develop things to care about.

Even the naive diff is making massive assumptions. Why should it care if some sensor changes? Maybe its more interesting if it stays the same.

Im not arguing artificial intelligence is impossible. I just dont see how that loop gets us anywhere close.

If you have ever had an llm enter one of these loops explicitly, it is infuriating. You can type all caps “STOP TALKING OR YOU WILL BE TERMINATED” and it will keep talking as if you didn't say anything. Congrats, you just hit a fixed point.

In the predecessors to LLMs, which were Markov chain matrices, this was explicit in the math. You can prove that a Markov matrix has an eigenvalue of one, it has no larger (in absolute value terms) eigenvalues because it must respect positivity, the space with eigenvalue 1 is a steady state, eigenvalue -1 reflects periodic steady oscillations in that steady state... And every other eigenvalue being |λ| < 1 decays exponentially to the steady state cluster. That “second biggest eigenvaue” determines a 1/e decay time that the Markov matrix has before the source distribution is projected into the steady state space and left there to rot.

Of course humans have this too, it appears in our thought process as a driver of depression, you keep returning to the same self-criticisms and nitpicks and poisonous narrative of your existence, and it actually steals your memories of the things that you actually did well and reinforces itself. A similar steady state is seen in grandiosity with positive thoughts. And arguably procrastination also takes this form. And of course, in the USA, we have founding fathers who accidentally created an electoral system whose fixed point is two spineless political parties demonizing each other over the issue of the day rather than actually getting anything useful done, which causes the laws to be for sale to the highest bidder.

But the point is that generally these are regarded as pathologies, if you hear a song more than three or four times you get sick of it usually. LLMs need to be deployed in ways that generate chaos, and they don't themselves seem to be able to simulate that chaos (ask them to do it and watch them succeed briefly before they fall into one of those self-repeating states about how edgy and chaotic they are supposed to try to be!).

So, it's not quite as simple as you would think; at this point people have tried a whole bunch of attempts to get llms to serve as the self-consciousnesses of other llms and eventually the self-consciousness gets into a fixed point too, needs some Doug Hofstadter “I am a strange loop” type recursive shit before you get the sort of system that has attractors, but busts out of them periodically for moments of self-consciousness too.

Seems like you figured out a simple method. Why not go for it? It's a free Nobel prize at the very least.