Language Support for Marginalia Search

Posted2 months agoActive2 months ago

Bogdanp

178 points

27 comments

marginalia.nuTechstory

calmmixed

Debate

60/100

Search EnginesNatural Language ProcessingInformation Retrieval

Key topics

Search Engines

Natural Language Processing

Information Retrieval

The Marginalia search engine has added language support, and the community discusses its features, limitations, and potential applications, with some users praising its simplicity and others criticizing its AI approach.

Snapshot generated from the HN discussion

Discussion Activity

Moderate engagement

First comment

42m

Peak period

0-3h

Avg / period

2.5

Comment distribution27 data points

Loading chart...

Based on 27 loaded comments

Key moments

01Story posted
Oct 21, 2025 at 2:48 AM EDT
2 months ago
Step 01
02First comment
Oct 21, 2025 at 3:30 AM EDT
42m after posting
Step 02
03Peak activity
8 comments in 0-3h
Hottest window of the conversation
Step 03
04Latest activity
Oct 23, 2025 at 3:02 AM EDT
2 months ago
Step 04

Generating AI Summary...

Analyzing up to 500 comments to identify key contributors and discussion patterns

Discussion (27 comments)

Showing 27 comments

ofalkaed

2 months ago

1 reply

Surprisingly informative for what is pretty much a press release, learned a good deal about search engines.

marginalia_nu

2 months ago

2 replies

(author)

I'm kinda allergic to writing "I did the thing" posts, so I can't help but tryhard and attempt to make them compelling somehow.

Writing in this manner is also very helpful in making sense of the work for myself. Takes a better understanding of the subject to thoroughly explain what you've built than to merely build it. Sometimes I've gone back and read through one of these updates to just get a refresher on what my thinking was when I built something.

LTom

2 months ago

1 reply

A quick question: are you looking for feedback on search results in other languages (as in, what I expect vs. what I get), or is it too early for that?

marginalia_nu

2 months ago

Yeah it's definitely helpful to have those types of reports.

ofalkaed

2 months ago

In my experience, that is pretty much what marginalia search is. I rarely get what I expect but I always get something very interesting that makes me understand my expectations better which is very helpful in accomplishing my goals. Thanks for your work, marginalia is probably my favorite little corner of the web.

reedf1

2 months ago

1 reply

Took me too long to realize this wasn't a tool to search for marginalia in scanned manuscripts.

iamnothere

2 months ago

1 reply

Hey, at least it isn’t named after a very large number, an excited exclamation, or a sound effect. Surely no product with one of those names would ever succeed.

marginalia_nu

2 months ago

1 reply

I probably should have named it cartoon-trombone.wav in retrospect.

reedf1

2 months ago

1 reply

It's a fine name! I had marginalia on the mind - I am reading The Name of the Rose.

iamnothere

2 months ago

1 reply

That makes sense. I am perhaps overly sensitive to the drive by “name haters” who seem to show up in every FOSS or indie project thread.

reedf1

2 months ago

I feel a bit bad it was interpreted that way.

Some fun context, I was trying to find a scanned copy of the first 'correct' book on optics (written by https://en.wikipedia.org/wiki/Ibn_al-Haytham). Possibly the first person to really use the scientific method in circa 1000CE (!!). And I found this (https://cudl.lib.cam.ac.uk/view/MS-PETERHOUSE-00209/103) filled with interesting optical diagrams like something out of my high school physics notebooks. Anyway - I was also thinking about how they might index interesting doodles in the margins. So it was on my mind.

internet_points

2 months ago

1 reply

What tools/data do you use for pos-tagging? I'm guessing it has to be fast, to run without a google data center :)

marginalia_nu

2 months ago

I'm using RDRPosTagger[1], though I've optimized the code a bit so that it's not just algorithmically efficient, but to use the language in a way that is fast. It isn't perfect, but it's good enough to be useful.

Language detection and sentence splitting are the other two slow bits of processing.

[1] https://github.com/datquocnguyen/RDRPOSTagger

mariusor

2 months ago

1 reply

Off topic, but would there be a way to integrate marginalia with a specific website? Similarly to how people use google search for their forums or how HN uses algolia?

I'm asking this as one of my projects is a link aggregator similar to old reddit (and HN to some extent) and I would like to be able to present to users a search box, but without having to implement document indexing and search. (I assume ad principio that the website is already aligned ethically and technologically with what Marginalia stands for :D)

marginalia_nu

2 months ago

1 reply

Should be soon-ish. I'm working right now on laying the ground works for ad-hoc domain filters. That's technically already possible but comes at a too big performance impact that it deteriorates the search results.

When it works, one of the things I have in mind is making a site search-esque functionality available, as well as exposing it via the public API so that it can be whiteboxed.

mariusor

2 months ago

1 reply

Nice. Is there a way to track the work you're doing there (and in general actually)?

marginalia_nu

2 months ago

Best is probably the search-engine tag on my blog[1]. It's the closest you get to release notes for the project.

[1] https://www.marginalia.nu/tags/search-engine/

vintermann

2 months ago

1 reply

This is never going to work. The author is apparently against AI in search in favor of "simplicity", but this sort of thing

> Sentences are stemmed and POS-tagged. Sentences, with stemming and POS-tag data is fed into keyword extraction algorithms

IS AI, it's just old fashioned and bad AI. What he's trying will never work well, for the same reason rule-based machine translation never worked well: there are just too many rules and exceptions. Simplicity is great when you can have it, but with human language, simplicity was never on the table.

He's going to have to bite the bullet and use document embedding models sooner or later.

marginalia_nu

2 months ago

This code is just for helping identify document topics, it literally doesn't need to be perfect. Embedding a billion documents with a server that has no GPU is neither practical nor something that yields good results.

smoghat

2 months ago

3 replies

I’m a little confused by Marginalia. I looked to find out what its purpose was, but couldn’t find it. My bad, I guess, but then again I’m not a search engine. It is pretty cool for a DIY project but the results were really off, especially for searches for individuals. Like take Ezra Klein as an example. Sure there is a link to his show from castbox, a service I have never heard of, and then a bunch of anti Ezra Klein articles. Wikipedia shows up, the last link of the first page is to Abundance. But no NYT? That seems like a big problem. I thought I’d look up Daring Fireball and the only link to his site was a ways down and was to a list of links in 2008. These are just two random searches. I did others, starting with myself, and my results were similar.

Likely I am totally not understanding what this search engine is for. I see this a lot on submissions here. I find something interesting sounding but I don’t understand the context. Maybe it’s just me, but it’s confusing.

FabCH

2 months ago

1 reply

It's a one-man Search engine developed and hosted in the EU.

If you read his about page, it is basically an anti-centralization anti-ad anti-spyware attempt at websearch. It is also "The project is independent in that it has no loans, no investors looking for a payday, no strings attached anywhere to pressure it into doing anything than providing as much and as good internet search as it is capable of."

It not indexing NYT seems precisely on brand.

marginalia_nu

2 months ago

It does index bits of NYT, but coverage is pretty spotty outside of their archives. They put a lot of crawler countermeasures up on their main site (which I guess is fair, they have a business to run), but author biographies are generally accessible, including Ezra's[1].

Though since the search engine doesn't really apply much in terms of domain authority, this doesn't rank very highly, the websites that talk about Ezra Klein rank higher.

[1] https://marginalia-search.com/search?query=site%3Anytimes.co...

iamnothere

2 months ago

It’s for finding results that are less common or more unlikely to appear on other engines, so your results make sense. Why would you need yet another link to an NYT article? That space is crowded. Every engine will find it.

Where it particularly shines is finding highly specific results that get buried in other search engines. Some topics (particularly topics of high commercial interest) have become impossible to research on mainstream search engines. Marginalia will actually find informative articles about these topics rather than page after page of product results and spam.

It may not be useful to you if you’re not a researcher, writer, or someone who often needs to dig deeply into subjects beyond the level of common knowledge.

marginalia_nu

2 months ago

The point of Marginalia Search, as far as there is one, is mostly to complement the bigger search engines by providing tools to find obscure stuff that's drowned out elsewhere, mostly by offering a bunch of filters.

It's not a google replacement, and if you already know what you're looking for then it's probably not the right tool.

Maybe you're looking for mechanical keyboard discussions, then maybe a search for "mechanical keyboard" in the Blogs or Forums filters will provide results you are into.

It's also pretty good at unearthing weird stuff. Say you want to read up on Jack Parsons[3], that Jet Propulsion Lab guy who dabbled in occultism, fell in with Alistair Crowley and then got scammed out of his wealth by L Ron Hubbard, and finally blew himself up, well that is the sort of topic Marginalia Search generally excels at.

[1] https://marginalia-search.com/search?query=mechanical+keyboa...

[2] https://marginalia-search.com/search?query=mechanical+keyboa...

[3] https://marginalia-search.com/search?query=Jack+Parsons&prof...

atombender

2 months ago

1 reply

> Thankfully the BM-25 model used in ranking is robust to this, as it relies on live data from the index itself.

I'm confused by this. TD-IDF incorporates the term frequency (the IDF part), which search engines precompute for the index as a whole. But so does BM25; its IDF formula is slightly different, but also relies on term frequencies. What's the difference?

marginalia_nu

2 months ago

The index has the most up-to-date term frequency information, but it is logistically inacessible, and it's not really practical to interrogate it when extracting keywords (as you need this information for 100 billion terms), so a somewhat stale version is kept in memory instead and used in that process.

When searching, doing BM25, it is a lot more accessible as you already fetch that information indirectly as part of looking up the documents lists, and this is typically only done up to about a dozen times per query.

juliend2

2 months ago

I remember asking you for this, so Thank you so much! It works quite well from what I can see.

Small UI issue: on Desktop, the left sidebar should be scrollable, because now on Firefox I can't reach the "Language" menu item in the search results view, unless I zoom-out.

View full discussion on Hacker News

ID: 45653143Type: storyLast synced: 11/20/2025, 3:50:08 PM

Want the full context?

Jump to the original sources

Read the primary article or dive into the live Hacker News thread when you're ready.

Open link View on HN