You Can't Parse XML with Regex. Let's Do It Anyways

Waste of time. Have some "AI" write it for you

But, more than most tools, it is important to learn what regular expressions are and are not for. They are for scanning and extracting text. They are not for parsing complex formats. If you need to actually parse complex text, you need a parser in your toolchain.

This doesn't necessarily require the hair pulling that the article indicates. Python's BeautifulSoup library does a great job of allowing you convenience and real parsing.

Also, if you write a complicated regular expression, I suggest looking for the /x modifier. You will have to do different things to get that. But it allows you to put comments inside of your regular expression. Which turns it from a cryptic code that makes your maintenance programmer scared, to something that is easy to understand. Plus if the expression is complicated enough, you might be that maintenance programmer! (Try writing a tokenizer as a regular expression. Internal comments pay off quickly!)

Who cares that some people are afraid to learn powerful tools. It's their loss. In the time of need, the greybeard is summoned to save the day.

https://xkcd.com/208/

> HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts.

With regex, I won’t. I rarely include much in terms of regex in my PRs, usually small filters for text inputs for example. More complex regexes are saved for my own use to either parse out oddly formatted data, or as vim find/replace commands (or both!).

When I do use a complex regex, I document it thoroughly - not only for those unfamiliar, but also for my future self so I have a head-start when I come back to it. Usually when I get called out on it in a PR, it’s one of two things:

- “Does this _need_ to be a regex?” I’m fine to justify it, and it’s a question I ask teammates especially if it’s a sufficiently complex expression I see - “What’s that do?” This is rarely coming from an “I don’t know regex” situation, and more from an “I’m unfamiliar with this specific part of regex” eg back references.

I think the former is 100% valid - it’s easy to use too much regex, or to use it where there are better methods that may not have been the first place one goes: need to ensure a text field always displays numbers? Use a type=number input; need to ensure a phone number is a valid NANP number? Regex, baby!

The latter is of course valid too, and I try to approach any question about why a regex was used, or what it does, with a link to a regex web interface and an explanation of my thinking. I’ve had coworkers occasionally start using more regex in daily tasks as a result, and that’s great! It can really speed up things tasks that would otherwise be crummy to do by hand or when finagling with a parser.

Bonus: some of my favorite regex adventures:

- Parsing out a heavily customizable theme’s ACF data stuffed into custom fields in a Wordpress database, only to shove them into a new database with a new and %better% ACF structure - Taking PDF bank statements in Gmail, copying the text, and using a handful of painstakingly written find/replace vim regexes to parse the goop into a CSV format because why would banks ever provide structured data?? - Copy/pasting all of the Music League votes and winners from a like 20-person season into a text doc and converting it to a JSON format via regex that I could use to create a visualization of stats. - Not parsing HTML (again, anyways)

The less like 'random' XML the document is the better the extraction will work. As soon as something oddball gets tossed in that drifts from the expected pattern things will break.

But in general we aren’t trying to parse arbitrary documents, we are trying to parse a document with a somewhat-known schema. In this sense, we can parse them so long as the input matches the schema we implicitly assumed.

Can regular expressions parse the subset of XML that I need to pull something out of a document: Maybe.

We have enough library "ergonomics" now that it's not any more difficult to use a regex vs a full XML parser now in dynlangs. Back when this wasn't the case, it really did mean the differnce between a one or two line solution, and about 300 lines of SAX boiler-pate.

With XML you dream of self-documenting structure but wake up to SVG arc commands.

Two positional flags. Two!

Truncation "shouldn't" be common, because chunking is very common for mainstream web servers (and clients of course). And TLS is supposed to explicit protect against this regardless of HTTP.

OTOH, especially behind proxies there are a lot of very minimal HTTP implementations. And, for one reason or another, it is fairly common to visibly see truncation for images and occasionally for HTML too.