Lessons From Building an AI Data Analyst

I would also add that context and tools are the product. This is super important to correctly tune the tools and details matter (ok you can also argue that tools are context in some way)

> The "Short Story" section definitely matches my experience at most companies, startups and bigger non-tech companies alike: They already have more data than they're aware of and know what to do with, and understanding what they have is the starting point before most analysis can be done.

exactly, most of them are concerned about the data they don't have, while in practice they do have a lot to generate good insights.

> Glad I read the post as I hadn't heard of Malloy before. Excuse me if I missed the answer to this, but: How much do you as Findly/Conversion Pattern implement the Semantic Layer on behalf of your users (and if so, I assume you have some process for auto-generating the Malloy models), or do your users have to do something to input the semantics themselves?

We do have an automatic semantic layer generation framework which works as a great starting point, but for the generic case you still have to manually edit / improve it based on the customer's internal context. User's can edit themselves in our UI too, but it usually requires some level of help from us.

We do have a vertical product for commodity trading and shipping: https://www.darlinganalytics.ai/ -> in that case the semantic layer is much more well defined, which makes setup way easier.

One big problem now is that LLMs are not great at writing Malloy, so it is important to have a intermediate DSL. In the future as the language models evolve or someone creates a fine-tuned model that can write Malloy well, we will be able to have more autonomous agents.

Agreed.

Data curation is barely on the radar of most non-tech industries. Even in tech, it's rare to have any meta data.

This is a huge blocker to many efforts.

Someone somewhere has to go through every table and field and document where it came from, when, and what it actually means.

Very very few places do this.

"Oh yeah I go to gold.inventory. I think it updates every night. Columns? Should be pretty intuitive, just look at the names."