Tin: full-text search for Postgres

Tiberium

If anyone's curious - https://planetscale.com/docs/postgres/search/get-started#loc... : They're not providing a local extension with the same performance at the time - it's only offered on their cloud services. The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.

bob1029

I struggle with FTS inside SQL (SQLite and MSSQL). There is often a fairly significant impedance mismatch between the relational concerns and how the documents need to be stored. I've always preferred to use SQL as the system of record and then build/maintain an external Lucene index. Do we think these integral FTS capabilities are at the point where a hybrid architecture doesn't make sense anymore? How much customization exists in this provider?

andrenotgiant

I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world. It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search Timescale has pg_textsearch https://github.com/timescale/pg_textsearch Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se... Now PlanetScale. AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification

alexnewman

What’s funny is I worked with a company with planet in the name Who could really use a full text search that was great in the Postgres

usernametaken29

Interestingly enough SQLites FTS supports Lucene queries out of the box with great performance characteristics. IIRC only writes become pretty slow after a while. I’ve always wondered what exactly would prevent PostgreSQL from strapping that implementation into its own database. My experience with ts_query hasn’t been particularly rosy. It can be better than LIKE but only marginally so and at the cost of insane index sizes… If this extension becomes open source and we can test it out in the real world I’m sure there’s a sweet spot

sick_of_slop

What are the advantages of Tin over using ts_vector with gin and gist indexes?

adityapatadia

It’s just me or there are others who keep seeing these updates and think mongodb had all of this years ago? Seriously so happy to be running our production stack on mongo.

aroman

I want to try Planetscale... but we're addicted to (and totally dependent on) Neon's branching model. They really got us hooked on that!

downsplat

I suppose it all depends on the scale of your project, but I've had pretty good luck using both MySQL's and SQLite's FTS capabilities. Surprised to hear that Open Source champion Postgres didn't have up-to-snuff FTS up to now...?

groundzeros2015

Please read the Postgres manual. It has incredible built-in search capability.

znpy

This was already posted and ignored at https://news.ycombinator.com/item?id=49751888 so i’ll ask the same question: again: I don’t see any github link, is this 21st century embrace, extend, extinguish ?

tannhaeuser

Postgres does have pg_fts (tsvector/tsquery/tsrank) which is a quite sophisticated full text search package integrated with functional indexing and query optimization. Why would I use something vibecoded that isn't part of core Postgres instead?

immmmmm

Please note possible name collision with PostGIS Triangulated Irregular Network (TIN) data type.

dorianmariecom

it does what it says on the TIN

Doohickey-d

There's another aspect which none of the FTS search solutions for Postgres do well in my opinion: multi-language support. For example this one: it doesn't mention support for CJK languages (meaning tokenization for e.g. Chinese will resolve to one token per character, which will technically work and give results, but is inefficient). Also word stemming (databases -> database) is also missing as far as I can see, so the kind of queries where you'd expect related words to show up will be missing. Just doing case-folding and accent-folding is a bit of a functional but bruteforce solution. Ideally I'd want something that supports: - language aware tokenization, with ability to define the language per record. Including stemming, etc. And have useful predefined configuration for common languages (e.g. the Postgres built in one is missing many languages). - CJK support, tokenizing at word boundaries. - Optional accent- and case-folding. Most solutions just seem to assume English content, I have not found anything that does all of this yet.

onesandofgrain

Why is this necessary? we have built in full-text search in postgres?