Re: Combine Top-k with similarity search extensions

Shmagi,

Take the first 20  text characters and compute and store the CRC32 or MD5  of that value.  That value acts as a signature. You can then find all distinct signatures,  or all rows with duplicate signatures for further analysis  You could event try building a signature on the full text string.

---

From: "Shmagi Kavtaradze" <kavtaradze.s@gmail.com>
To: pgsql-novice@postgresql.org
Sent: Friday, November 20, 2015 2:21:36 AM
Subject: [NOVICE] Combine Top-k with similarity search extensions

I am performing similarity check over a column in a table with about 3500 entries. Column is populated with text data from text file. Performing a check results in 3500 * 3500 rows and it takes forever to calculate for my virtual machine. Is there any way to calculate for top-k results, to decrease amount and time needed? What I mean is that, for example when checking two sentences, if first several words does not match, to stop checking that sentences and move on.