Commit Graph

23 Commits

Author SHA1 Message Date
Claire cc94c70970 Clamp dates when serializing to Elasticsearch API (#28081) 2023-12-04 15:28:02 +01:00
Eugen Rochko a9b64b24d6
Change algorithm of `tootctl search deploy` to improve performance (#18463) 2022-05-22 22:16:43 +02:00
Eugen Rochko 679b7158e3
Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Jeong Arm ea38327816
Let votes statuses are also searchable (#18070) 2022-04-23 21:47:27 +02:00
Claire ef196c913c
Fix error MethodError in Chewy::Strategy::Sidekiq::Worker (#17912)
Also refactor a bit to reduce code duplication.
2022-03-31 00:49:24 +02:00
Claire 2c45859ca9
Refactor account and status formatting (#17877)
* Refactor status formatting

* Add account formatting helpers

* Remove StatusFormatter

* Fixup

* Fix copied typo
2022-03-28 01:17:17 +02:00
Eugen Rochko cefa526c6d
Refactor formatter (#17828)
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko d17fb70131
Change how changes to media attachments are stored for edits (#17696)
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko 6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Takeshi Umeda 3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Eugen Rochko 4abe3be321
Change `tootctl search deploy` algorithm (#14300) 2020-07-14 18:10:35 +02:00
Takeshi Umeda e6706d171a
Fix searchable status without explicitly mentioning it (#13829) 2020-05-23 05:47:25 +02:00
Takeshi Umeda 5a1fb6efc2
Fix bookmarks also searchable (#13293) 2020-03-22 02:40:03 +01:00
Eugen Rochko 575dc11cb2
Fix needlessly indexing unsearchable statuses into ElasticSearch (#12041) 2019-10-02 20:04:46 +02:00
Eugen Rochko cc0a55cf9a
Add more accurate hashtag search (#11579)
* Add more accurate hashtag search

Using ElasticSearch to index hashtags with edge n-grams and score
them by usage within the last 7 days since last activity. Only
hashtags that have been reviewed and are listable can appear in
searches, unless they match the query exactly

* Fix search analyzer dropping non-ascii characters
2019-08-18 03:45:51 +02:00
Eugen Rochko 70da6d6630
Fix accounts search by full/partial display name and others (#11580)
- Restrict followers counts to local users to minimize local advantage
- Fix emoji shortcodes causing error in search
- Fix search syntax parse errors not being caught
2019-08-16 13:00:30 +02:00
Eugen Rochko 8fdff2748f
Add more accurate account search (#11537)
* Add more accurate account search

When ElasticSearch is available, a more accurate search is implemented:

- Using edge n-gram index for acct and display name
- Using asciifolding and cjk width normalization on display names
- Using Gaussian decay on account activity for additional scoring (recency)
- Using followers/friends ratio for additional scoring (spamminess)
- Using followers number for additional scoring (size)

The exact match precedence only takes effect when the input conforms
to the username format and the username part of it is complete, i.e.
when the user started typing the domain part.

* Support single-letter usernames

* Fix tests

* Fix not picking up account updates

* Add weights and normalization for scores, skip zero terms queries

* Use local counts for accounts index, adjust search parameters

* Fix mistakes

* Using updated_at of accounts is inadequate for remote accounts
2019-08-16 01:24:03 +02:00
mayaeh a02f4b7cd4 Fix NameError (#11192) 2019-06-27 09:16:55 +02:00
ThibG 47ef4a6c7a Apply filters to poll options (#11174)
* Apply filters to poll options in WebUI

Fixes #11128

* Apply filters to poll options server-side

* Add poll options to searchable text
2019-06-25 14:45:14 +02:00
Eugen Rochko e7f20cc43f
Add type, limit, offset, min_id, max_id, account_id to search API (#10091)
* Add type, limit, offset, min_id, max_id, account_id to search API

Fix #8939

* Make the offset work on accounts and hashtags search as well

* Assure brakeman we are not doing mass assignment here

* Do not allow paginating unless a type is chosen

* Fix search query and index id field on statuses instead of created_at
2019-02-26 15:21:36 +01:00
Ben Lubar 309043b158 Improve image description user experience (#10036)
* Add image descriptions to searchable post content.

* Allow multi-line image descriptions.

* Request image descriptions in the same query as posts when creating the search index.

(see https://github.com/tootsuite/mastodon/pull/10036#discussion_r256551624)
2019-02-14 01:04:43 +01:00
Eugen Rochko 28de046b8b
Get rid of Chewy order/limit warning (#8355) 2018-08-22 02:44:56 +02:00
Eugen Rochko 3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00