Commit graph

1065 commits

Author SHA1 Message Date
Matt Jankowski
50c2a03695
Rails 7 update (#24241) 2023-07-02 10:38:53 +02:00
Claire
a209d1e683
Fix ResolveURLService not resolving local URLs for remote content (#25637) 2023-06-29 14:48:54 +02:00
jsgoldstein
4581a528f7
Change account search to match by text when opted-in (#25599)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2023-06-29 13:05:21 +02:00
Eugen Rochko
bb4756c823
Change files to be deleted in batches instead of one-by-one (#23302) 2023-06-26 14:17:41 +02:00
Emelia Smith
3a91603b15
Prevent UserCleanupScheduler from overwhelming streaming (#25519) 2023-06-20 18:04:35 +02:00
Eugen Rochko
4c9406bdb0
Add time zone preference (#25342) 2023-06-10 03:29:37 +02:00
Nick Schonning
c66250abf1
Autofix Rubocop Regex Style rules (#23690)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-06-06 14:50:51 +02:00
Claire
8884d1ece0
Add support for importing lists (#25203) 2023-06-01 14:47:31 +02:00
Christian Schmidt
69057467cb
Translate CW, poll options and media descriptions (#24175)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-06-01 00:10:21 +02:00
S.H
8c183a9831
Align argument type to keyword argument for private process method in FetchResourceService (#25189) 2023-05-30 09:28:01 +02:00
Matt Jankowski
9f5deb310b
Fix Performance/MapCompact cop (#24797)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-05-23 10:49:12 +02:00
Essem
5fd8d1e417
Fix oversight in backup service (#25034) 2023-05-19 11:27:10 +02:00
Matt Jankowski
2c6c398c60
Fix Performance/CollectionLiteralInLoop cop (#24819) 2023-05-04 05:33:55 +02:00
Claire
a2a22bad23
Fix various edge cases with local moves (#24812) 2023-05-03 19:19:25 +02:00
Claire
c98b012583
Change Move handler to also move list memberships (#24808) 2023-05-03 14:03:38 +02:00
Matt Jankowski
668a19a2f3
Fix Performance/DeletePrefix cop (#24796) 2023-05-02 21:07:45 +02:00
Matt Jankowski
f1c1dd0118
Rename with_lock to with_redis_lock to avoid confusion with ActiveRecord's method (#24741) 2023-05-02 18:16:07 +02:00
Claire
a89b02af92
Check domain suspensions before trying to fetch resources (#24535) 2023-05-02 15:22:19 +02:00
Claire
b233da5996
Optimize archive export service and export zip files instead of gzipped tar files (#23360) 2023-05-02 15:09:43 +02:00
Claire
32a030dd74
Rewrite import feature (#21054) 2023-05-02 12:08:48 +02:00
Matt Jankowski
d902a707a3
Fix Rails/CompactBlank cop (#24690) 2023-04-30 14:07:21 +02:00
Nick Schonning
49fad26eca
Drop EOL Ruby 2.7 (#24237) 2023-04-27 01:46:18 +02:00
Matt Jankowski
0a5f0a8b20
Remove instance variables from helper usage (#24203) 2023-04-23 22:35:54 +02:00
Claire
85b1b45820
Fix crash in NotifyService when trying to send an email notification for polls (#24558) 2023-04-17 13:13:36 +02:00
Claire
99e3e152cd
Fix crash in NotifyService when trying to send an email notification for post edits (#24460) 2023-04-08 12:51:14 +02:00
Eugen Rochko
a9b5598c97
Change user settings to be stored in a more optimal way (#23630)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2023-03-30 14:44:00 +02:00
Christian Schmidt
eb38e9df31
Requeue expiration notification (#24311) 2023-03-29 10:52:40 +02:00
Takeshi Umeda
38c84f57b6
Refactoring relations_map (#24195) 2023-03-21 10:32:58 +01:00
Christian Schmidt
bd047acc35
Replace Status#translatable? with language matrix in separate endpoint (#24037) 2023-03-16 11:07:24 +01:00
Claire
1d0ad558ff
Change sidekiq-bulk's batch size from 10,000 to 1,000 jobs in one Redis call (#24034) 2023-03-15 03:45:15 +01:00
Jean byroot Boussier
922837dc96
Upgrade to latest redis-rb 4.x and fix deprecations (#23616)
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
2023-03-04 16:38:28 +01:00
Claire
050f1669c6
Fix original account being unfollowed on migration before the follow request could be sent (#21957) 2023-03-03 21:13:55 +01:00
Christian Schmidt
5a8c651e8f
Only offer translation for supported languages (#23879) 2023-03-03 21:06:31 +01:00
Claire
c2a046ded1
Fix “Remove all followers from the selected domains” being more destructive than it claims (#23805) 2023-03-03 20:25:15 +01:00
Nick Schonning
56489cdc4f
Autofix Rubocop Style/HashConversion (#23852) 2023-02-28 22:59:19 +09:00
Nick Schonning
4ff44be134
Autofix Rubocop Rails/Blank (#23765) 2023-02-22 09:57:56 +09:00
Nick Schonning
0cfdd1a401
Enable Rubocop Style/StringConcatenation defaults (#23792) 2023-02-22 09:54:36 +09:00
Nick Schonning
59c8d43d94
Autofix Rubocop Style/RescueStandardError (#23745) 2023-02-20 11:01:20 +01:00
Nick Schonning
717683d1c3
Autofix Rubocop remaining Layout rules (#23679) 2023-02-20 06:58:28 +01:00
Nick Schonning
2177daeae9
Autofix Rubocop Style/RedundantBegin (#23703) 2023-02-19 07:09:40 +09:00
Nick Schonning
e2a3ebb271
Autofix Rubocop Style/IfUnlessModifier (#23697) 2023-02-18 12:37:47 +01:00
Nick Schonning
634368c491
Autofix Rubocop Lint/SymbolConversion (#23683) 2023-02-18 03:23:49 +01:00
Nick Schonning
669f6d2c0a
Run rubocop formatting except line length (#23632) 2023-02-18 06:56:20 +09:00
Claire
d6930b3847
Add API parameter to safeguard unexpect mentions in new posts (#18350) 2023-02-13 16:36:29 +01:00
Claire
0c9eac80d8
Fix unbounded recursion in post discovery (#23506)
* Add a limit to how many posts can get fetched as a result of a single request

* Add tests

* Always pass `request_id` when processing `Announce` activities

---------

Co-authored-by: nametoolong <nametoolong@users.noreply.github.com>
2023-02-10 22:16:37 +01:00
Nick Schonning
f68bb52556
Apply Rubocop Style/NegatedIfElseCondition (#23451) 2023-02-08 07:07:36 +01:00
Nick Schonning
ed570050c6
Autofix Rails/EagerEvaluationLogMessage (#23429)
* Autofix Rails/EagerEvaluationLogMessage

* Update spec for debug block syntax
2023-02-07 03:44:36 +01:00
Claire
bb89f0af8a
Fix ActivityPub::ProcessingWorker error on incoming malformed JSON-LD (#23416) 2023-02-06 21:00:58 +01:00
Markus Unterwaditzer
f2a6e71bb6
Suppress AddressFamilyError in link verification (#23204)
* Suppress AddressFamilyError

* clarify comment
2023-01-23 13:05:54 +01:00
Claire
628dcbb732
Revert "Remove LDSignature on actor Delete activities (#21466)" (#23185)
This reverts commit f4f2b062ec.
2023-01-21 15:33:21 +01:00
Claire
68dcbcb7bf
Add more specific error messages to HTTP signature verification (#21617)
* Return specific error on failure to parse Date header

* Add error message when preferredUsername is not set

* Change error report to be JSON and include more details

* Change error report to differentiate unknown account and failed refresh

* Add tests
2023-01-18 16:47:56 +01:00
Claire
4b92e59f4f
Add support for editing media description and focus point of already-posted statuses (#20878)
* Add backend support for editing media attachments of existing posts

* Allow editing media attachments of already-posted toots

* Add tests
2023-01-18 16:33:55 +01:00
Markus Unterwaditzer
0c689b9d01
fix: allow verification when page size exceeds 1MB (using HTML5 parser) (#22879)
* fix: allow verification when page size exceeds 1MB
Truncates the page after 1MB instead

Closes #15316

* switch to HTML5 parser, fix rubocop errors

* undo rubocop fixes

Co-authored-by: Chris Zubak-Skees <chriszs@gmail.com>
2023-01-11 21:59:13 +01:00
Kaspar V
ae62e5fa53
Fix/remove calling private method with send in model (#22951)
* fix(status): remove send usage for private unlink_from_conversations

- make unlink_from_conversations public method
- rename unlink_from_conversations to unlink_from_conversations!
- fix send call on private method in statuses_vacuum and batched_remove_status_service

* fix(feeds_vacuum): replace find_in_batches with in_batches

because active record query results should be a little more efficient than
itterating with map and each. Postgres can grasp such lists of ids much quicker
than ruby can.
Will probably make allmost no difference, but cannot hurt either.
2023-01-11 21:57:24 +01:00
Claire
18fb01ef7c
Fix possible race conditions when suspending/unsuspending accounts (#22363)
* Fix possible race conditions when suspending/unsuspending accounts

* Fix tests

Tests were assuming SuspensionWorker and UnsuspensionWorker would do the
suspending/unsuspending themselves, but this has changed.
2023-01-05 13:47:21 +01:00
Effy Elden
4dc9152b3e
Be more lenient with OEmbed detection and validation (#22533) 2023-01-05 13:36:24 +01:00
Claire
c1de673060
Fix suspension worker crashing on S3-compatible setups without ACL support (#22487) 2022-12-19 17:55:17 +01:00
Claire
bf1c7e2122
Ensure exact match is the first result in hashtag searches (#21315)
Fixes #17494
2022-12-15 18:41:40 +01:00
Claire
2644a28cb3
Change remote media files to be downloaded outside of transactions (#21796) 2022-12-15 18:09:48 +01:00
Brian Campbell
2d12948220
Fix idempotency when database writes are slow (#21840)
There is an idempotency key generated by clients when authoring a post,
and stored in Redis, to ensure that if a user or client retries posting
the same status, we don't get a duplicate.

Hachyderm.io has been experiencing some filesystem and database
performance issues, causing database writes to be slow. This can mean
that there are successful posts, but the reverse proxy returns 504
Gateway Timeout before the idempotency status has been updated; users or
clients who retry (such as Tusky which retries automatically, see
tuskyapp/Tusky#2951) can re-try the same post with the same idempotency
key before it has actually been recorded in Redis, leading to duplicate
posts.

To address this issue, move all of the database updates after the
initial transaction that creates the status into the
`postprocess_status!` method, so we can insert the idempotency key
immediately after the status has been created, significantly reducing
the window in which the status could be created but the idempotency key
not yet stored.

Note: this has not yet been tested; I'm submitting this PR for
discussion and to offer to the Hachyderm.io admins to try out to fix the
multiple posting problem.

Co-authored-by: Brian Campbell <brcampbell@beta.team>
2022-12-15 18:08:40 +01:00
Claire
c8849d6cee
Fix unbounded recursion in account discovery (#22025)
* Fix trying to fetch posts from other users when fetching featured posts

* Rate-limit discovery of new subdomains

* Put a limit on recursively discovering new accounts
2022-12-07 00:15:24 +01:00
Claire
f4f2b062ec
Remove LDSignature on actor Delete activities (#21466)
They are currently not used for anything and represent more than half of the
payload size.
2022-11-27 20:43:24 +01:00
Claire
625216d8e1
Fix attachments of edited statuses not being fetched (#21565)
* Fix attachments of edited statuses not being fetched

* Fix tests
2022-11-27 20:39:05 +01:00
Joshua Wood
daf6f3453e
Handle links with no href in VerifyLinkService (#20741)
Before this change, the following error would cause VerifyAccountLinksWorker to fail:

NoMethodError: undefined method `downcase' for nil:NilClass
  [PROJECT_ROOT]/app/services/verify_link_service.rb:31 :in `block in link_back_present?`
2022-11-17 10:59:35 +01:00
Hampton Lintorn-Catlin
147d8bd8fc
Support UTF-8 Characters in Domains During CSV Import (#20592)
* Support UTF-8 Characters in Domains During Import

* Update Changelong
2022-11-14 05:52:13 +01:00
James Tucker
78a6b871fe
Improve performance by avoiding regex construction (#20215)
```ruby
10.times { p /#{FOO}/.object_id }
10.times { p FOO_RE.object_id }
```
2022-11-10 05:49:30 +01:00
Claire
a5394980f2
Fix NameError in Webfinger redirect handling in ActivityPub::FetchRemoteActorService (#20260) 2022-11-09 20:10:38 +01:00
Eugen Rochko
e98833748e
Fix being able to spoof link verification (#20217)
- Change verification to happen in `default` queue
- Change verification worker to only be queued if there's something to do
- Add `link` tags from metadata fields to page header of profiles
2022-11-09 08:24:21 +01:00
trwnh
b1a48e05b6
Change Report category to "violation" if rule IDs are provided (#20137)
* Change Report category to "violation" if rule IDs are provided

* Fix LiteralAsCondition

* Add parentheses to conditional statement
2022-11-08 17:28:02 +01:00
Claire
bbf74498f5
Fix validation error in SynchronizeFeaturedTagsCollectionWorker (#20018)
* Fix followers count not being updated when migrating follows

Fixes #19900

* Fix validation error in SynchronizeFeaturedTagsCollectionWorker

Also saves remote user's chosen case for hashtags

* Limit remote featured tags before validation
2022-11-07 22:35:53 +01:00
Eugen Rochko
c4b92b1aee
Fix n+1 query during status removal (#19753) 2022-11-05 00:09:52 +01:00
Claire
c2170991c7
Fix reblogs being discarded after the reblogged status (#19731) 2022-11-04 16:31:44 +01:00
Eugen Rochko
5f9e47be34
Add caching for payload serialization during fan-out (#19642) 2022-11-04 13:21:06 +01:00
Claire
4fb0aae636
Change mentions of blocked users to not be processed (#19725)
Fixes #19698
2022-11-04 13:19:12 +01:00
Claire
e0eb39d41b
Fix bookmark import stopping at the first failure (#19669)
Fixes #19389
2022-11-02 16:38:23 +01:00
Eugen Rochko
dc5c86add7
Fix account migration form ever using outdated account data (#18429) 2022-10-29 01:31:45 +02:00
Eugen Rochko
f6bcf86caf
Fix wrong math function used in search query (#19481) 2022-10-27 02:10:38 +02:00
Eugen Rochko
7d25f72b9f
Fix negatives values in search index causing queries to fail (#19464) 2022-10-26 13:00:43 +02:00
Eugen Rochko
1ae508bf2f
Change unauthenticated search to not support pagination in REST API (#19326)
- Only exact search matches for queries with < 5 characters
- Do not support queries with `offset` (pagination)
- Return HTTP 401 on truthy `resolve` instead of overriding to false
2022-10-26 12:10:02 +02:00
Yamagishi Kazutoshi
45d3b32488
Fix Settings::FeaturedTagsController (#19418)
Regression from #19409
2022-10-22 23:14:58 +02:00
Takeshi Umeda
74ead7d106
Change featured tag updates to add/remove activity (#19409)
* Change featured tag updates to add/remove activity

* Fix to check for the existence of feature tag

* Rename service and worker

* Merge AddHashtagSerializer with AddSerializer

* Undo removal of sidekiq_options
2022-10-22 18:30:55 +02:00
Yamagishi Kazutoshi
94feb2b93f
Fix FetchFeaturedCollectionService spec (#19401)
Regression from #19380
2022-10-21 11:48:22 +02:00
Takeshi Umeda
b0e3f0312c
Add synchronization of remote featured tags (#19380)
* Add LIMIT of featured tag to instance API response

* Add featured_tags_collection_url to Account

* Add synchronization of remote featured tags

* Deliver update activity when updating featured tag

* Remove featured_tags_collection_url

* Revert "Add featured_tags_collection_url to Account"

This reverts commit cff349fc27b104ded2df6bb5665132dc24dab09c.

* Add hashtag sync from featured collections

* Fix tag name normalize

* Add target option to fetch featured collection

* Refactor fetch_featured_tags_collection_service

* Add LIMIT of featured tag to v1/instance API response
2022-10-20 09:15:52 +02:00
Claire
cedcece0cc
Fix deleted pinned posts potentially counting towards the pinned posts limit (#19005)
Fixes #18938
2022-10-05 00:16:40 +02:00
Eugen Rochko
55a2e9b5be
Fix translations not being formatted, other issues in web UI (#19245)
Fix #19237
2022-09-28 01:02:01 +02:00
Claire
85890bc80f
Fix crash in FetchRemoteKeyService (#19225)
Fix regression from #19212
2022-09-24 07:41:01 +02:00
Eugen Rochko
0d6b878808
Add user content translations with configurable backends (#19218) 2022-09-23 23:00:12 +02:00
Claire
8cf7006d4e
Refactor ActivityPub handling to prepare for non-Account actors (#19212)
* Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

ActivityPub::FetchRemoteAccountService is kept as a wrapper for when the actor is
specifically required to be an Account

* Refactor SignatureVerification to allow non-Account actors

* fixup! Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

* Refactor ActivityPub::FetchRemoteKeyService to potentially return non-Account actors

* Refactor inbound ActivityPub payload processing to accept non-Account actors

* Refactor inbound ActivityPub processing to accept activities relayed through non-Account

* Refactor how Account key URIs are built

* Refactor Request and drop unused key_id_format parameter

* Rename ActivityPub::Dereferencer `signature_account` to `signature_actor`
2022-09-21 22:45:57 +02:00
Eugen Rochko
50948b46aa
Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
Claire
7b38cb88ca
Fix ProcessMentionService swallowing unprocessed mentions to unconfirmed/unapproved users (#19191) 2022-09-20 23:49:00 +02:00
Claire
1145dbd327
Improve error reporting and logging when processing remote accounts (#15605)
* Add a more descriptive PrivateNetworkAddressError exception class

* Remove unnecessary exception class to rescue clause

* Remove unnecessary include to JsonLdHelper

* Give more neutral error message when too many webfinger redirects

* Remove unnecessary guard condition

* Rework how “ActivityPub::FetchRemoteAccountService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteAccountService#call (default/previous behavior).

* Rework how “ActivityPub::FetchRemoteKeyService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteKeyService#call (default/previous behavior).

* Fix Webfinger::RedirectError not being a subclass of Webfinger::Error

* Add suppress_errors option to ResolveAccountService

Defaults to true (to preserve previous behavior). If set to false,
errors will be raised instead of caught, allowing the caller to be
informed of what went wrong.

* Return more precise error when failing to fetch account signing AP payloads

* Add tests

* Fixes

* Refactor error handling a bit

* Fix various issues

* Add specific error when provided Digest is not 256 bits of base64-encoded data

* Please CodeClimate

* Improve webfinger error reporting
2022-09-20 23:30:26 +02:00
Claire
c7147bab90
Fix incorrect and slow cache invalidation in ClearDomainMediaService (#19062)
Fixes #19060
2022-08-27 00:45:54 +02:00
Eugen Rochko
d83faa1a89
Add ability to block sign-ups from IP (#19037) 2022-08-24 19:00:37 +02:00
Jeong Arm
6aa83b13ba
Properly delete remote account's avatar/header when fetch/update (#18973) 2022-08-15 20:32:21 +02:00
Eugen Rochko
c3f0621a59
Add ability to follow hashtags (#18809) 2022-07-17 13:49:29 +02:00
Eugen Rochko
44b2ee3485
Add customizable user roles (#18641)
* Add customizable user roles

* Various fixes and improvements

* Add migration for old settings and fix tootctl role management
2022-07-05 02:41:40 +02:00
Claire
1b4054256f
Fix crash when a remote Flag activity mentions a private post (#18760)
* Add tests

* Fix crash when a remote Flag activity mentions a private post
2022-07-04 11:08:30 +02:00
Eugen Rochko
2936f42a14
Add notifications for new reports (#18697) 2022-06-27 09:30:15 +02:00
Eugen Rochko
a2871cd747
Add administrative webhooks (#18510)
* Add administrative webhooks

* Fix error when webhook is deleted before delivery worker runs
2022-06-09 21:57:36 +02:00
Eugen Rochko
52f4e834f2
Fix concurrent unfollowing decrementing follower count more than once (#18527) 2022-05-26 22:14:47 +02:00