* Add --include-subdomains option to tootctl domains purge
* Add support for '*.' subdomain wildcard patterns in `tootctl domains purge`
* Fix custom emojis deletion not following subdomain and URI options
* Change `tootctl domains purge` to not purge domain blocks unless --purge-domain-blocks is passed
* Refactor `tootctl domains purge`
* Add feedback on deleted domain blocks
* Bump version to 3.5.2
* Change some entries to be more clear
* Add some extra notes
* Fix line wrap
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Update fix-duplicates task to 2022_02_10_153119
Also add support for Appeal to AccountMerging#merge_with!
* Update fix-duplicates task to 2022_03_07_094650
* Update fix-duplicates task to 2022_03_09_213005
* Update fix-duplicates task to 2022_03_07_083603
* Update fix-duplicates task to 2022_03_10_060626
* Update fix-duplicates script to 2022_03_07_083603
* Update fix-duplicates task to 2022_03_10_060706
* Update fix-duplicates task to 2022_03_10_060959
* Silence CodeClimate
* Fix null values being included in some indexes
* Update lib/mastodon/migration_helpers.rb
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add documentation link to corruption error message
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Improve Emoji import
Skip macOS '._' shadow files in tar archive to speed up import
* Fix codeclimate format issue with whitespace
* Update lib/mastodon/emoji_cli.rb
suggestions from Gargron to improve comment
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Update emoji_cli.rb
Remove extraneous comment (macOS-specific comment now with correct line)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Fix maintenance script not re-indexing some indexes on textual values
Fixes#15475
* Refresh instance view at the end of the maintenance script run
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add admin option to remove canonical email blocks from a deleted account
* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
* Add remove orphans to tootctl statuses remove
* Add REINDEX and change option from vacuum to compression-database
* Changed to extract the deletion target of conversations to a temporary table
* Support progress bar and exceptions when media remove
* Add continue option
* Fix compression to compress
* Remove skip_remove_orphans
* Add trending links
* Add overriding specific links trendability
* Add link type to preview cards and only trend articles
Change trends review notifications from being sent every 5 minutes to being sent every 2 hours
Change threshold from 5 unique accounts to 15 unique accounts
* Fix tests
* Add account statuses cleanup policy model
* Record last inspected toot to delete to speed up successive calls to statuses_to_delete
* Add service to cleanup a given account's statuses within a budget
* Add worker to go through account policies and delete old toots
* Fix last inspected status id logic
All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.
* Add tests
* Refactor scheduler and add tests
* Add user interface
* Add support for discriminating based on boosts/favs
* Add UI support for min_reblogs and min_favs, rework UI
* Address first round of review comments
* Replace Snowflake#id_at_start with with_random parameter
* Add tests
* Add tests for StatusesCleanupController
* Rework settings page
* Adjust load-avoiding mechanisms
* Please CodeClimate
* Change references to tootsuite/mastodon to mastodon/mastodon
* Remove obsolete test fixture
* Replace occurrences of tootsuite/mastodon with mastodon/mastodon in CHANGELOG
And a few other places
* Fix account deletion sometimes failing because of optimistic locks
In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.
Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.
To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.
Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.
[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602
* Bump MAX_SUPPORTED_VERSION in maintenance script
Mastodon::MigrationHelpers has been forked from Gitlab a long time ago, but
Mastodon has never supported using a MySQL database.
Removing MySQL support from Mastodon::MigrationHelpers makes it a little easier
to maintain. In particular, it removes code that would need updating with
Rails 6.
* Use ActiveRecord::Result#to_ary instead of deprecated to_hash
They do the same thing, and to_hash has been removed from Rails 6.1
* Explicitly name polymorphic indexes to workaround a bug in Rails 6.1
cf. https://github.com/rails/rails/issues/41693
* Fix incorrect usage of “foreign_key” in migration script
* Use `ActiveModel::Errors#delete` instead of deprecated clear method
* Fix link headers tests on Rails 6.1
Rails 6.1 adds values to the Link header by default, thus it is not a
LinkHeader object anymore. Fix the test to parse the Link header instead
of assuming it is a LinkHeader.
* Fix misuse of foreign_type
* Fix use of removed "add_template_helper"
* Use response.media_type instead of response.content_type in tests
* Fix CSV export controller test on Rails 6
Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.
This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.
* Fix emoji formatting with Rails 6
* Make emoji output more idiomatic and robust
* Switch from redis-rails gem to built-in Rails redis cache storage
* Added .deepsource.toml
* Removed bad use of `alias`
* Fixed operand order in the binary expression
* Prefixed unused method arguments with an underscore
* Replaced the old OpenSSL algorithmic constants with the newer strings initializers.
* Removed unnecessary UTF-8 encoding comment
* Fix maintenance script not re-indexing some indexes on textual values
Fixes#15475
* Refresh instance view at the end of the maintenance script run
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Improve Emoji import
Skip macOS '._' shadow files in tar archive to speed up import
* Fix codeclimate format issue with whitespace
* Update lib/mastodon/emoji_cli.rb
suggestions from Gargron to improve comment
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Update emoji_cli.rb
Remove extraneous comment (macOS-specific comment now with correct line)
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Fix being able to import more than allowed number of follows
Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.
This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).
* Catch validation errors to not re-queue failed follows
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix external user creation failing when invite request text is required
Also fixes tootctl-based user creation.
* Add test about invites when invite request text is otherwise required
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix 2FA/sign-in token sessions being valid after password change (#14802)
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
* Fix Move handler not being triggered when failing to fetch target (#15107)
When failing to fetch the target account, the ProcessingWorker fails
as expected, but since it hasn't cleared the `move_in_progress` flag,
the next attempt at processing skips the `Move` activity altogether.
This commit changes it to clear the flag when encountering any
unexpected error on fetching the target account. This is likely to
occur because, of, e.g., a timeout, when many instances query the
same actor at the same time.
* Fix slow distinct queries where grouped queries are faster (#15287)
About 2x speed-up on inboxes query
* Fix possible inconsistencies in tag search (#14906)
Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
downcasing before query but not before insertion may mean that some
tags with some casings are not searchable
* Fix updating account counters when account_stat is not yet created (#15108)
* Fix account processing failing because of large collections (#15027)
Fixes#15025
* Fix downloading remote media files when server returns empty filename (#14867)
Fixes#14817
* Fix webfinger redirect handling in ResolveAccountService (#15187)
* Fix webfinger redirect handling in ResolveAccountService
ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.
Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.
* Add tests
* Remove dependency on unused and unmaintained http_parser.rb gem (#14574)
It seems that years ago, the “http” gem dependend on the “http_parser.rb” gem
(it now depends on the “http-parser” gem), and, still years ago, we pulled
it from git in order to benefit from a bugfix that wasn't released yet (#7467).
* Add tootctl maintenance fix-duplicates (#14860, #15201, #15264, #15349, #15359)
* Fix old migration script not being able to run if it fails midway (#15361)
* Fix old migration script not being able to run if it fails midway
Improve the robustness of a migration script likely to fail because of database
corruption so it can run again once database corruptions are fixed.
* Display a specific error message in case of index corruption
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix ResolveAccountService accepting mismatching acct: URI
* Set attributes that should be updated regardless of suspension
* Fix key fetching
* Automatically merge remote accounts with duplicate `uri`
* Add tests
* Add "tootctl accounts fix-duplicates"
Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add tootctl maintenance fix-duplicates
This tool goes through the database to detect and fix duplicates.
This operation is very slow and may cause data loss (of data that would be
inaccessible without intervention because of the existing index corruptions).
It tries its best to make sensible decisions, and asks the user in some cases.
* Add warning message in db:migrate hook
* Clear Rails cache after being done with database deduplication
Avoids followers hash cache being incorrect, among other things
* Increase DNS timeout from 1 second to 5 seconds for MX check
1 seconds is rather short when using a recursive DNS resolver which
hasn't got a cached result already available. Use 5 seconds instead,
which is the timeout value we use for outgoing HTTP queries.
* Add more precise error messages for invalid e-mail addresses
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
* Add tootctl email_domains (block|unblock)
* fix codeclimate issues.
* fix codeclimate issues.
* fix codeclimate issues.
* add list subcommand, remove log_action.
* fix codeclimate issues.
* filter duplicate hostnames,ips before block
* rebase from currnet master branch.
rename email_domains_cli.rb to email_domain_blocks_cli.rb .
rename Mastodon::EmailDomainsCLI to Mastodon::EmailDomainBlocksCLI .
rename command email_domains to email-domain-blocks . (Thor recognizes both of - and _ )
rename subcommand block to add .
rename subcommand unblock to remove .
change the color in list subcommand to while for domain or cyan for childlen.
don't use include() in list subcommand.
suppress console output about succeeded entry.
add console output about count of processed/skipped.
remove capitalization in subcommand description.
remove long_desc in subcommand 'remove'.
remove duplicate where in subcommand 'remove'.
* fix codeclimate issue.
- Fix not moving original files of custom emojis
- Fix command failing to move any files with S3 storage
- Fix command marking records as upgraded when move failed
Fix#13594
* add emoji export command to cli
* fix codeclimate issues
* add error when no matching category was found
* add other suggestions
* exit 1 when no matching category is found
* changes according to suggestions
* 👀
* RubyNein
Y u always autoformat :c
* Fix comments in statuses_cli
The comment for the part of query keeping statuses which have replies was
incorrect, explaining the part of the query kept favourited statuses instead
* Keep statuses favourited by local users in tootctl statuses remove
* Do delete old replies that are not direct interactions with local users
* Skip reblogs of local statuses instead of all reblogs
* Optimize some queries
* Keep old statuses with recent reblogs
* Add option to clear toots from followed accounts too
* Add "--exclude-suspended" to tootctl domains crawl
This new option ignores any instances suspended server-wide as
well as their associated subdomains. This queries all domain
blocks up front, then runs a regexp on each domain. This improves
performance over what may be the obvious implementation, which is
to ask `DomainBlocks.blocked?(domain)` for each domain -- this
hits the DB many times, slowing things down considerably.
* cleaning up code style
* Compiling regex
* Removing ternary operator
