* Fix connect timeout not being enforced
The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds.
* Use timeout on each IP attempt, but limit to 2 attempts
* Fix code style issue
* Do not break Request#perform if no block given
* Update method stub in spec for Request
* Move timeout inside the begin/rescue block
* Use Resolv::DNS with timeout of 1 to get IP addresses
* Update Request spec to stub Resolv::DNS instead of Addrinfo
* Fix Resolve::DNS stubs in Request spec
* Add silent column to mentions
* Save silent mentions in ActivityPub Create handler and optimize it
Move networking calls out of the database transaction
* Add "limited" visibility level masked as "private" in the API
Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic
* Ensure silent column is non-null, add spec
* Ensure filters don't check silent mentions for blocks/mutes
As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering
* Clean up code
* Use Status#active_mentions to limit returned mentions
* Fix code style issues
* Use Status#active_mentions in Notification
And remove stream_entry eager-loading from Notification
* Downcase signed_headers string before building the signed string
The HTTP Signatures draft does not mandate the “headers” field to be downcased,
but mandates the header field names to be downcased in the signed string, which
means that prior to this patch, Mastodon could fail to process signatures from
some compliant clients. It also means that it would not actually check the
Digest of non-compliant clients that wouldn't use a lowercased Digest field
name.
Thankfully, I don't know of any such client.
* Revert "Remove dead code (#8919)"
This reverts commit a00ce8c92c.
* Restore time window checking, change it to 12 hours
By checking the Date header, we can prevent replaying old vulnerable
signatures. The focus is to prevent replaying old vulnerable requests
from software that has been fixed in the meantime, so a somewhat long
window should be fine and accounts for timezone misconfiguration.
* Escape users' URLs when formatting them
Fixes possible HTML injection
* Escape all string interpolations in Formatter class
Slightly improve performance by reducing class allocations
from repeated Formatter#encode calls
* Fix code style issues
Mention and emoji code may perform network calls, but does not need
to do that inside the database transaction. This may improve availability
of database connections when using pgBouncer in transaction mode.
* Add conversations API
* Add web UI for conversations
* Add test for conversations API
* Add tests for ConversationAccount
* Improve web UI
* Rename ConversationAccount to AccountConversation
* Remove conversations on block and mute
* Change last_status_id to be a denormalization of status_ids
* Add optimistic locking
* Changed list behaviour
I added the following line to the FeedManager (app/lib/feed_manager.rb) in the push_to_list function:
`return false if status.reply?`
Now all posts that are replies are filtered out, so that now only "genuine" posts are displayed in the list.
This is a first approach to solve issue #5916
* Update feed_manager.rb
As suggested by @Gargron
* Verify link ownership with rel="me"
* Add explanation about verification to UI
* Perform link verifications
* Add click-to-copy widget for verification HTML
* Redesign edit profile page
* Redesign forms
* Improve responsive design of settings pages
* Restore landing page sign-up form
* Fix typo
* Support <link> tags, add spec
* Fix links not being verified on first discovery and passive updates
* Add animate custom emoji param to embed pages
* Rename param, use it for avatars and gifs
* Fix issues pointed by codeclimate and breaking test
* Ignore brakeman warning
* If an Update is signed with known key, skip re-following procedure
Because it means the remote actor did *not* lose their database
* Add CLI method for rotating keys
bin/tootctl accounts rotate [USERNAME]
Generates a new RSA key per account and sends out an Update activity
signed with the old key.
* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts
* Skip suspended accounts in key rotation
* Add default_settings class method to ScopedSettings
ScopedSettings was extended to use value of unscoped setting instead of
only using defaults set in config/settings.yml for selected settings.
This adds possibility for admins to set default values of users' settings,
for example default theme (as requested in #7092).
* Add ability to change an instance default theme
Closes#7092
If the input text is blank after preparation (only mention, or
only URL, or empty as in a media post), then use nil as language,
since it's OK to show to everyone.
Otherwise, always fall back to the server's default locale
* Re-add follow recommendations API
GET /api/v1/suggestions
Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.
* Track interactions with people you don't follow
Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:
- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20
Following them, muting or blocking will remove them from the list,
obviously.
* Remove triadic closures, ensure potential friendships are trimmed
If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address.
I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
* Add keyword filtering
GET|POST /api/v1/filters
GET|PUT|DELETE /api/v1/filters/:id
- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration
* Make sure expired filters don't get applied client-side
* Add missing API methods
* Remove "regex filter" from column settings
* Add tests
* Add test for FeedManager
* Add CustomFilter test
* Add UI for managing filters
* Add streaming API event to allow syncing filters
* Fix tests
* Do not accept ActivityPub follow requests from blocked user
Fix#7745
* Deliver auto-rejection immediately when follow-requested by blocked account
* Fix trailing whitespace
* Add preference to hide following/followers lists
- Public pages
- ActivityPub collections (does not return pages but does give total)
- REST API (unless it's your own) (does not federate)
Fix#6901
* Add preference
* Add delegation
* Fix issue
* Fix issue
When an ActivityPub Announce is processed and the boosted toot is not known,
fetch it on behalf of one of the booster's followers. This is to allow
fetching self-boosts of previously-unknown private toots.
If fetching on behalf of a user fails, try fetching it anonymously: the
selected follower of a boosting user may be banned by the boosted toot's
author.
* If an OStatus message contains nsfw hashtag, mark it as sensitive
Undo parts of #7048
* Put nsfw hashtag on OStatus messages if they have any media
* Fix code style issues
Same URI passed between follow request and follow, since they are
the same thing in ActivityPub. Local URIs are generated during
creation using UUIDs and are passed to serializers.
* Revert "Fixes/do not override timestamps (#7331)"
This reverts commit 581a5c9d29.
* Document Snowflake ID corner-case a bit more
Snowflake IDs are used for two purposes: making object identifiers harder to
guess and ensuring they are in chronological order. For this reason, they
are based on the `created_at` attribute of the object.
Unfortunately, inserting items with older snowflakes IDs will break the
assumption of consumers of the paging APIs that new items will always have
a greater identifier than the last seen one.
* Add `override_timestamps` virtual attribute to not correlate snowflake ID with created_at
* Do not override timestamps for incoming toots
* Remove every reference to override_timestamps
Statuses are now created with the announced publishing date
and are only pushed to timelines if that date is at most
6 hours earlier than the time at which it is processed.
* No need to re-require sidekiq plugins, they are required via Gemfile
* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile
* Replace ruby-oembed with FetchOEmbedService
Reduce startup by 45382 allocated objects
* Remove preloaded JSON-LD in favour of caching HTTP responses
Reduce boot RAM by about 6 MiB
* Fix tests
* Fix test suite by stubbing out JSON-LD contexts
* Remove most behaviour disparities between blocks and mutes
The only differences between block and mute should be:
- Mutes can optionally NOT affect notifications
- Mutes should not be visible to the muted
Fix#7230Fix#5713
* Do not allow boosting someone you blocked
Fix#7248
* Do not allow favouriting someone you blocked
* Fix nil error in StatusPolicy
* Add equals_or_includes_any? helper in JsonLdHelper
* Support arrays in JSON-LD type fields for actors/tags/objects.
* Spec for resolving accounts with extension types
* Style tweaks for codeclimate
Just don't try to save space by only selecting few attributes. If
anyone is wondering, this is needed because the emoji entity cache
is not really only used for entities, it's accessed again to
generate Emoji tags in ActivityPub/OStatus, so a lot more properties
are used than what is needed in HTML alone...
* Add entity cache
Use a caching layer for mentions and custom emojis that are
dynamically extracted from text.
Reduce duplicate text extractions
* Fix code style issue
* Add support for HTTP client proxy
* Add access control for darknet
Supress error when access to darknet via transparent proxy
* Fix the codes pointed out
* Lint
* Fix an omission + lint
* any? -> include?
* Change detection method to regexp to avoid test fail
* Add bio fields
- Fix#3211
- Fix#232
- Fix#121
* Display bio fields in web UI
* Fix output of links and missing fields
* Federate bio fields over ActivityPub as PropertyValue
* Improve how the fields are stored, add to Edit profile form
* Add rel=me to links in fields
Fix#121
* Enable updating additional account information from user preferences via rest api
Resolves#6553
* Pacify rubocop
* Decoerce incoming settings in UserSettingsDecorator
* Create user preferences hash directly from incoming credentials instead of going through ActionController::Parameters
* Clean up user preferences update
* Use ActiveModel::Type::Boolean instead of manually checking stringified number equivalence
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
* request: in the event of failure, try other IPs (#6761)
In the case where a name has multiple A/AAAA records, we should
try subsequent records instead of immediately failing when we have a
failure on the first IP address.
This significantly improves delivery success when there are network
connectivity problems affecting only IPv4 or IPv6.
* fix method call style
* request_spec: adjust test case to use Addrinfo
* request: Request/open: move private addr check to within begin/rescue
* request_spec: add case to test failover, fix exception check
* Double Addrinfo.foreach so that it correctly yields instances
Up until now, the order seemed to be in the *opposite* order,
which caused the WebUI to populate mentions in reversed order
when replying to toots local to one's instance.
* fix validation error (media only status)
* Incorporating review suggestions
* Reflect similar fix to OStatus side
* Fix not to include media in transaction
* Restore the limit of the number of media
* Fix not to return nil
A complemental change for precompute_feed_service_spec.rb also fixes its
random failure which is caused by the Snowlake randomization of the order
of an original status and its reblog.
* Add focus param to media API, center thumbnails on focus point
* Add UI for setting a focal point
* Improve focal point icon on upload item
* Use focal point in upload preview
* Add focalPoint property to ActivityPub
* Don't show focal point button for non-image attachments
* Fix avatar and header issues by using custom geometry detector
Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.
One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.
Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.
* Only invoke convert if dimension or format needs to be changed
* Add full-text search for authorized statuses
- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index
Fix#5880Fix#4293Fix#1152
* Add commented out docker-compose configuration for ES container
* Optimize index import, filter search results
* Add basic normalization to the index
* Add better stemming and normalization to the index
* Skip webfinger request if search query includes both @ and a space
* Fix code style
* Visually separate search result sections
* Fix code style issues
This makes slightly more sense, and ensures that the author of a post is always referenced in the audience (which some servers might rely on). And the announce is POSTed to the author's inbox anyways.
* Fix actors accepting invalid URI schemes or different host between URI and URL
* Fix statuses accepting invalid URI scheme or different host to actor
* Adjust tests to new requirements
* Improve readability of mismatching_origin?/invalid_origin? methods
* Don't normalize URLs in toots
URL normalization is ill-defined and may cause certain links to break.
* Change specs since we are not normalizing user-provided URLs
* Sanitize classlist properly
* Actually properly sanitize every class after the first
* Improve Formatter spec to check for multiple classes and non-space whitespace
There's no reason for an Account record to persist after Delete->Actor is received. SuspendAccountService is necessary to make sure deleted toots get sent over streaming API properly and home feeds get cleaned up. By removing Account record, we can ensure that if in the future the account is restored remotely (or username reused), it can start with a clean slate.
* Add GET /api/v1/instance/peers API to reveal known domains
* Add GET /api/v1/instance/activity API
* Make new APIs disableable, exclude private statuses from activity stats
* Fix code style issue
* Fix week timestamps
* Add semi-support for Video/Image objects in ActivityPub
Video and Image objects will create corresponding status records
with manually crafted text contents (title + URL)
* Extract html-url-finding logic into JsonLdHelper
* Fallback to id when url missing, extract supported object types
* Avoid sending explicit Undo->Announce when original deleted
* Do not forward a reply back to the server that sent it
* Deduplicate inboxes of rebloggers' followers for delete forwarding
* Adjust test
* Fix wrong class, bad SQL, wrong variable, outdated comment
* Allow hiding of reblogs from followed users
This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).
This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.
The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.
Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.
Tests included.
See also <https://github.com/glitch-soc/mastodon/pull/212>.
* Rubocop fixes
* Code review changes
* Test fixes
This patchset closes#648 and resolves#3271.
* Rubocop fix
* Revert reblogs defaulting in argument, fix tests
It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.
We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
* Serialize moved accounts into REST and ActivityPub APIs
* Parse federated moved accounts from ActivityPub
* Add note about moved accounts to public profiles
* Add moved account message to web UI
* Fix code style issues
* Add structure for lists
* Add list timeline streaming API
* Add list APIs, bind list-account relation to follow relation
* Add API for adding/removing accounts from lists
* Add pagination to lists API
* Add pagination to list accounts API
* Adjust scopes for new APIs
- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope
* Add test for wrong user context on list timeline
* Clean up tests
* Scrub text of html before detecting language.
* Detect language on statuses coming from activitypub.
* Fix rubocop comments.
* Remove custom emoji from text before language detection
* Clean up reblog-tracking sets from FeedManager
Builds on #5419, with a few minor optimizations and cleanup of sets
after they are no longer needed.
* Update tests, fix multiply-reblogged case
Previously, we would have lost the fact that a given status was
reblogged if the displayed reblog of it was removed, now we don't.
Also added tests to make sure FeedManager#trim cleans up our reblog
tracking keys, fixed up FeedCleanupScheduler to use the right loop,
and fixed the test for it.
* Keep references to all reblogs of a status on home feed
When inserting reblog: Add to set of reblogs of this status on
the feed, if original status was present in the feed, add it to
that set as well.
When removing a reblog: Remove it from that set. Take random
remaining item from the set. If one exists, re-insert it into feed,
otherwise do not re-insert anything.
Fix#4210
* When original is removed, toss out reblog references
Fix#5398
Ordering the home timeline query by account_id meant that the first
100 items belonged to a single account. There was also no reason to
reverse-iterate over the statuses. Assuming the user accesses the
feed halfway-through, it's better to have recent statuses already
available at the top. Therefore working from newer->older is ideal.
If the algorithm ends up filtering all items out during last-mile
filtering, repeat again a page further. The algorithm terminates
when either at least one item has been added, or if the database
query returns nothing (end of data reached)
We've changed un-reblogging behavior when we implement Snowflake, to insert un-reblogged status at the position reblogging status existed.
However, our API expects home timeline is ordered by status ids, and max_id/since_id filters by zset score. Due to this, un-reblogged status appears as a last item of result set, and timeline expansion may skips many statuses.
So this reverts that change...reblogged status inserted at corresponding position to its id.
* Add option to reduce motion
* Use HOC to wrap all Motion calls
* fix case-sensitive issue
* Avoid updating too frequently
* Get rid of unnecessary change to _simple_status.html.haml
- For some reason, :if option on before_action did not work. It got
executed every time, returned false, and the action run anyway,
which led to the current_sign_in_at and sign_in_count being
updated on every request
- Return "do not filter" early in FeedManager#filter_from_home? if
the status is authored by receiver. Usually this method is not
called for own statuses at all, but it is called when Feed#get
uses the database
- Return early if #reload_stale_associations! has nothing to load
to save a database query with WHERE 1=0
Do NOT send "delete" through streaming API when unmerging from
home timeline. "delete" implies that the original status was
deleted, which is not true!
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
