Commit graph

497 commits

Author SHA1 Message Date
ThibG
81bda7d67c Add setting to not aggregate reblogs (#9248)
* Add setting to not aggregate reblogs

Fixes #9222

* Handle cases where user is nil in add_to_home and add_to_list

* Add hint for setting_aggregate_reblogs option

* Reword setting_aggregate_reblogs label
2018-12-09 13:03:01 +01:00
Eugen Rochko
c39d7e7b2b
Fix TLS handshake timeout not being enforced (#9381)
Follow-up to #9329
2018-11-27 19:46:05 +01:00
Eugen Rochko
43c311b3a1
Fix nil error when no DNS addresses are found for host (#9379) 2018-11-27 18:13:36 +01:00
Eugen Rochko
fd8145d232
Fix connect timeout not being enforced (#9329)
* Fix connect timeout not being enforced

The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds.

* Use timeout on each IP attempt, but limit to 2 attempts

* Fix code style issue

* Do not break Request#perform if no block given

* Update method stub in spec for Request

* Move timeout inside the begin/rescue block

* Use Resolv::DNS with timeout of 1 to get IP addresses

* Update Request spec to stub Resolv::DNS instead of Addrinfo

* Fix Resolve::DNS stubs in Request spec
2018-11-22 20:12:04 +01:00
ThibG
466e3d710c Include replies to list owner and replies to list members in list statuses (#9324) 2018-11-21 17:02:58 +01:00
Eugen Rochko
384e953b75
Revert connect timeout from 1s to 10s (#9319)
The failure rate in Sidekiq is too high
2018-11-21 17:00:56 +01:00
Eugen Rochko
9311430ed7
Prevent multiple handlers for Delete of Actor from running (#9292) 2018-11-16 19:46:23 +01:00
Eugen Rochko
8069fd636b
Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
ThibG
7f5a4be580 Fix emoji update date processing (#9255) 2018-11-10 23:59:51 +01:00
Eugen Rochko
4615512285
Reduce connect timeout limit and limit signature failures by source IP (#9236)
* Reduce connect timeout from 10s to 1s

* Limit failing signature verifications per source IP
2018-11-08 21:35:58 +01:00
Eugen Rochko
be202f9377
Accept the same payload in multiple inboxes and deliver (#9150) 2018-10-30 15:03:55 +01:00
m.b
2b18f5f85d Add Page AP type support (#9121) 2018-10-29 13:23:29 +01:00
ThibG
e53cc673e7 Ignore invalid hashtags on remote statuses instead of rejecting them (#9118)
Fixes #9115
2018-10-26 22:48:35 +02:00
ThibG
82e7988afc Fix missing mention argument when processing incoming Create activities (#9114)
* Fix missing `mention` argument when processing incoming Create activities

* Fix typo (param → params)
2018-10-26 12:59:59 +02:00
Eugen Rochko
d4cf963749
Allow inbox owner to view implicitly targeted ActivityPub payload (#9093)
Fix #9091
2018-10-25 18:12:22 +02:00
Eugen Rochko
fd5285658f
Add option to block reports from domain (#8830) 2018-10-20 08:02:44 +02:00
Eugen Rochko
ddd30f331c
Improve support for aspects/circles (#8950)
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
2018-10-17 17:13:04 +02:00
Eugen Rochko
21ad21cb50
Improve signature verification safeguards (#8959)
* Downcase signed_headers string before building the signed string

The HTTP Signatures draft does not mandate the “headers” field to be downcased,
but mandates the header field names to be downcased in the signed string, which
means that prior to this patch, Mastodon could fail to process signatures from
some compliant clients. It also means that it would not actually check the
Digest of non-compliant clients that wouldn't use a lowercased Digest field
name.

Thankfully, I don't know of any such client.

* Revert "Remove dead code (#8919)"

This reverts commit a00ce8c92c.

* Restore time window checking, change it to 12 hours

By checking the Date header, we can prevent replaying old vulnerable
signatures. The focus is to prevent replaying old vulnerable requests
from software that has been fixed in the meantime, so a somewhat long
window should be fine and accounts for timezone misconfiguration.

* Escape users' URLs when formatting them

Fixes possible HTML injection

* Escape all string interpolations in Formatter class

Slightly improve performance by reducing class allocations
from repeated Formatter#encode calls

* Fix code style issues
2018-10-12 00:15:55 +02:00
Eugen Rochko
61d44dd11f
Fix typo in ActivityPub Create handler (#8952)
Regression from #8951
2018-10-11 02:10:15 +02:00
Eugen Rochko
790d3bc637
Move network calls out of transaction in ActivityPub handler (#8951)
Mention and emoji code may perform network calls, but does not need
to do that inside the database transaction. This may improve availability
of database connections when using pgBouncer in transaction mode.
2018-10-11 00:50:18 +02:00
Eugen Rochko
774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Jeong Arm
144d73730d Leave unknown language as nil if account is remote (#8861)
* Force use language detector if account is remote

* Set unknown remote toot's language as nil
2018-10-05 19:17:46 +02:00
ThibG
ba444797d2 Fix handling of ActivityPub activities lacking some attributes (#8864) 2018-10-03 23:44:13 +02:00
Eugen Rochko
ef69c655cc
Fix class autoloading issue in ActivityPub::Activity::Create (#8820) 2018-09-28 17:02:53 +02:00
cbayerlein
4b78546135 Exclude replies from list timelines (#8683)
* Changed list behaviour

I added the following line to the FeedManager (app/lib/feed_manager.rb) in the push_to_list function:

`return false if status.reply?`

Now all posts that are replies are filtered out, so that now only "genuine" posts are displayed in the list.

This is a first approach to solve issue #5916

* Update feed_manager.rb

As suggested by @Gargron
2018-09-28 00:37:21 +02:00
ふぁぼ原
f7a6f9489d Add a new preference to always hide all media (#8569) 2018-09-25 05:09:35 +02:00
Matt Sweetman
1889526e23 Add user preference to always expand toots marked with content warnings (#8762) 2018-09-24 05:44:01 +02:00
Eugen Rochko
f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
Renato "Lond" Cerqueira
fe56d26f7b Fix autoplay issue with spoiler tag (#8540)
Add tests to avoid similar issues in the future
2018-08-31 15:16:59 +02:00
Eugen Rochko
e3764bdb52
Do not sign useless User-Agent or Accept-Encoding headers (#8533)
Fix #8080
2018-08-31 04:22:52 +02:00
Renato "Lond" Cerqueira
11658d8653 Add animate custom emoji param to embed pages (#8507)
* Add animate custom emoji param to embed pages

* Rename param, use it for avatars and gifs

* Fix issues pointed by codeclimate and breaking test

* Ignore brakeman warning
2018-08-30 23:14:01 +02:00
Eugen Rochko
cabdbb7f9c
Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
Quint Guvernator
da13fa5021 Fix low-hanging rubocop gripes (#8458)
* rubocop: quit being so picky

* rubocop: miscellany

* rubocop: prefer present to blank
2018-08-26 19:22:46 +02:00
Eugen Rochko
b4ba4b1b5d
Spread out crawling randomly to avoid DDoSing the link (#8445)
* Spread out crawling randomly to avoid DDoSing the link

Fix #4486

* Remove trailing whitespace
2018-08-26 00:33:57 +02:00
M Somerville
95bd0d4528 Support ActivityStreams’ summaryMap. (#8422)
In the same way as contentMap and nameMap.
2018-08-25 13:27:34 +02:00
Jakub Mendyk
6cb3514d64 Add ability to change an instance default theme from the administration panel (#7092) (#8381)
* Add default_settings class method to ScopedSettings

ScopedSettings was extended to use value of unscoped setting instead of
only using defaults set in config/settings.yml for selected settings.
This adds possibility for admins to set default values of users' settings,
for example default theme (as requested in #7092).

* Add ability to change an instance default theme

Closes #7092
2018-08-23 14:17:35 +02:00
ThibG
42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes #8376) (#8377) 2018-08-23 00:27:58 +02:00
Eugen Rochko
802cf6a4c5
Improve federated ID validation (#8372)
* Fix URI not being sufficiently validated with prefetched JSON

* Add additional id validation to OStatus documents, when possible
2018-08-22 20:55:14 +02:00
ThibG
59f7f4c923 Implement Undo { Accept { Follow } } (fixes #8234) (#8245)
* Add Follow#revoke_request!

* Implement Undo { Accept { Follow } } (fixes #8234)
2018-08-17 16:24:56 +02:00
ThibG
1ee675d68b Use correct activity id in Accept when receiving duplicate Follow (fixes #8218) (#8244) 2018-08-17 14:08:17 +02:00
Eugen Rochko
39e361a56d
Expect relays to answer with accept/reject (#8179) 2018-08-12 18:16:26 +02:00
ThibG
c52bcc0331 Serialize text-less statuses as '.' over OStatus (fixes #7856) (#8126) 2018-08-05 12:22:23 +02:00
Eugen Rochko
07b799468d
Fix incorrect context definition for the "featured" keyword (#8090)
* Fix incorrect context definition for the "featured" keyword

Fix #8077

* Adjust context definition for movedTo
2018-07-30 19:33:37 +02:00
Eugen Rochko
a23ac107e4
Fix activity:interactions counter to count all interactions (#8037) 2018-07-16 18:35:43 +02:00
Eugen Rochko
8f64b17d98 Whitelist dat/ipfs/gopher links in sanitizer (#8034)
Fix #7994
2018-07-16 22:17:43 +09:00
Eugen Rochko
2354b10eb5
Add admin dashboard (#8029) 2018-07-16 01:11:53 +02:00
Eugen Rochko
38e9662d78
Disable language detection for texts shorter than 140 characters (#8010)
If the input text is blank after preparation (only mention, or
only URL, or empty as in a media post), then use nil as language,
since it's OK to show to everyone.

Otherwise, always fall back to the server's default locale
2018-07-14 04:05:36 +02:00
Eugen Rochko
47c59591e0 Do not count self in potential friendships (#8014) 2018-07-14 10:44:23 +09:00
Eugen Rochko
20fefdb714
Make whole-word filter regex consistent between Ruby and JS (#7987) 2018-07-10 03:01:50 +02:00
ThibG
1ca4e51eb3 Add option to not consider word boundaries when processing keyword filtering (#7975)
* Add option to not consider word boundaries when filtering phrases

* Add a few tests for keyword/phrase filtering
2018-07-09 02:22:09 +02:00
Eugen Rochko
404c7702ec
In keyword filter, account for reblogs, HTML and whole-words (#7960)
* In keyword filter, account for reblogs, HTML and whole-words

* Match whole words in JS filter, too

* Fix typo
2018-07-06 02:15:44 +02:00
Eugen Rochko
4b198b172d
Check reblogged status for blocked/muted mentions (#7957) 2018-07-05 18:43:37 +02:00
Eugen Rochko
da8fe8079e
Re-add follow recommendations API (#7918)
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
2018-07-03 01:47:56 +02:00
MIYAGI Hikaru
ddd0bb69e1 Merge HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY into ALLOW_ACCESS_TO_HIDDEN_SERVICE (#7901)
If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address.
I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
2018-06-29 15:36:02 +02:00
Eugen Rochko
cdb101340a
Keyword/phrase filtering (#7905)
* Add keyword filtering

    GET|POST       /api/v1/filters
    GET|PUT|DELETE /api/v1/filters/:id

- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration

* Make sure expired filters don't get applied client-side

* Add missing API methods

* Remove "regex filter" from column settings

* Add tests

* Add test for FeedManager

* Add CustomFilter test

* Add UI for managing filters

* Add streaming API event to allow syncing filters

* Fix tests
2018-06-29 15:34:36 +02:00
Eugen Rochko
a58ec29631
Allow selecting default posting language instead of auto-detect (#7828)
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
2018-06-17 18:57:31 +02:00
Eugen Rochko
6ba7d9d0d9
Do not accept ActivityPub follow requests from blocked user (#7756)
* Do not accept ActivityPub follow requests from blocked user

Fix #7745

* Deliver auto-rejection immediately when follow-requested by blocked account

* Fix trailing whitespace
2018-06-08 00:14:06 +02:00
Eugen Rochko
1e938b966e
Exclude unlisted, private and direct toots from affecting trends (#7686) 2018-05-31 16:47:28 +02:00
abcang
90908fc24b Fix N+1 on AtomSerializer (#7669) 2018-05-29 13:34:02 +02:00
Eugen Rochko
ab36e0ef72 Record trending tags from ActivityPub, too (#7647) 2018-05-28 12:21:04 +09:00
ThibG
cdbdf7f98b Ignore multiple occurrences of a hashtag within a status (fixes #7585) (#7606) 2018-05-25 14:26:45 +02:00
Eugen Rochko
57b503d4ef
Resolve unknown status from Add activity, skip Remove if unknown (#7526)
Fix #7518
2018-05-18 11:33:56 +02:00
Eugen Rochko
1e02dc8715
Add preference to hide following/followers lists (#7532)
* Add preference to hide following/followers lists

- Public pages
- ActivityPub collections (does not return pages but does give total)
- REST API (unless it's your own) (does not federate)

Fix #6901

* Add preference

* Add delegation

* Fix issue

* Fix issue
2018-05-18 02:26:51 +02:00
MIYAGI Hikaru
919eef3098 User agent for WebFinger (#7531)
* User agent for WebFinger

* local_domain → web_domain

* 'http' is away accidentally...
2018-05-18 01:47:22 +02:00
Akihiko Odaki
55fd55714a Raise Mastodon::RaceConditionError if Redis lock failed (#7511)
An explicit error allows user agents to know the error and Sidekiq to
retry.
2018-05-16 12:29:45 +02:00
ThibG
7467361d70 Fetch boosted statuses on behalf of a follower (fixes #7426) (#7459)
When an ActivityPub Announce is processed and the boosted toot is not known,
fetch it on behalf of one of the booster's followers. This is to allow
fetching self-boosts of previously-unknown private toots.

If fetching on behalf of a user fails, try fetching it anonymously: the
selected follower of a boosting user may be banned by the boosted toot's
author.
2018-05-12 16:48:32 +02:00
Yamagishi Kazutoshi
6588f6a0a9 Disable inflate gzip on head method (#7432) 2018-05-10 14:36:12 +02:00
Yamagishi Kazutoshi
edf882320a Support gzip encoding on Request (#7425) 2018-05-09 16:59:39 +02:00
Eugen Rochko
6208ea5a53
If an OStatus message contains nsfw hashtag, mark it as sensitive (#7398)
* If an OStatus message contains nsfw hashtag, mark it as sensitive

Undo parts of #7048

* Put nsfw hashtag on OStatus messages if they have any media

* Fix code style issues
2018-05-07 14:49:13 +02:00
Eugen Rochko
61a9018607
Enable custom emojis in profiles (notes, field values, display names) (#7374)
Follow-up to #6124
2018-05-06 11:48:51 +02:00
Eugen Rochko
c947e2e4c5
Fix handling of malformed ActivityPub payloads when URIs are nil (#7370)
* Fix handling of malformed ActivityPub payloads when URIs are nil

* Gracefully handle JSON-LD canonicalization failures
2018-05-05 18:22:34 +02:00
Eugen Rochko
6793bec4c6
Store URIs of follows, follow requests and blocks for ActivityPub (#7160)
Same URI passed between follow request and follow, since they are
the same thing in ActivityPub. Local URIs are generated during
creation using UUIDs and are passed to serializers.
2018-05-04 21:14:34 +02:00
ThibG
a24605961a Fixes/do not override timestamps (#7336)
* Revert "Fixes/do not override timestamps (#7331)"

This reverts commit 581a5c9d29.

* Document Snowflake ID corner-case a bit more

Snowflake IDs are used for two purposes: making object identifiers harder to
guess and ensuring they are in chronological order. For this reason, they
are based on the `created_at` attribute of the object.

Unfortunately, inserting items with older snowflakes IDs will break the
assumption of consumers of the paging APIs that new items will always have
a greater identifier than the last seen one.

* Add `override_timestamps` virtual attribute to not correlate snowflake ID with created_at
2018-05-03 23:02:46 +02:00
ThibG
581a5c9d29 Fixes/do not override timestamps (#7331)
* Do not override timestamps for incoming toots

* Remove every reference to override_timestamps

Statuses are now created with the announced publishing date
and are only pushed to timelines if that date is at most
6 hours earlier than the time at which it is processed.
2018-05-03 13:33:08 +02:00
Eugen Rochko
a5293fdf61
Fix n+1 queries in StatusThreadingConcern (#7321) 2018-05-03 10:41:58 +02:00
Eugen Rochko
cb5b5cb5f7
Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
ThibG
d0cdd5cf94 Accept actor object updates from all supported actor types (#7312) 2018-05-02 16:08:16 +02:00
Eugen Rochko
f62539ce5c
Remove most behaviour disparities between blocks and mutes (#7231)
* Remove most behaviour disparities between blocks and mutes

The only differences between block and mute should be:

- Mutes can optionally NOT affect notifications
- Mutes should not be visible to the muted

Fix #7230
Fix #5713

* Do not allow boosting someone you blocked

Fix #7248

* Do not allow favouriting someone you blocked

* Fix nil error in StatusPolicy
2018-05-02 15:50:20 +02:00
Eugen Rochko
965345316f
Guard against nil URLs in Request class (#7284)
Fix #7265
2018-05-02 15:44:22 +02:00
Surinna Curtis
dc786c0cf4 Support Actors/Statuses with multiple types (#7305)
* Add equals_or_includes_any? helper in JsonLdHelper

* Support arrays in JSON-LD type fields for actors/tags/objects.

* Spec for resolving accounts with extension types

* Style tweaks for codeclimate
2018-05-02 12:40:24 +02:00
Eugen Rochko
705f1d7bf1
Fix missing updated_at attribute on emoji EntityCache (#7297)
Just don't try to save space by only selecting few attributes. If
anyone is wondering, this is needed because the emoji entity cache
is not really only used for entities, it's accessed again to
generate Emoji tags in ActivityPub/OStatus, so a lot more properties
are used than what is needed in HTML alone...
2018-04-30 22:49:33 +02:00
Eugen Rochko
295e3ef02b
Fix missing domain attribute in EntityCache for emoji (#7290) 2018-04-30 09:12:55 +02:00
Eugen Rochko
a872392cd9
Add entity cache (#7271)
* Add entity cache

Use a caching layer for mentions and custom emojis that are
dynamically extracted from text.

Reduce duplicate text extractions

* Fix code style issue
2018-04-27 01:38:10 +02:00
MIYAGI Hikaru
f58dcbc981 HTTP proxy support for outgoing request, manage access to hidden service (#7134)
* Add support for HTTP client proxy

* Add access control for darknet

Supress error when access to darknet via transparent proxy

* Fix the codes pointed out

* Lint

* Fix an omission + lint

* any? -> include?

* Change detection method to regexp to avoid test fail
2018-04-25 02:14:49 +02:00
Eugen Rochko
9d4710ed00
Add RSS feeds for end-users (#7259)
* Add RSS feed for accounts

* Add RSS feeds for hashtags

* Fix code style issues

* Fix code style issues
2018-04-25 02:10:02 +02:00
Eugen Rochko
7db7d68136
Detect and prevent image bombs, max. processable dimension 4096^2 (#7229) 2018-04-23 09:16:38 +02:00
Eugen Rochko
75c4ab9d12
Remove "nsfw" category for sensitive statuses in OStatus serializer (#7048)
Fix #7011
2018-04-22 22:09:03 +02:00
Eugen Rochko
78ed4ab75f
Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
2018-04-14 12:41:08 +02:00
Levi Bard
cd0eaa349c Enable updating additional account information from user preferences via rest api (#6789)
* Enable updating additional account information from user preferences via rest api
Resolves #6553

* Pacify rubocop

* Decoerce incoming settings in UserSettingsDecorator

* Create user preferences hash directly from incoming credentials instead of going through ActionController::Parameters

* Clean up user preferences update

* Use ActiveModel::Type::Boolean instead of manually checking stringified number equivalence
2018-04-08 13:43:10 +02:00
Eugen Rochko
b5726def55
Forward deletes on the same path as reply forwarding (#7058)
* Forward deletes on the same path as reply forwarding

* Remove trailing whitespace
2018-04-07 18:54:46 +02:00
David Underwood
123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar (#6124)
Federate custom emojis with accounts
2018-04-01 23:55:42 +02:00
ThibG
e573bb0990 Fix compatibility with PeerTube (#6968)
* Support fetching objects of convertible types by URL (fixes #6924)

* Ignore invalid hashtags
2018-03-30 15:44:54 +02:00
ThibG
9ed5eebd7c Do not ignore unknown media attachments, only skip them (#6948)
That way, they are displayed in a list below the corresponding toot.
2018-03-29 00:52:24 +02:00
Akihiko Odaki
40e5d2303b Validate HTTP response length while receiving (#6891)
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
2018-03-26 14:02:10 +02:00
Akihiko Odaki
54b273bf99 Close http connection in perform method of Request class (#6889)
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Rey Tucker
36b5703796 request: in the event of failure, try other IPs (#6761) (#6813)
* request: in the event of failure, try other IPs (#6761)

In the case where a name has multiple A/AAAA records, we should
try subsequent records instead of immediately failing when we have a
failure on the first IP address.

This significantly improves delivery success when there are network
connectivity problems affecting only IPv4 or IPv6.

* fix method call style

* request_spec: adjust test case to use Addrinfo

* request: Request/open: move private addr check to within begin/rescue

* request_spec: add case to test failover, fix exception check

* Double Addrinfo.foreach so that it correctly yields instances
2018-03-20 09:06:08 +01:00
ThibG
ff6b8a6443 Serialize mentions in the order they are added (#6836)
Up until now, the order seemed to be in the *opposite* order,
which caused the WebUI to populate mentions in reversed order
when replying to toots local to one's instance.
2018-03-19 20:19:35 +01:00
Eugen Rochko
f02411da40
Ignore media validation when attaching to status during processing (#6822)
Fix #6821
2018-03-19 01:51:19 +01:00
MitarashiDango
6dcf96271e fix validation error (media only status) (#6684)
* fix validation error (media only status)

* Incorporating review suggestions

* Reflect similar fix to OStatus side

* Fix not to include media in transaction

* Restore the limit of the number of media

* Fix not to return nil
2018-03-08 01:22:47 +01:00
Eugen Rochko
cfa9b6e13a
Remove text requirement when media attached from statuses (#6672) 2018-03-07 08:28:52 +01:00
Eugen Rochko
e6520c0270
Fix #6657 - Use target instead of origin in Remove activity (#6664) 2018-03-07 03:54:46 +01:00
Eugen Rochko
9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
Eugen Rochko
e852872846
Fix #5708: Reject->Follow will remove the follow if it exists (#6571) 2018-02-28 06:55:06 +01:00
Eugen Rochko
41a01bec23
Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
Akihiko Odaki
2e8a492e88 Raise Mastodon::HostValidationError when host for HTTP request is private (#6410) 2018-02-24 19:16:11 +01:00
Akihiko Odaki
f8f0572ee0 Do not push status to feed if its reblog is already inserted (#6488)
A complemental change for precompute_feed_service_spec.rb also fixes its
random failure which is caused by the Snowlake randomization of the order
of an original status and its reblog.
2018-02-24 05:40:18 +01:00
Eugen Rochko
90f12f2e5a
Focal points (#6520)
* Add focus param to media API, center thumbnails on focus point

* Add UI for setting a focal point

* Improve focal point icon on upload item

* Use focal point in upload preview

* Add focalPoint property to ActivityPub

* Don't show focal point button for non-image attachments
2018-02-22 00:35:46 +01:00
Eugen Rochko
a7171af0a3
Fix avatar and header issues by using custom geometry detector (#6515)
* Fix avatar and header issues by using custom geometry detector

Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.

One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.

Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.

* Only invoke convert if dimension or format needs to be changed
2018-02-21 03:40:12 +01:00
Eugen Rochko
a71af98401
Push discovered status through streaming API within a time window (#6484)
Time window of 6 hours
2018-02-17 14:28:48 +01:00
Eugen Rochko
3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00
Eugen Rochko
76f3d5d16b
Add preference to always display sensitive media (#6448) 2018-02-09 00:26:57 +01:00
Akihiko Odaki
0be9a1e321 Accept ActivityPub announce from the author of the original note (#6236) 2018-02-02 10:22:15 +01:00
ThibG
b1e03197fa Process mentions and reblogs even from resolved threads (#6299)
This may lead to out-of-order notifications, but this is better than not having
notifications at all.
2018-01-19 19:11:35 +01:00
Eugen Rochko
35b84985a8
Skip ActivityPub Announces of non-public objects (#6230)
* Skip ActivityPub Announces of non-public objects

* Skip OStatus reblogs of non-public statuses
2018-01-09 19:35:10 +01:00
puckipedia
628358aeea Add the author of a status to cc if reblogged (#6226)
This makes slightly more sense, and ensures that the author of a post is always referenced in the audience (which some servers might rely on). And the announce is POSTed to the author's inbox anyways.
2018-01-09 00:47:43 +01:00
Eugen Rochko
dbda87c31f
Revert #5772 (#6221) 2018-01-08 10:57:52 +01:00
Eugen Rochko
e4a241abef
Fix bad URL schemes being accepted (#6219)
* Fix actors accepting invalid URI schemes or different host between URI and URL

* Fix statuses accepting invalid URI scheme or different host to actor

* Adjust tests to new requirements

* Improve readability of mismatching_origin?/invalid_origin? methods
2018-01-08 05:00:23 +01:00
puckipedia
0eff42d688 Move Article from supported to converted types (#6218) 2018-01-08 00:21:14 +01:00
ThibG
a8b51124ba Don't normalize URLs in toots (#6134)
* Don't normalize URLs in toots

URL normalization is ill-defined and may cause certain links to break.

* Change specs since we are not normalizing user-provided URLs
2018-01-03 20:51:33 +01:00
Akihiko Odaki
161c72d66d Allow to dereference Follow object for ActivityPub (#5772)
* Allow to dereference Follow object for ActivityPub

* Accept IRI as object representation for Accept activity
2018-01-03 18:08:57 +01:00
puckipedia
545095b3ce [!] Sanitize incoming classlist properly (#6162)
* Sanitize classlist properly

* Actually properly sanitize every class after the first

* Improve Formatter spec to check for multiple classes and non-space whitespace
2018-01-03 03:54:08 +01:00
Eugen Rochko
d60fd87e01
Don't leave behind husk of remotely-deleted profile (#6159)
There's no reason for an Account record to persist after Delete->Actor is received. SuspendAccountService is necessary to make sure deleted toots get sent over streaming API properly and home feeds get cleaned up. By removing Account record, we can ensure that if in the future the account is restored remotely (or username reused), it can start with a clean slate.
2018-01-03 00:38:02 +01:00
Eugen Rochko
1419f656e2
Fix stats expiring too quickly because of variable mistake (#6155) 2018-01-02 14:02:53 +01:00
Eugen Rochko
38fc1b498d
Add more instance stats APIs (#6125)
* Add GET /api/v1/instance/peers API to reveal known domains

* Add GET /api/v1/instance/activity API

* Make new APIs disableable, exclude private statuses from activity stats

* Fix code style issue

* Fix week timestamps
2017-12-29 19:52:04 +01:00
Akihiko Odaki
e0ef7f9d79 Fix XML oEmbed support discovery (#6104) 2017-12-27 03:29:49 +01:00
Eugen Rochko
3d3b403359
Do not hide statuses from silenced accounts from other silenced accounts (#6030) 2017-12-15 01:54:05 +01:00
Eugen Rochko
a8deb6648b
Fix redundant HTTP request in FetchLinkCardService (#6002) 2017-12-13 12:15:28 +01:00
abcang
3caec1ecc2 Save media outside transaction (#5959) 2017-12-10 16:33:52 +01:00
Yamagishi Kazutoshi
b21db9bbde Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
Eugen Rochko
4c6b5dbe96
Add semi-support for Video/Image objects in ActivityPub (#5848)
* Add semi-support for Video/Image objects in ActivityPub

Video and Image objects will create corresponding status records
with manually crafted text contents (title + URL)

* Extract html-url-finding logic into JsonLdHelper

* Fallback to id when url missing, extract supported object types
2017-11-30 04:06:20 +01:00
Eugen Rochko
85e97ecab6
Fix too many forwards (#5854)
* Avoid sending explicit Undo->Announce when original deleted

* Do not forward a reply back to the server that sent it

* Deduplicate inboxes of rebloggers' followers for delete forwarding

* Adjust test

* Fix wrong class, bad SQL, wrong variable, outdated comment
2017-11-30 03:50:05 +01:00
aschmitz
eeaec39888 Allow hiding of reblogs from followed users (#5762)
* Allow hiding of reblogs from followed users

This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).

This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.

The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.

Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.

Tests included.

See also <https://github.com/glitch-soc/mastodon/pull/212>.

* Rubocop fixes

* Code review changes

* Test fixes

This patchset closes #648 and resolves #3271.

* Rubocop fix

* Revert reblogs defaulting in argument, fix tests

It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.

We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
2017-11-28 15:00:35 +01:00
Eugen Rochko
58cede4808
Profile redirect notes (#5746)
* Serialize moved accounts into REST and ActivityPub APIs

* Parse federated moved accounts from ActivityPub

* Add note about moved accounts to public profiles

* Add moved account message to web UI

* Fix code style issues
2017-11-18 19:39:02 +01:00
Eugen Rochko
24cafd73a2
Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
Eugen Rochko
3e90987c8b Fix some rubocop style issues (#5730) 2017-11-17 10:06:26 +09:00
Renato "Lond" Cerqueira
ad207456d6 Improve language filter (#5724)
* Scrub text of html before detecting language.

* Detect language on statuses coming from activitypub.

* Fix rubocop comments.

* Remove custom emoji from text before language detection
2017-11-16 13:51:38 +01:00
MIYAGI Hikaru
782224c991 Avoid emojifying on invisible text (#5558) 2017-11-07 14:48:13 +01:00
puckipedia
0cb329f63a Allow ActivityPub Note's tag and attachment to be single objects (#5534) 2017-10-27 16:10:36 +02:00
Eugen Rochko
484208ce12 When status is fetched instead of delivered, do not stream it (#5437) 2017-10-17 20:05:21 +02:00
aschmitz
554c2fd8af Clean up reblog tracking keys, related improvements (#5428)
* Clean up reblog-tracking sets from FeedManager

Builds on #5419, with a few minor optimizations and cleanup of sets
after they are no longer needed.

* Update tests, fix multiply-reblogged case

Previously, we would have lost the fact that a given status was
reblogged if the displayed reblog of it was removed, now we don't.

Also added tests to make sure FeedManager#trim cleans up our reblog
tracking keys, fixed up FeedCleanupScheduler to use the right loop,
and fixed the test for it.
2017-10-17 11:45:06 +02:00
Eugen Rochko
34118169ac Keep references to all reblogs of a status on home feed (#5419)
* Keep references to all reblogs of a status on home feed

When inserting reblog: Add to set of reblogs of this status on
the feed, if original status was present in the feed, add it to
that set as well.

When removing a reblog: Remove it from that set. Take random
remaining item from the set. If one exists, re-insert it into feed,
otherwise do not re-insert anything.

Fix #4210

* When original is removed, toss out reblog references
2017-10-16 20:44:31 +02:00
Eugen Rochko
7cc71748ce Ensure that feed renegeration restores non-zero items (#5409)
Fix #5398

Ordering the home timeline query by account_id meant that the first
100 items belonged to a single account. There was also no reason to
reverse-iterate over the statuses. Assuming the user accesses the
feed halfway-through, it's better to have recent statuses already
available at the top. Therefore working from newer->older is ideal.

If the algorithm ends up filtering all items out during last-mile
filtering, repeat again a page further. The algorithm terminates
when either at least one item has been added, or if the database
query returns nothing (end of data reached)
2017-10-16 16:08:51 +02:00
unarist
6f490b4bfe Fix un-reblogged status being at wrong position in the home timeline (#5418)
We've changed un-reblogging behavior when we implement Snowflake, to insert un-reblogged status at the position reblogging status existed.

However, our API expects home timeline is ordered by status ids, and max_id/since_id filters by zset score. Due to this, un-reblogged status appears as a last item of result set, and timeline expansion may skips many statuses.

So this reverts that change...reblogged status inserted at corresponding position to its id.
2017-10-16 15:58:23 +02:00
Nolan Lawson
fa0be3f834 Add option to reduce motion (#5393)
* Add option to reduce motion

* Use HOC to wrap all Motion calls

* fix case-sensitive issue

* Avoid updating too frequently

* Get rid of unnecessary change to _simple_status.html.haml
2017-10-16 09:36:15 +02:00
unarist
8125fdc19f Use atomUri in Undo activity of Announce (#5376)
This allows deletion of reblogs which delivered before with OStatus URI.
2017-10-14 14:42:09 +02:00
Eugen Rochko
df7dbc41ae Fix NameError: uninitialized constant OStatus::AtomSerializer::TagManager (#5371)
This error occurred at least in development environment
2017-10-13 16:44:43 +02:00
Eugen Rochko
b8db386e05 Fix UserTrackingConcern firing on every request, optimize some queries (#5368)
- For some reason, :if option on before_action did not work. It got
  executed every time, returned false, and the action run anyway,
  which led to the current_sign_in_at and sign_in_count being
  updated on every request
- Return "do not filter" early in FeedManager#filter_from_home? if
  the status is authored by receiver. Usually this method is not
  called for own statuses at all, but it is called when Feed#get
  uses the database
- Return early if #reload_stale_associations! has nothing to load
  to save a database query with WHERE 1=0
2017-10-13 16:44:29 +02:00
Eugen Rochko
388d093beb When unfollowing, remove from home in web UI immediately (#5369)
Do NOT send "delete" through streaming API when unmerging from
home timeline. "delete" implies that the original status was
deleted, which is not true!
2017-10-13 16:44:02 +02:00
Eugen Rochko
cfa68907ae Fix #5271 - Fix missing attribute in remove_from_feed (#5277)
Regression from #4801
2017-10-08 21:55:34 +02:00
Eugen Rochko
0717d9b3e6 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00