Commit graph

20 commits

Author SHA1 Message Date
Claire
abcc0b38fa
Fix incorrect PostgreSQL version check () 2023-09-19 17:31:58 +02:00
Eugen Rochko
0e2a4d3897
Fix adding column with default value taking longer on Postgres >= 11 () 2023-08-08 09:09:58 +02:00
Nick Schonning
e2567df860
Enable Lint/RedundantCopDisableDirective () 2023-02-18 04:30:14 +01:00
Claire
cb45c04d26
Fix migration error handling () 2022-04-07 20:46:30 +02:00
Claire
5f0fc639da
Fix error re-running some migrations if they get interrupted at the wrong moment () 2022-04-07 20:17:49 +02:00
Eugen Rochko
75e33fd08f
Fix null values being included in some indexes ()
* Fix null values being included in some indexes

* Update lib/mastodon/migration_helpers.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add documentation link to corruption error message

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-12 08:12:57 +01:00
Claire
a99adeaad3
Fix edge case in migration helpers that caused crash because of PostgreSQL quirks () 2022-01-30 22:34:54 +01:00
Claire
11d3c065a5
Fix migration script not being able to run if it fails midway ()
* Fix migration script not being able to run if it fails midway

* Fix old migration script

* Fix old migration script

* Refactor CorruptionError
2021-06-02 19:15:17 +02:00
Claire
a5f91a11d0
Fix older migrations on Ruby 3 () 2021-05-07 15:56:45 +02:00
Claire
c31c95ffe4
Remove MySQL-specific code from Mastodon::MigrationHelpers ()
Mastodon::MigrationHelpers has been forked from Gitlab a long time ago, but
Mastodon has never supported using a MySQL database.

Removing MySQL support from Mastodon::MigrationHelpers makes it a little easier
to maintain. In particular, it removes code that would need updating with
Rails 6.
2021-03-19 13:14:40 +01:00
Claire
b358229834
Further preparation for Rails 6 ()
* Use ActiveRecord::Result#to_ary instead of deprecated to_hash

They do the same thing, and to_hash has been removed from Rails 6.1

* Explicitly name polymorphic indexes to workaround a bug in Rails 6.1

cf. https://github.com/rails/rails/issues/41693

* Fix incorrect usage of “foreign_key” in migration script

* Use `ActiveModel::Errors#delete` instead of deprecated clear method

* Fix link headers tests on Rails 6.1

Rails 6.1 adds values to the Link header by default, thus it is not a
LinkHeader object anymore. Fix the test to parse the Link header instead
of assuming it is a LinkHeader.
2021-03-19 02:45:34 +01:00
Marcin Cieślak
bd4bbba98a
Do not suggest to login into the GitLab database ()
As pointed out on Discourse:

https://discourse.joinmastodon.org/t/obscure-wtf-error-message-running-migrations-for-3-1/2524

The message the administrators were getting was telling them
to log in to the "GitLab database" and also mentions MySQL
2020-02-15 12:52:57 +01:00
Stanislas
01bc2f84a9 Fix SUPERUSER postgres command () 2019-01-27 23:56:07 +01:00
ashleyhull-versent
f194857ac9 rubocop issues - Cleaning up ()
* cleanup pass

* undo mistakes

* fixed.

* revert
2018-10-08 04:50:11 +02:00
luzpaz
40dd19be37 Misc. typos ()
Found via `codespell -q 3 --skip="./app/javascript/mastodon/locales,./config/locales"`
2018-09-14 00:53:09 +02:00
Yamagishi Kazutoshi
084cf0babf Add extract_foreign_key_action to Mastodon::MigrationHelpers () 2018-04-20 12:21:28 +02:00
Yamagishi Kazutoshi
b21db9bbde Using double splat operator () 2017-12-06 11:41:57 +01:00
ysksn
9dd5e329ab Remove empty strings () 2017-11-17 10:52:30 +09:00
aschmitz
e5d8166a12 Fix ()
This fixes , which occurred when using MigrationHelpers on a
table that Postgres estimated to be non-empty when it was actually
empty.
2017-10-11 21:20:39 +02:00
aschmitz
97c02c3389 Make IdsToBigints (mostly!) non-blocking ()
* Make IdsToBigints (mostly!) non-blocking

This pulls in GitLab's MigrationHelpers, which include code to make
column changes in ways that Postgres can do without locking. In general,
this involves creating a new column, adding an index and any foreign
keys as appropriate, adding a trigger to keep it populated alongside
the old column, and then progressively copying data over to the new
column, before removing the old column and replacing it with the new
one.

A few changes to GitLab's MigrationHelpers were necessary:

* Some changes were made to remove dependencies on other GitLab code.
* We explicitly wait for index creation before forging ahead on column
  replacements.
* We use different temporary column names, to avoid running into index
  name length limits.
* We rename the generated indices back to what they "should" be after
  replacing columns.
* We rename the generated foreign keys to use the new column names when
  we had to create them. (This allows the migration to be rolled back
  without incident.)

# Big Scary Warning

There are two things here that may trip up large instances:

1. The change for tables' "id" columns is not concurrent. In
   particular, the stream_entries table may be big, and does not
   concurrently migrate its id column. (On the other hand, x_id type
   columns are all concurrent.)
2. This migration will take a long time to run, *but it should not
   lock tables during that time* (with the exception of the "id"
   columns as described above). That means this should probably be run
   in `screen` or some other session that can be run for a long time.
   Notably, the migration will take *longer* than it would without
   these changes, but the website will still be responsive during that
   time.

These changes were tested on a relatively large statuses table (256k
entries), and the service remained responsive during the migration.
Migrations both forward and backward were tested.

* Rubocop fixes

* MigrationHelpers: Support ID columns in some cases

This doesn't work in cases where the ID column is referred to as a
foreign key by another table.

* MigrationHelpers: support foreign keys for ID cols

Note that this does not yet support foreign keys on non-primary-key
columns, but Mastodon also doesn't yet have any that we've needed to
migrate.

This means we can perform fully "concurrent" migrations to change ID
column types, and the IdsToBigints migration can happen with effectively
no downtime. (A few operations require a transaction, such as renaming
columns or deleting them, but these transactions should not block for
noticeable amounts of time.)

The algorithm for generating foreign key names has changed with this,
and therefore all of those changed in schema.rb.

* Provide status, allow for interruptions

The MigrationHelpers now allow restarting the rename of a column if it
was interrupted, by removing the old "new column" and re-starting the
process.

Along with this, they now provide status updates on the changes which
are happening, as well as indications about when the changes can be
safely interrupted (when there are at least 10 seconds estimated to be
left before copying data is complete).

The IdsToBigints migration now also sorts the columns it migrates by
size, starting with the largest tables. This should provide
administrators a worst-case scenario estimate for the length of
migrations: each successive change will get faster, giving admins a
chance to abort early on if they need to run the migration later. The
idea is that this does not force them to try to time interruptions
between smaller migrations.

* Fix column sorting in IdsToBigints

Not a significant change, but it impacts the order of columns in the
database and db/schema.rb.

* Actually pause before IdsToBigints
2017-10-02 21:28:59 +02:00