Commit Graph

16 Commits

Author SHA1 Message Date
evazion
fefa6036fb tests: fix broken upload tests.
* Fix broken Skeb test caused by 404'd image.
* Fix broken Sta.sh tests caused by DeviantArt URL changes.
* Fix broken Nijie tests caused by Nijie URL changes.
2022-02-15 20:33:52 -06:00
evazion
892c3899a8 uploads: fix more tests. 2022-01-29 23:23:29 -06:00
evazion
40d6351249 Fix #4940: DanbooruBot retiring banned_artist implications. 2022-01-11 11:53:35 -06:00
evazion
a5f589f9e0 aliases/implications: change automatic retirement rules.
Change the rules for automatically retiring aliases and implications:

* Retire aliases to tags that are empty, or that are for a general or
  artist tag that hasn't received any new posts in the last two years.
* Retire implications from tags that are empty.
* Don't retire aliases or implications for character, copyright, or
  meta tags any more, unless the tags are empty.
2021-11-05 05:46:50 -05:00
evazion
5e8c91700c tests: fix amcheck job tests. 2021-10-13 04:19:44 -05:00
evazion
1653392361 posts: stop updating fav_string attribute.
Stop updating the fav_string attribute on posts. The column still exists
on the table, but is no longer used or updated.

Like the pool_string in 7d503f08, the fav_string was used in the past to
facilitate `fav:X` searches. Posts had a hidden fav_string column that
contained a list of every user who favorited the post. These were
treated like fake hidden tags on the post so that a search for `fav:X`
was treated like a tag search.

The fav_string attribute has been unused for search purposes for a while
now. It was only kept because of technicalities that required
departitioning the favorites table first (340e1008e) before it could be
removed. Basically, removing favorites with `@favorite.destroy` was
slow because Rails always deletes object by ID, but we didn't have an
index on favorites.id, and we couldn't easily add one until the
favorites table was departitioned.

Fixes #4652. See https://github.com/danbooru/danbooru/issues/4652#issuecomment-754993802
for more discussion of issues caused by the fav_string (in short: write
amplification, post table bloat, and favorite inconsistency problems).
2021-10-09 22:36:26 -05:00
evazion
950bc608c2 maintenance: add job to check for database corruption.
Add a job to run pg_amcheck hourly to check for corrupt database indexes.

https://www.postgresql.org/docs/14/app-pgamcheck.html
2021-10-06 08:08:52 -05:00
evazion
52bf4a3a6b maintenance: break maintenance tasks into individual jobs.
Break the hourly/daily/weekly/monthly maintenance tasks down into
individual delayed jobs. This way if one task fails, it won't prevent
other tasks from running. Also, jobs can be run in parallel, and can be
individually retried if they fail.
2021-09-26 20:38:30 -05:00
evazion
9ba84efc07 BURs: process BURs sequentially in a single job.
Change the way BURs are processed. Before, we spawned a background job
for each line of the BUR, then processed each job sequentially. Now, we
process the entire BUR sequentially in a single background job.

This means that:

* BURs are truly sequential now. Before certain things like removing
  aliases weren't actually performed in a background job, so they were
  performed out-of-order before everything else in the BUR.

* Before, if an alias or implication line failed, then subsequent alias
  or implication lines would still be processed. This was because each
  alias or implication line was queued as a separate job, so a failure
  of one job didn't block another. Now, if any alias or implication
  fails, the entire BUR will fail and stop processing after that line.
  This may be good or bad, depending on whether we actually need the BUR
  to be processed in order or not.

* Before, BURs were processed inside a database transaction (except for the
  actual updating of posts). Now they're not. This is because we can't
  afford to hold transactions open while processing long-running aliases
  or implications. This means that if BUR fails in the middle when it is
  initially approved, it will be left in a half-complete state. Before
  it would be rolled back and left in a pending state with no changes
  performed.

* Before, only one BUR at a time could be processed. If multiple BURs
  were approved at the same time, then they would queue up and be
  processed one at a time. Now, multiple BURs can be processed at the
  same time. This may be undesirable when processing large BURs, or BURs
  that must be approved in a specific order.

* Before, large tag category changes could time out. This was because
  they weren't actually performed in a background job. Now they are, so
  they shouldn't time out.
2021-09-20 01:12:14 -05:00
evazion
9e37f5a588 BURs: don't log mod actions for aliases/implications/mass updates.
Don't log mod actions when aliases, implications, or mass updates are
processed.

Originally aliases and implications were logged because they could be
approved outside of a BUR. Mass updates could also be performed by mods
without making a forum request. This is no longer the case.

They were also logged for debugging purposes. This is no longer needed.
This generated a lot of spam in the mod action logs when a large BUR was
approved.
2020-12-02 12:20:28 -06:00
evazion
23944a1794 Fix #4491: Have tag rename option for bulk update requests.
* Add a `rename A -> B` command for bulk update requests.
* Change mass updates to only retag the posts, not to move saved
  searches or blacklists.

A tag rename does the same thing an alias does, except it doesn't
create a permanent alias. More precisely, a tag rename:

* Moves the wiki.
* Moves the artist entry.
* Moves saved searches.
* Moves blacklists.
* Merges the wikis, if both tags have wiki pages.
* Merges the artist entries, if both tags have artist pages.
* Fixes links in wiki pages to point to the new tag.
* Retags the posts.
2020-08-26 19:53:04 -05:00
evazion
5817af4014 burs/show: remove BUR update count estimate.
Remove the post update count estimate from BUR show pages. This was
complex, slow, and usually inaccurate since it assumed that requests in
a BUR had no overlap with each other, which usually wasn't the case.
2020-02-16 19:21:56 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
dab43d96c9 jobs: migrate mass updates to ActiveJob.
Also fixes a bug where mod actions weren't logged on mass updates.
Creating the mod action silently failed because it was called when
CurrentUser wasn' set.
2019-08-19 00:46:31 -05:00
evazion
4e8b3d9310 jobs: migrate user deletions to ActiveJob. 2019-08-16 20:49:34 -05:00
evazion
822dce20ed jobs: add base ApplicationJob. 2019-08-16 20:49:34 -05:00