Commit Graph

12009 Commits

Author SHA1 Message Date
evazion
bc169fc98c posts: fix exception in random:1 filetype:png. 2022-11-15 19:33:25 -06:00
evazion
e935f01358 uploads: fix temp files not being cleaned up quickly enough.
Fix temp files generated during the upload process not being cleaned up quickly enough. This included
downloaded files, generated preview images, and Ugoira video conversions.

Before we relied on `Tempfile` cleaning up files automatically. But this only happened when the
Tempfile object was garbage collected, which could take a long time. In the meantime we could have
hundreds of megabytes of temp files hanging around.

The fix is to explicitly close temp files when we're done with them. But the standard `Tempfile`
class doesn't immediately delete the file when it's closed. So we also have to introduce a
Danbooru::Tempfile wrapper that deletes the tempfile as soon as it's closed.
2022-11-15 18:50:50 -06:00
evazion
21a779455f discord: fix /tagme command to work with html page URLs. 2022-11-15 15:22:56 -06:00
evazion
aa21f002be tests: fix cleanup of upload temp files.
Fix upload temp files sometimes not being cleaned up when tests exit.
2022-11-15 15:16:50 -06:00
evazion
53f40b3331 gems: add back pry gem.
The `pry` gem was removed in e698bf91 because we replaced `pry-byebug` with the standard debugger.
Add it back because `pry` is better than `irb` and we still can use the standard debugger with it.
2022-11-15 14:18:22 -06:00
evazion
f128c48485 archives: add code for detecting .rar and .7z files. 2022-11-14 20:14:37 -06:00
evazion
5f92f452fe media file: factor out file type detection code.
Factor out the file type detection code from MediaFile into a FileTypeDetector class so we can use
it to detect archive files (.zip, .rar, .7z) too.
2022-11-14 20:14:37 -06:00
evazion
0c1e9a1618 Add Danbooru::Archive library for handling .zip and .rar files.
Introduce a new Danbooru::Archive library. This is a wrapper around libarchive that lets us extract
.zip, .rar, .7z, and other archive formats. Replace the rubyzip library in MediaFile::Ugoira with
the new Danbooru::Archive library.

This is a step towards fixing #5340: Add support for extracting archive attachments from certain sources.

This adds a new dependency on libarchive. Downstream users should `apt-get install libarchive13` if
they're not using Docker.

https://github.com/chef/ffi-libarchive
https://github.com/libarchive/libarchive
https://www.rubydoc.info/gems/ffi-libarchive/0.4.2
https://github.com/libarchive/libarchive/wiki/Examples#a-complete-extractor
2022-11-14 20:14:37 -06:00
evazion
f942768ce8 Fix #5347: Don't use exception template for post validation errors
Also fixes #5173: Parenting a post to itself using the "parent id" box leads to an error.
2022-11-13 02:01:18 -06:00
evazion
33e4cbeb8d Fix #5179: Add wiki page versions to wiki page secondary links plus standardization
* Call it "Changes" when referring to the global list of changes (pool changes, artist changes, etc).
* Call it "History" when referring to the history of a single item (pool history, artist history, etc).
* Put the artist history search form on the /artist_versions page, and remove the /artist_versions/search endpoint.
2022-11-13 00:58:11 -06:00
evazion
d053dc7356 Fix #5143: Allow name changes with same name but changed case. 2022-11-12 23:04:45 -06:00
evazion
22f232f4e9 Fix #5209: dtext link search doesn't work right. 2022-11-12 22:18:11 -06:00
evazion
d7005c6b3d Fix #5195: Export ArtistCommentary Javascript Functions. 2022-11-12 22:11:04 -06:00
evazion
e12ce9b472 Fix #5343: Add icons for websites linked to at least 9 artists. 2022-11-12 19:49:20 -06:00
evazion
220db642e1 Fix #5346: Can add tags beginning with newpool:, causing the next edit to add the post to a pool. 2022-11-12 17:01:52 -06:00
evazion
86669fa605 Fix #5344: populate.rb creates users with invalid names. 2022-11-12 16:44:30 -06:00
evazion
d5676bf0fd Merge pull request #5345 from nottalulah/patch-1
wiki pages: don't show create links to anonymous users on non-existent pages
2022-11-12 16:41:27 -06:00
Lily
32fbc4e65f wiki pages: don't show create links to anonymous users on non-existent pages 2022-11-12 09:21:20 -04:00
evazion
a167091bf9 Danbooru::Http: fix bug when using proxy option.
Fix external HTTP requests not working when the HTTP proxy was enabled. Caused by the `public_only`
option (which prevents SSRF attacks by validating that the URL doesn't resolve to a local IP) being
incompatible with the `proxy` option.
2022-11-12 02:44:05 -06:00
evazion
215df49050 config: rework http proxy options.
Replace the http_proxy_host, http_proxy_port, http_proxy_username, http_proxy_password options with
a single `Danbooru.config.http_proxy` option.
2022-11-12 02:44:01 -06:00
evazion
4c3b988d15 sources: fix site icons. 2022-11-11 14:03:06 -06:00
evazion
81cba9b6f3 posts: autodetect more AI-generated images. 2022-11-11 14:03:06 -06:00
nonamethanks
d525df9ad5 Newgrounds: fix exception for deleted videos
Fixup for 8c0f2255f9
2022-11-11 12:28:23 +01:00
nonamethanks
8c0f2255f9 Newgrounds: fix support for some old videos 2022-11-11 11:01:13 +01:00
evazion
0fd95375fa sources: add more site icons.
Add site icons for the following sites:

* 4chan
* Catbox
* Danbooru
* Drawcrowd
* E-Hentai
* Enty
* Gelbooru
* Hitomi
* Imgur
* Joyreactor
* Kemono Party
* Konachan
* Luscious.net
* Mega.nz
* Monappy
* Mihoyo
* Overdoll
* Pinterest
* Rule34.us
* Sankaku Complex
* Steam
* Webmshare
* Yande.re
* Zerochan

These are mainly used for source links on the https://danbooru.donmai.us/post_events?search[category]=Replacement page.
2022-11-11 00:54:50 -06:00
evazion
0a792fbb8a replacements: link to old and new media assets.
On the /post_events and /post_replacements pages, include links to the old and new media assets
pages so you can compare the two images.
2022-11-11 00:54:10 -06:00
evazion
6f61abc6a7 users: don't log mod action for user deletions.
Don't log a mod action when a user deletes their own account. This isn't a moderator action, so it
doesn't belong here. Account deletions are still logged on the /user_events page (visible to mods only).

A mod action is still logged when the Owner-level user deletes someone else's account.
2022-11-10 14:30:23 -06:00
evazion
15f611ca8b Add Rule34.us support. 2022-11-10 12:53:06 -06:00
evazion
ec87cffe02 Add Rule34.xxx support. 2022-11-10 00:39:52 -06:00
evazion
0bad53e6ac Add TBIB.org support. 2022-11-09 16:49:20 -06:00
evazion
5b4856273f Add safebooru.org support.
Refactor the Gelbooru source extractor to support Safebooru.org as well.
2022-11-09 15:44:04 -06:00
nonamethanks
35bfcbc3bd Newgrounds: support video uploads 2022-11-09 15:01:28 +01:00
nonamethanks
d7d35cb16f Fanbox: add support for embedded videos
Fixes #5339.
2022-11-09 14:02:48 +01:00
evazion
53d4052387 Merge pull request #5329 from nonamethanks/feat-bilibili
Add bilibili support
2022-11-09 01:17:18 -06:00
evazion
df241028ef replacements: make replacements searchable by media asset.
In particular, allow these searches to find replacements without a media asset:

* https://danbooru.donmai.us/post_replacements?search[has_media_asset]=false
* https://danbooru.donmai.us/post_replacements?search[has_old_media_asset]=false
2022-11-09 01:08:47 -06:00
evazion
80b3e34bd1 replacements: initialize media_asset_id, old_media_asset_id columns. 2022-11-09 00:22:17 -06:00
evazion
b472ae5c87 replacements: add media_asset_id, old_media_asset_id columns.
Add media_asset_id and old_media_asset_id columns for associating replacements with media assets.
This way we can easily tell which replacements don't have a media asset (with the md5 alone we can't
tell whether the media asset actually exists).
2022-11-08 23:41:49 -06:00
evazion
83d14a281f replacements: backfill images in parallel. 2022-11-08 21:41:59 -06:00
evazion
153d5f3e67 moebooru: support md5-based post URLs.
Support URLs like these:

* https://yande.re/post/show?md5=2c95b8975b73744da2bcbed9619c1d59
* https://konachan.com/post/show?md5=955aa45f3b452b415509b47dcc9475ac

This makes scraping images from these sites by md5 easier.
2022-11-08 21:40:16 -06:00
evazion
09f1ace357 replacements: add fix script to backfill old images from Gelbooru.
Add a fix script to download images from Gelbooru for old replacements where we deleted the original
image. For archival purposes, we want to try to find the original file for every replacement.

These images will be uploaded as unposted assets under DanbooruBot's name.
2022-11-08 15:45:57 -06:00
evazion
03a4f1a46e gelbooru: fix exception when fetching data for deleted post.
Fix exceptions when fetching deleted or nonexistent posts from Gelbooru. The Gelbooru API doesn't
return any data for deleted posts.
2022-11-08 15:34:44 -06:00
evazion
00db63e885 Fix #5336: Nuke old danboorubot replacement comments
Add a fix script that imports the md5 for old post replacements from the corresponding DanbooruBot
replacement comment, then deletes all replacement comments.

There are about 250 replacements left that still have a null md5 because they don't have a matching
comment. This is because if a post was replaced but the file didn't change, it didn't leave a comment.
2022-11-08 02:26:50 -06:00
evazion
a2690dcb4f db: add users.is_deleted to structure.sql.
Forgotten in f083f29c3.
2022-11-07 00:09:37 -06:00
evazion
174c8e0067 Fix #5335: Queries with "ordfav:<username>" and geometry attributes (e.g. "ratio:", "height:") crashes the api/site.
Fix `Relation passed to #and must be structurally compatible. Incompatible values: [:joins] (ArgumentError)`
exception in `ordfav:evazion ratio:4:3` search. Broken by e849d8f1c.

We were effectively doing this:

    q1 = Post.joins(:favorites, :media_asset).where("favorites.user_id = ?", 52664).order("favorites.id DESC")
    q2 = Post.joins(:media_asset, :favorites).where("ROUND(media_assets.image_width::numeric / media_assets.image_height::numeric, 2) = 1.33")
    q3 = q1.and(q2)

This failed because Rails didn't like the fact that the joins were in a different order when the
queries were `and`-ed together.
2022-11-06 21:13:48 -06:00
evazion
c133866cb7 users: don't allow users to choose reserved names.
Don't allow users to choose names that conflict with search syntax, like `any` or `none`, or names
that impersonate user levels, like `Admin`, `Moderator`, `Anonymous`, etc.
2022-11-06 16:00:07 -06:00
evazion
8bd60e41a1 Fix #4555: Invalidate sessions for deleted users
Fix three exploits that allowed one to keep using their account after it was deleted:

* It was possible to use session cookies from another computer to login after you deleted your account.
* It was possible to use API keys to make API requests after you deleted your account.
* It was possible to request a password reset, delete your account, then use the password reset link
  to change your password and login to your deleted account.
2022-11-06 14:58:08 -06:00
evazion
6f08e1427b users: set is_deleted flag when account is deleted.
* Set the `is_deleted` flag when the user is deleted.
* Return the `is_deleted` flag in the /users.json API.
2022-11-06 13:18:49 -06:00
nonamethanks
0a78a6188d Reddit: support user galleries and embedded images 2022-11-06 15:04:05 +01:00
evazion
f083f29c3b users: add is_deleted flag.
Add is_deleted flag to users table in preparation for fixing #4555.
2022-11-06 01:41:14 -05:00
evazion
b43a913ad7 users: delete more data when user deactivates their account.
* Don't delete the user's favorites unless private favorites are enabled. The general rule is that
  public account activity is kept and private account activity is deleted.
* Delete the user's API keys, forum topics visits, private favgroups, downvotes, and upvotes (if
  privacy is enabled).
* Reset all of the user's account settings to default. This means custom CSS is deleted, where it
  wasn't before.
* Delete everything but the user's name and password asynchronously.
* Don't log the current user out if it's the owner deleting another user's account.
* Fix #5067 (Mod actions sometimes not created for user deletions) by wrapping the deletion process
  in a transaction.
2022-11-06 00:05:18 -05:00