Commit Graph

10018 Commits

Author SHA1 Message Date
evazion
2deae38a4e uploads: allow uploading .zip, .rar., and .7z files from disk.
Allow uploading .zip, .rar, and .7z files from disk. The archive will be extracted and the images
inside will be uploaded.

This only works for archive files uploaded from disk, not from a source URL.

Post source URLs will look something like this: "file://foo.zip/1.jpg", "file://foo.zip/2.jpg", etc.
Sometimes artists uses Shift JIS or other encodings instead of UTF-8 for filenames. In these cases
we just assume the filename is UTF-8 and replace invalid characters with '?', so filenames might be
wrong in some cases.

There are various protections to prevent uploading malicious archive files:

* Archives with more than 100 files aren't allowed.
* Archives that decompress to more than 100MB aren't allowed.
* Archives with filenames containing '..' components aren't allowed (e.g. '../../../../../etc/passwd').
* Archives with filenames containing absolute paths aren't allowed (e.g. '/etc/passwd').
* Archives containing symlinks aren't allowed (e.g. 'foo -> /etc/passwd').
* Archive types other than .zip, .rar, and .7z aren't allowed (e.g. .tar.gz, .cpio).
* File permissions, owners, and other metadata are ignored.

Partial fix for #5340: Add support for extracting archive attachments from certain sources
2022-11-16 16:47:37 -06:00
evazion
d791924aad posts: fix /posts/random?tags=... redirect.
Fix /posts/random?tags=touhou to redirect to /posts/1234?q=touhou instead of /posts/1234?tags=touhou.

Fixes the search bar on random posts not containing the current search.
2022-11-15 19:38:43 -06:00
evazion
bc169fc98c posts: fix exception in random:1 filetype:png. 2022-11-15 19:33:25 -06:00
evazion
e935f01358 uploads: fix temp files not being cleaned up quickly enough.
Fix temp files generated during the upload process not being cleaned up quickly enough. This included
downloaded files, generated preview images, and Ugoira video conversions.

Before we relied on `Tempfile` cleaning up files automatically. But this only happened when the
Tempfile object was garbage collected, which could take a long time. In the meantime we could have
hundreds of megabytes of temp files hanging around.

The fix is to explicitly close temp files when we're done with them. But the standard `Tempfile`
class doesn't immediately delete the file when it's closed. So we also have to introduce a
Danbooru::Tempfile wrapper that deletes the tempfile as soon as it's closed.
2022-11-15 18:50:50 -06:00
evazion
21a779455f discord: fix /tagme command to work with html page URLs. 2022-11-15 15:22:56 -06:00
evazion
f128c48485 archives: add code for detecting .rar and .7z files. 2022-11-14 20:14:37 -06:00
evazion
5f92f452fe media file: factor out file type detection code.
Factor out the file type detection code from MediaFile into a FileTypeDetector class so we can use
it to detect archive files (.zip, .rar, .7z) too.
2022-11-14 20:14:37 -06:00
evazion
0c1e9a1618 Add Danbooru::Archive library for handling .zip and .rar files.
Introduce a new Danbooru::Archive library. This is a wrapper around libarchive that lets us extract
.zip, .rar, .7z, and other archive formats. Replace the rubyzip library in MediaFile::Ugoira with
the new Danbooru::Archive library.

This is a step towards fixing #5340: Add support for extracting archive attachments from certain sources.

This adds a new dependency on libarchive. Downstream users should `apt-get install libarchive13` if
they're not using Docker.

https://github.com/chef/ffi-libarchive
https://github.com/libarchive/libarchive
https://www.rubydoc.info/gems/ffi-libarchive/0.4.2
https://github.com/libarchive/libarchive/wiki/Examples#a-complete-extractor
2022-11-14 20:14:37 -06:00
evazion
f942768ce8 Fix #5347: Don't use exception template for post validation errors
Also fixes #5173: Parenting a post to itself using the "parent id" box leads to an error.
2022-11-13 02:01:18 -06:00
evazion
33e4cbeb8d Fix #5179: Add wiki page versions to wiki page secondary links plus standardization
* Call it "Changes" when referring to the global list of changes (pool changes, artist changes, etc).
* Call it "History" when referring to the history of a single item (pool history, artist history, etc).
* Put the artist history search form on the /artist_versions page, and remove the /artist_versions/search endpoint.
2022-11-13 00:58:11 -06:00
evazion
d053dc7356 Fix #5143: Allow name changes with same name but changed case. 2022-11-12 23:04:45 -06:00
evazion
22f232f4e9 Fix #5209: dtext link search doesn't work right. 2022-11-12 22:18:11 -06:00
evazion
d7005c6b3d Fix #5195: Export ArtistCommentary Javascript Functions. 2022-11-12 22:11:04 -06:00
evazion
e12ce9b472 Fix #5343: Add icons for websites linked to at least 9 artists. 2022-11-12 19:49:20 -06:00
evazion
220db642e1 Fix #5346: Can add tags beginning with newpool:, causing the next edit to add the post to a pool. 2022-11-12 17:01:52 -06:00
Lily
32fbc4e65f wiki pages: don't show create links to anonymous users on non-existent pages 2022-11-12 09:21:20 -04:00
evazion
a167091bf9 Danbooru::Http: fix bug when using proxy option.
Fix external HTTP requests not working when the HTTP proxy was enabled. Caused by the `public_only`
option (which prevents SSRF attacks by validating that the URL doesn't resolve to a local IP) being
incompatible with the `proxy` option.
2022-11-12 02:44:05 -06:00
evazion
215df49050 config: rework http proxy options.
Replace the http_proxy_host, http_proxy_port, http_proxy_username, http_proxy_password options with
a single `Danbooru.config.http_proxy` option.
2022-11-12 02:44:01 -06:00
evazion
4c3b988d15 sources: fix site icons. 2022-11-11 14:03:06 -06:00
evazion
81cba9b6f3 posts: autodetect more AI-generated images. 2022-11-11 14:03:06 -06:00
nonamethanks
d525df9ad5 Newgrounds: fix exception for deleted videos
Fixup for 8c0f2255f9
2022-11-11 12:28:23 +01:00
nonamethanks
8c0f2255f9 Newgrounds: fix support for some old videos 2022-11-11 11:01:13 +01:00
evazion
0fd95375fa sources: add more site icons.
Add site icons for the following sites:

* 4chan
* Catbox
* Danbooru
* Drawcrowd
* E-Hentai
* Enty
* Gelbooru
* Hitomi
* Imgur
* Joyreactor
* Kemono Party
* Konachan
* Luscious.net
* Mega.nz
* Monappy
* Mihoyo
* Overdoll
* Pinterest
* Rule34.us
* Sankaku Complex
* Steam
* Webmshare
* Yande.re
* Zerochan

These are mainly used for source links on the https://danbooru.donmai.us/post_events?search[category]=Replacement page.
2022-11-11 00:54:50 -06:00
evazion
0a792fbb8a replacements: link to old and new media assets.
On the /post_events and /post_replacements pages, include links to the old and new media assets
pages so you can compare the two images.
2022-11-11 00:54:10 -06:00
evazion
6f61abc6a7 users: don't log mod action for user deletions.
Don't log a mod action when a user deletes their own account. This isn't a moderator action, so it
doesn't belong here. Account deletions are still logged on the /user_events page (visible to mods only).

A mod action is still logged when the Owner-level user deletes someone else's account.
2022-11-10 14:30:23 -06:00
evazion
15f611ca8b Add Rule34.us support. 2022-11-10 12:53:06 -06:00
evazion
ec87cffe02 Add Rule34.xxx support. 2022-11-10 00:39:52 -06:00
evazion
0bad53e6ac Add TBIB.org support. 2022-11-09 16:49:20 -06:00
evazion
5b4856273f Add safebooru.org support.
Refactor the Gelbooru source extractor to support Safebooru.org as well.
2022-11-09 15:44:04 -06:00
nonamethanks
35bfcbc3bd Newgrounds: support video uploads 2022-11-09 15:01:28 +01:00
nonamethanks
d7d35cb16f Fanbox: add support for embedded videos
Fixes #5339.
2022-11-09 14:02:48 +01:00
evazion
53d4052387 Merge pull request #5329 from nonamethanks/feat-bilibili
Add bilibili support
2022-11-09 01:17:18 -06:00
evazion
df241028ef replacements: make replacements searchable by media asset.
In particular, allow these searches to find replacements without a media asset:

* https://danbooru.donmai.us/post_replacements?search[has_media_asset]=false
* https://danbooru.donmai.us/post_replacements?search[has_old_media_asset]=false
2022-11-09 01:08:47 -06:00
evazion
80b3e34bd1 replacements: initialize media_asset_id, old_media_asset_id columns. 2022-11-09 00:22:17 -06:00
evazion
153d5f3e67 moebooru: support md5-based post URLs.
Support URLs like these:

* https://yande.re/post/show?md5=2c95b8975b73744da2bcbed9619c1d59
* https://konachan.com/post/show?md5=955aa45f3b452b415509b47dcc9475ac

This makes scraping images from these sites by md5 easier.
2022-11-08 21:40:16 -06:00
evazion
03a4f1a46e gelbooru: fix exception when fetching data for deleted post.
Fix exceptions when fetching deleted or nonexistent posts from Gelbooru. The Gelbooru API doesn't
return any data for deleted posts.
2022-11-08 15:34:44 -06:00
evazion
174c8e0067 Fix #5335: Queries with "ordfav:<username>" and geometry attributes (e.g. "ratio:", "height:") crashes the api/site.
Fix `Relation passed to #and must be structurally compatible. Incompatible values: [:joins] (ArgumentError)`
exception in `ordfav:evazion ratio:4:3` search. Broken by e849d8f1c.

We were effectively doing this:

    q1 = Post.joins(:favorites, :media_asset).where("favorites.user_id = ?", 52664).order("favorites.id DESC")
    q2 = Post.joins(:media_asset, :favorites).where("ROUND(media_assets.image_width::numeric / media_assets.image_height::numeric, 2) = 1.33")
    q3 = q1.and(q2)

This failed because Rails didn't like the fact that the joins were in a different order when the
queries were `and`-ed together.
2022-11-06 21:13:48 -06:00
evazion
c133866cb7 users: don't allow users to choose reserved names.
Don't allow users to choose names that conflict with search syntax, like `any` or `none`, or names
that impersonate user levels, like `Admin`, `Moderator`, `Anonymous`, etc.
2022-11-06 16:00:07 -06:00
evazion
8bd60e41a1 Fix #4555: Invalidate sessions for deleted users
Fix three exploits that allowed one to keep using their account after it was deleted:

* It was possible to use session cookies from another computer to login after you deleted your account.
* It was possible to use API keys to make API requests after you deleted your account.
* It was possible to request a password reset, delete your account, then use the password reset link
  to change your password and login to your deleted account.
2022-11-06 14:58:08 -06:00
evazion
6f08e1427b users: set is_deleted flag when account is deleted.
* Set the `is_deleted` flag when the user is deleted.
* Return the `is_deleted` flag in the /users.json API.
2022-11-06 13:18:49 -06:00
nonamethanks
0a78a6188d Reddit: support user galleries and embedded images 2022-11-06 15:04:05 +01:00
evazion
b43a913ad7 users: delete more data when user deactivates their account.
* Don't delete the user's favorites unless private favorites are enabled. The general rule is that
  public account activity is kept and private account activity is deleted.
* Delete the user's API keys, forum topics visits, private favgroups, downvotes, and upvotes (if
  privacy is enabled).
* Reset all of the user's account settings to default. This means custom CSS is deleted, where it
  wasn't before.
* Delete everything but the user's name and password asynchronously.
* Don't log the current user out if it's the owner deleting another user's account.
* Fix #5067 (Mod actions sometimes not created for user deletions) by wrapping the deletion process
  in a transaction.
2022-11-06 00:05:18 -05:00
evazion
3ffde5b23d users: move account deletion endpoint to /users/:id/deactivate.
Move the account deletion endpoint from /maintenance/users/deletion to either:

* https://danbooru.donmai.us/users/deactivate
* https://danbooru.donmai.us/users/:id/deactivate

This incidentally allows the Owner-level user to deactivate accounts belonging to other users. This
is meant for things like deactivating inactive accounts with invalid or abusive names. This is
limited to accounts below Gold level for security.
2022-11-05 19:09:56 -05:00
evazion
59872d2ed5 comments: fix N+1 query when loading comment votes.
Doing both `@comments.includes(:votes)` and `comment.votes.active` forced votes to be loaded twice.
2022-11-05 19:09:56 -05:00
evazion
21a3763f0f users: set archives DB timeout on profile pages.
Fixes profile pages potentially hanging for 60+ seconds if the archives
database is slow to respond.
2022-11-05 19:09:56 -05:00
evazion
af9e570419 skeb: parse www.skeb.jp URLs. 2022-11-05 19:09:56 -05:00
evazion
ab08ad24e6 baraag: parse https://baraag.net/web/@{name} URLs. 2022-11-05 19:09:56 -05:00
evazion
fcb68048e3 fanbox: parse https://pixiv.net/fanbox URLs. 2022-11-05 19:09:56 -05:00
evazion
f5b003fc94 furaffinity: parse https://www.furaffinity.net/stats/{name} URLs. 2022-11-05 19:09:56 -05:00
nonamethanks
9f3f302328 Lofter: support another theme
Fixes #5334.
2022-11-06 00:03:15 +01:00