Fix bug where it was possible to submit blank text in various text fields.
Caused by `String#blank?` not considering certain Unicode characters as blank. `blank?` is defined
as `match?(/\A[[:space:]]*\z/)`, where `[[:space:]]` matches ASCII spaces (space, tab, newline, etc)
and Unicode characters in the Space category ([1]). However, there are other space-like characters
not in the Space category. This includes U+200B (Zero-Width Space), and many more.
It turns out the "Default ignorable code points" [2][3] are what we're after. These are the set of 400
or so formatting and control characters that are invisible when displayed.
Note that there are other control characters that aren't invisible when rendered, instead they're
shown with a placeholder glyph. These include the ASCII C0 and C1 control codes [4], certain Unicode
control characters [5], and unassigned, reserved, and private use codepoints.
There is one outlier: the Braille pattern blank (U+2800) [6]. This character is visually blank, but is
not considered to be a space or an ignorable code point.
[1]: https://codepoints.net/search?gc[]=Z
[2]: https://codepoints.net/search?DI=1
[3]: https://www.unicode.org/review/pr-5.html
[4]: https://codepoints.net/search?gc[]=Cc
[5]: https://codepoints.net/search?gc[]=Cf
[6]: https://codepoints.net/U+2800
[7]: https://en.wikipedia.org/wiki/Whitespace_character
[8]: https://character.construction/blanks
[9]: https://invisible-characters.com
Fix the test suite failing when trying to run it in the default state
with no config file or API keys configured. Most source sites require
API keys or login credentials to be set in order to work. Skip these
tests when credentials aren't configured.
When a dmail is hard deleted, update the user's unread dmail count and
delete any associated mod reports.
Dmails aren't normally hard deleted, except when there's a large spam
wave that necessitates manual intervention.
* Move emails from users table to email_addresses table.
* Validate that addresses are formatted correctly and are unique across
users. Existing invalid emails are grandfathered in.
* Add is_verified flag (the address has been confirmed by the user).
* Add is_deliverable flag (an undeliverable address is an address that bounces).
* Normalize addresses to prevent registering multiple accounts with the
same email address (using tricks like Gmail's plus addressing).
Set the sender name and IP addresses explicitly in the controller rather
than implicitly in the model.
Fixes cases where automated dmails from DanbooruBot had their IP
addresses set to the person who triggered the dmail, even though they
didn't actually send the dmail themselves.
* Automatically generate a mod report when a comment, forum post, or
dmail is detected as spam.
* Automatically ban users that receive too many automatic spam reports
within a short window of time.
* Automatically mark spam dmails as deleted.
* Change ban threshold from 10 spam reports in 24 hours to 10 reports in 1 hour.
* Change ban length from 3 days to forever.
* Add ability to mark dmails as unread.
* Fix users.unread_dmail_count to not count deleted dmails.
* Fix show action so that API calls don't mark dmails as read.
* Don't show the unread dmail notice on the /dmails page itself.
* Stop using users.has_mail flag.
Few people used dmail filters (~900 users in 5 years) and even fewer
used them correctly. Most people used them to try to block dmail spam,
but usually they either blocked too much (by adding common words that
are present in nearly all dmails, causing all mails to them to be
filtered) or too little (blocking specific email addresses or urls,
which usually are never seen again after the spammer is banned).
Nowadays the spam detection system does a better job of filtering spam.
Bug: sending a dmail containing a wiki link (ex: [[tagme]]) failed when
the recipient had email notifications turned on.
Cause: wiki links inside email notifications use absolute urls, which
the dtext postprocessor didn't parse correctly.
Bug: sending dmails failed for members.
Cause: using lambdas with `rakismet_attrs` failed because unexpected
arguments are passed to the lambdas. Using procs works because the
arguments are ignored.
Also fix the tests to actually test akismet. We didn't catch this
because the tests mocked out the `spam?` call.
* Allow using ApplicationRecord#attribute_matches to search text attributes,
and standardize models on using this instead of duplicating code.
* Remove restrictions that limited wildcard searches to Builders only in various places.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.
misc models: convert to strong params.
artist commentaries: convert to strong params.
* Disallow changing or setting post_id to a nonexistent post.
artists: convert to strong params.
* Disallow setting `is_banned` in create/update actions. Changing it
this way instead of with the ban/unban actions would leave the artist in
a partially banned state.
bans: convert to strong params.
* Disallow changing the user_id after the ban has been created.
comments: convert to strong params.
favorite groups: convert to strong params.
news updates: convert to strong params.
post appeals: convert to strong params.
post flags: convert to strong params.
* Disallow users from setting the `is_deleted` / `is_resolved` flags.
ip bans: convert to strong params.
user feedbacks: convert to strong params.
* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.
notes: convert to strong params.
wiki pages: convert to strong params.
* Also fix non-Builders being able to delete wiki pages.
saved searches: convert to strong params.
pools: convert to strong params.
* Disallow setting `post_count` or `is_deleted` in create/update actions.
janitor trials: convert to strong params.
post disapprovals: convert to strong params.
* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
of Approve button.
dmail filters: convert to strong params.
password resets: convert to strong params.
user name change requests: convert to strong params.
posts: convert to strong params.
users: convert to strong params.
* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
has_mail, and dmail_filter_attributes[user_id].
* Remove initialize_default_image_size (dead code).
uploads: convert to strong params.
* Remove `initialize_status` because status already defaults to pending
in the database.
tag aliases/implications: convert to strong params.
tags: convert to strong params.
forum posts: convert to strong params.
* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).
forum topics: convert to strong params.
* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references
Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>
add diffs
Each dmail creates two copies, one for the sender and one for the
receiver. Only spam check the receiver's copy.
Prevents senders from being able to tell when their messages are being
spam filtered.
Bug introduced in 1400f64; that commit changed dmails so that
creator_ip_addr defaulted to CurrentUser.ip_addr like this:
after_initialize :initialize_attributes, if: :new_record?
def initialize_attributes
self.from_id ||= CurrentUser.id
self.creator_ip_addr ||= CurrentUser.ip_addr
end
...but creator_ip_addr already defaulted to 127.0.0.1 from the database,
so the ||= assignment didn't work. Remove the database default so we
always default to CurrentUser.ip_addr.
There's not much sense in saving copies of everything DanbooruBot sends
in DanbooruBot's inbox. They probably won't be checked so it just bloats
the dmails table.
