posts: normalize Unicode to NFC form in post sources.

Fix strings like "pokémon" (NFD form) and "pokémon" (NFC form) being considered different strings in sources. Also add a fix script to fix existing sources. There were only 15 posts with unnormalized sources.
2022-01-31 10:56:27 -06:00
parent 0132c5f0a5
commit 61c043c6b1
4 changed files with 26 additions and 33 deletions
--- a/test/unit/post_test.rb
+++ b/test/unit/post_test.rb
@@ -1340,6 +1340,15 @@ class PostTest < ActiveSupport::TestCase
      end

      context "with a source" do
+        context "that contains unicode characters" do
+          should "normalize the source to NFC form" do
+            source1 = "poke\u0301mon" # pokémon (nfd form)
+            source2 = "pok\u00e9mon"  # pokémon (nfc form)
+            @post.update!(source: source1)
+            assert_equal(source2, @post.source)
+          end
+        end
+
        context "that is not from pixiv" do
          should "clear the pixiv id" do
            @post.pixiv_id = 1234