* Have CI build Docker images for both x86 and ARM.
* Add a `bin/rails danbooru:docker:build-arm` command for building a Docker image locally for ARM.
Usage:
* Test the image:
docker run --rm -it --platform linux/arm64 ghcr.io/danbooru/danbooru bash
* Build the image:
bin/rails danbooru:docker:build-arm
* Build the image by hand:
git archive HEAD | docker buildx build - --platform linux/amd64 --build-arg SOURCE_COMMIT=$(git rev-parse HEAD) -t danbooru -f Dockerfile --load
Support grabbing the full image for Tinami uploads, rather than the sample.
Getting the full image requires making a request like this:
curl -X POST \
-H 'Referer: https://www.tinami.com/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Cookie: Tinami2SESSID=<redacted>;' \
--data-raw 'action_view_original=true&cont_id=1087268ðna_csrf=<redacted>' \
https://www.tinami.com/view/1087268
Then scraping the <img> tag from the resulting HTML page.
If the post has multiple images, then we need to scrape and pass the
`sub_id` of the image too.
Fixes#2818.
NicoSeiga changed it so that on every login, you must enter a 2FA code
sent by email. This broke the NicoSeiga strategy. The fix is to just use
a static session cookie instead (and hope it doesn't expire, and isn't
tied to an IP).
The `nico_seiga_login` and `nico_seiga_password` config settings have
been removed from config/danbooru_default_config.rb and replaced by
`nico_seiga_user_session`. If you run your own Danbooru instance, you
will have to update your config file manually.
This reverts commit 9d62f71cd9.
This caused a problem where if you pushed multiple branches or tags at
once, for example to the betabooru and production branches, then the
Docker image would only get built for one branch. This led to deploys
not fetching the latest image.
Just trigger on pushes to the betabooru branch instead of on the Docker
image being built. The workflow_run event keeps triggering on the master
branch even though it's supposed to only trigger on the betabooru
branch. This makes it fail because it doesn't have access to the secrets
in the betabooru environment.
This does mean the deploy will fail if the image hasn't been previously
built by a push to master first.
Make it so that when ARCHIVE_DATABASE_URL isn't set, it defaults to
DATABASE_URL. In other words, if you don't have a separate archive
database configured, then default to using the main database for
post/pool versions.
Fixes an issue where running the test suite would fail if you didn't
explicitly set ARCHIVE_DATABASE_URL because it tried to use
`archive_test` as the post/pool versions database name.
Make it so pull requests from outside contributors can't edit workflows
under .github/workflows/ without approval. Also limit workflows to the
minimum permissions necessary.
Split up the Github workflow. Instead of one workflow with two jobs, one
to build the Docker image and one to test it, split it into two separate
workflows, one to build and one to test. This way if the Docker build
fails it doesn't try to run the tests, and if the tests fail it only
marks the test workflow as failed, not the entire workflow.
This is especially so the workflows page doesn't show everything as
failing just because the tests failed.
https://github.com/danbooru/danbooru/actions
Fix the Docker image not including the git hash of the build. On
DockerHub we had to set it with a build hook, but now with Github we
set it as part of the build-push-action.
Run the tests using the Docker image we just built, instead of
installing everything manually and running the tests outside of Docker.
Ensures the Docker image we built really works.
* Export daily public database dumps to BigQuery and Google Cloud Storage.
* Only data visible to anonymous users is exported. Some tables have
null or missing fields because of this.
* The bans table is excluded because some bans have an expires_at
timestamp set beyond year 9999, which BigQuery doesn't support.
* The favorites table is excluded because it's too slow to dump (it
doesn't have an id index, which is needed by find_each).
* Version tables are excluded because dumping them every day is
inefficient, streaming insertions should be used instead.
Links:
* https://console.cloud.google.com/bigquery?project=danbooru1
* https://console.cloud.google.com/storage/browser/danbooru_public
* https://storage.googleapis.com/danbooru_public/data/posts.json
Fix the Pixiv API no longer working by rewriting the Pixiv strategy to
use the Ajax API instead of the mobile API.
Before we could authenticate in the mobile API by using the OAuth 2.0
grant_type=password authentication flow. This no longer works. Now it
requires logging in through a HTML page, which is protected by Google
reCaptcha. This makes using the mobile API infeasible.
Instead we switch to the Ajax API, which only needs a PHPSESSID to
authenticate. This can be obtained by logging in manually and using the
devtools to extract the cookie.
This also temporarily removes support for Pixiv novels. This should be
moved to a separate source strategy.
Fix tests not working in Github. They were failing because the latest
version of Webpack needs a version of Node newer than the version in
shipped Ubuntu 20.04.
Also fix the Docker build failing because of the system timezone
database not being installed in Ubuntu 20.10.
Comparison:
* Codecov has a simpler integration and a better UI.
* Codeclimate tracks both linter warnings (Rubocop, ESLint) and code
coverage, but its UI for code coverage is worse than Codecov's.
* Codeclimate doesn't support Simplecov 0.18 because Codeclimate doesn't
support 0.18's new coverage format yet.
Fix gem version conflicts described in 20abd8a5f. Nokogiri couldn't be
upgraded past 1.10.9 because 1.11.0 causes a build failure in Nokogumbo
2.0.2, but we couldn't stay on 1.10.9 either because it has a hard
requirement on Ruby <2.7 and we require Ruby >=2.7. This made `bundle
update` fail with a Gemfile conflict.
The fix is to disable libxml2 support when building Nokogumbo. Nokogumbo
wants to use the same version of libxml2 as Nokogiri, but Nokogiri
1.11.0 changed how it reports which version of libxml2 it's using, which
causes Nokogumbo's build to fail. Disabling libxml2 may reduce
performance of Nokogumbo ([1]).
While we're at it, we also make Nokogiri use the system version of
libxml2 instead of its own bundled version. Nokogiri really wants
us to use its own patched version of libxml2 instead of the system
version, but the patches it applies look relatively minor and don't seem
relevant to us ([2]). Using the system version reduces build time during CI.
This adds libxml2 and libxslt as OS-level dependencies of Danbooru. You
may need to do `sudo apt-get install libxml2-dev libxslt-dev` to install
these libraries after this commit.
[1]: https://github.com/rubys/nokogumbo#flavors-of-nokogumbo
[2]: https://github.com/sparklemotion/nokogiri/tree/master/patches/libxml2
