From 31802fb666c1a86987b4f57f37a7a13149f14b07 Mon Sep 17 00:00:00 2001 From: evazion Date: Mon, 22 Jun 2020 22:21:17 -0500 Subject: [PATCH] nijie: fix parallel test failures. Nijie tests fail often under parallel testing. This is because every test needs to login to Nijie first, but Nijie rate-limits the login endpoint, so eventually we hit the limit and tests start failing. This is made worse by a thundering herd problem. Eight test processes try to login to Nijie at the same time, but only one succeeds, so the rest sleep and try again, but they all wakeup and try again at the same time, hitting the rate limits again. The workaround is to set the retry limit ridiculously high, higher than we would ideally like in production. Another workaround would be to serialize the Nijie tests in the test suite. This can be done with lockfiles and flock(2). This helps, but we can still hit the rate limit even under serialized execution. --- app/logical/sources/strategies/nijie.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/logical/sources/strategies/nijie.rb b/app/logical/sources/strategies/nijie.rb index b2a6fcc7a..4556f274a 100644 --- a/app/logical/sources/strategies/nijie.rb +++ b/app/logical/sources/strategies/nijie.rb @@ -182,7 +182,7 @@ module Sources form = { email: Danbooru.config.nijie_login, password: Danbooru.config.nijie_password } # XXX `retriable` must come after `cache` so that retries don't return cached error responses. - response = http.cache(1.hour).use(:retriable).post("https://nijie.info/login_int.php", form: form) + response = http.cache(1.hour).use(retriable: { max_retries: 20 }).post("https://nijie.info/login_int.php", form: form) DanbooruLogger.info "Nijie login failed (#{url}, #{response.status})" if response.status != 200 return nil unless response.status == 200