Ubuntu × JRubyでSSLError: certificate verify failed

昨日から急に自社のJenkinsサーバ(Ubuntu)で実行しているEmbulkのタスクが以下のようなエラーを吐いて失敗するようになりました。

 at RUBY.block in call(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:43)
    at RUBY.with_net_http_connection(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87)
    at RUBY.call(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32)
    at RUBY.call(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15)
    at RUBY.build_response(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139)
    at RUBY.run_request(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377)
    at RUBY.post(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:177)
    at RUBY.fetch_access_token(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/signet-0.7.3/lib/signet/oauth_2/client.rb:960)
    at RUBY.fetch_access_token!(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/signet-0.7.3/lib/signet/oauth_2/client.rb:998)
    at RUBY.fetch_access_token!(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/googleauth-0.5.1/lib/googleauth/signet.rb:69)
    at RUBY.apply!(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/googleauth-0.5.1/lib/googleauth/signet.rb:45)
    at RUBY.apply!(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/googleauth-0.5.1/lib/googleauth/service_account.rb:93)
    at RUBY.apply_request_options(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/http_command.rb:313)
    at RUBY.execute_once(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/http_command.rb:289)
    at RUBY.block in execute(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/http_command.rb:104)
    at RUBY.block in retriable(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/retriable-3.0.2/lib/retriable.rb:53)
    at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:305)
    at RUBY.retriable(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/retriable-3.0.2/lib/retriable.rb:49)
    at RUBY.block in execute(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/http_command.rb:101)
    at RUBY.block in retriable(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/retriable-3.0.2/lib/retriable.rb:53)
    at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:305)
    at RUBY.retriable(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/retriable-3.0.2/lib/retriable.rb:49)
    at RUBY.execute(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/http_command.rb:93)
    at RUBY.execute_or_queue_command(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/lib/google/apis/core/base_service.rb:360)
    at RUBY.get_dataset(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.12.0/generated/google/apis/bigquery_v2/service.rb:134)
    at RUBY.block in get_dataset(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/embulk-output-bigquery-0.4.5/lib/embulk/output/bigquery/bigquery_client.rb:369)
    at RUBY.with_network_retry(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/embulk-output-bigquery-0.4.5/lib/embulk/output/bigquery/google_client.rb:81)
    at RUBY.get_dataset(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/embulk-output-bigquery-0.4.5/lib/embulk/output/bigquery/bigquery_client.rb:369)
    at RUBY.auto_create(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/embulk-output-bigquery-0.4.5/lib/embulk/output/bigquery.rb:288)
    at RUBY.transaction(/jenkins/workspace/some-jenkins-job/vendor/bundle/jruby/2.3.0/gems/embulk-output-bigquery-0.4.5/lib/embulk/output/bigquery.rb:343)
    at RUBY.transaction(uri:classloader:/embulk/output_plugin.rb:64)
    at RUBY.run(uri:classloader:/embulk/runner.rb:84)
    at RUBY.run(uri:classloader:/embulk/command/embulk_run.rb:307)
    at RUBY.<main>(uri:classloader:/embulk/command/embulk_main.rb:2)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:956)
    at jenkins.home.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<main>(file:/jenkins/home/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:30)

Error: org.jruby.exceptions.RaiseException: (SSLError) certificate verify failed

embulk-output-bigqueryでBigQuery APIにアクセスする際のエラーのようですが、全然心当たりがありません。というのをTwitterでつぶやいたらTreasureData社の方に拾ってもらって同じような現象の方の話を教えてもらいました、神。。！

embulk-output-bigqueryで同じ現象が起きているんですが、さっきツイート見かけたのでこちらのスレッドご参考までに。https://t.co/Dg8ChFGm2e
— Satoshi Akama (@oreradio) 2017年10月3日

その後色々調べる感じ、Ebmulkが使ってるJRuby (or Java)のレイヤーで起こる特有のエラーっぽいと言うことが分かりました。

調査結果

スクリプトとDockerfileは全部GitHubにおいてます (https://github.com/kamatama41/snippets/tree/master/jruby-ssl-error)

以下のようなRuby スクリプトを用意します

require 'net/http'
require 'optparse'

opts = ARGV.getopts('', 'set-cert')
puts opts

url = 'https://www.google.co.jp'
puts "Start to access #{url}"
uri = URI.parse(url)
http = Net::HTTP.new(uri.hostname, uri.port)
http.use_ssl = true
if opts['set-cert']
  http.ca_file = '/Equifax_Secure_Certificate_Authority.pem'
end

get = Net::HTTP::Get.new(uri)
res = http.request(get)
puts "#{res.code}: #{res.message}"
puts 'Done'

CRubyで実行すると普通に成功します。

% docker run jruby-ssl-error ruby
Run: ~/.rbenv/versions/2.4.2/bin/ruby /test.rb 
{"set-cert"=>false}
Start to access https://www.google.co.jp
200: OK
Done

JRubyで実行するとcertificate verify failedで失敗します

% docker run jruby-ssl-error jruby
Run: ~/.rbenv/versions/jruby-9.1.13.0/bin/jruby /test.rb 
{"set-cert"=>false}
Start to access https://www.google.co.jp
OpenSSL::SSL::SSLError: certificate verify failed
  connect_nonblock at org/jruby/ext/openssl/SSLSocket.java:228
           connect at /root/.rbenv/versions/jruby-9.1.13.0/lib/ruby/stdlib/net/http.rb:938
          do_start at /root/.rbenv/versions/jruby-9.1.13.0/lib/ruby/stdlib/net/http.rb:868
             start at /root/.rbenv/versions/jruby-9.1.13.0/lib/ruby/stdlib/net/http.rb:857
           request at /root/.rbenv/versions/jruby-9.1.13.0/lib/ruby/stdlib/net/http.rb:1409
            <main> at /test.rb:17

JRubyでも、今回消されたEquifax_Secure_Certificate_Authority.pemを設定すると通ります。 https://knowledge.geotrust.com/support/knowledge-base/index?vproductcat=G&vdomain=GEOTRUST_COM&page=content&id=SO5761&locale=en_US&redirected=true

% docker run jruby-ssl-error jruby --set-cert
Run: ~/.rbenv/versions/jruby-9.1.13.0/bin/jruby /test.rb --set-cert
{"set-cert"=>true}
Start to access https://www.google.co.jp
200: OK
Done

対応策

ということで、根本原因はよく分かりませんが、、とりあえず動かせるようにするためには消された Equifax Secure Certificate Authority のルートCA証明書を再インストールすれば動くようになります。

(注意: セキュリティアップデートで消されたものを再インストールするということセキュリティ的になにかあるかもしれないので、自己責任でお願いします)

CA証明書を /usr/share/ca-certificates 配下に保存します

$ sudo vi /usr/share/ca-certificates/Equifax_Secure_Certificate_Authority.pem

-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIENd70zzANBgkqhkiG9w0BAQUFADBOMQswCQYDVQQGEwJV
...

保存した証明書名を /etc/ca-certificates.conf に追記します

$ sudo vi /etc/ca-certificates.conf 

...
mozilla/ePKI_Root_Certification_Authority.crt
mozilla/thawte_Primary_Root_CA.crt
mozilla/thawte_Primary_Root_CA_-_G2.crt
mozilla/thawte_Primary_Root_CA_-_G3.crt
Equifax_Secure_Certificate_Authority.pem # これ

update-ca-certificates コマンドを実行します

$ sudo update-ca-certificates 
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:Equifax_Secure_Certificate_Authority.pem.pem
done.
done.

追記

ちなみにJavaで同じようにSSLアクセスするコード書いてみましたが、普通に通りました。謎。。

import javax.net.ssl.HttpsURLConnection;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;

public class Test {
    public static void main(String[] args) throws IOException {
        URL url = new URL("https://www.google.co.jp");
        HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
        connection.setRequestMethod("GET");
        BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String line;
        StringBuilder builder = new StringBuilder();
        while ((line = reader.readLine()) != null) {
            builder.append(line);
        }
        reader.close();
        System.out.println(connection.getResponseCode());
        System.out.println(connection.getResponseMessage());
    }
}

% docker run jruby-ssl-error java Test
Run: java Test
200
OK

追記2

JRubyにissueが上がっているようです、jruby-opensslの不具合(?)の模様

JRubyでも話題になっているっぽいですね。https://t.co/02n4nqMUrN
— hiroyuki sato (@hiroysato) 2017年10月5日

かまたま日記3

プログラミングメイン、たまに日常

Ubuntu × JRubyでSSLError: certificate verify failed

調査結果

対応策

追記

追記2