KeyError after OAuth login

[aaclan-ebi]

Software version

seems to be happening in all env (dev, integration, staging, prod)

Expected behavior

User is redirected to the callback url specified by the app requesting authentication

Actual behavior

User sees the following error:

  File "/var/task/chalicelib/fusillade/api/__init__.py", line 126, in dispatch
    flask_res = self.connexion_full_dispatch_request()
  File "/var/task/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/var/task/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/var/task/flask/_compat.py", line 39, in reraise
    raise value
  File "/var/task/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/var/task/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/task/connexion/decorators/decorator.py", line 48, in wrapper
    response = function(request)
  File "/var/task/connexion/decorators/uri_parsing.py", line 143, in wrapper
    response = function(request)
  File "/var/task/connexion/decorators/response.py", line 109, in wrapper
    response = function(request)
  File "/var/task/connexion/decorators/parameter.py", line 126, in wrapper
    return function(**kwargs)
  File "/var/task/chalicelib/fusillade/api/oauth.py", line 164, in cb
    key=public_keys[token_header["kid"]],
KeyError: 'MDc2OTM3ODI4ODY2NUU5REVGRDVEM0MyOEYwQTkzNDZDRDlEQzNBRQ'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/chalicelib/fusillade/api/__init__.py", line 131, in dispatch
    exception=ex
  File "/var/lang/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/var/lang/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'KeyError' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/chalice/app.py", line 1070, in _get_view_function_response
    response = view_function(**function_args)
  File "/var/task/chalicelib/fusillade/api/__init__.py", line 139, in dispatch
    status_code=status_code,
UnboundLocalError: local variable 'status_code' referenced before assignment

Steps to reproduce the behavior

1. Login in Ingest UI
   (the UI does a GET request to
   https://auth.integration.data.humancellatlas.org/oauth/authorize?redirect_uri=https://ui.ingest.integration.data.humancellatlas.org/callback)
2. Use gmail account to authenticate

[Bento007]

I think this maybe cause by hitting the rate limit on the https://humancellatlas.auth0.com/.well-known/jwks.json endpoint. I'm gong to add some logging to try and confirm this.