Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Description • 9 years ago

We have ES 1.2.4 on the dev ES cluster which we use for the Input -dev and -stage environments.

If you go to:

    https://input.allizom.org/api/v1/feedback/

You get an HTTP 500:

Traceback (most recent call last):

  File "/data/www/input.allizom.org/input/vendor/src/django/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/data/www/input.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/framework_django.py", line 497, in wrapper
    return wrapped(*args, **kwargs)

  File "/data/www/input.allizom.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)

  File "/data/www/input.allizom.org/input/fjord/base/utils.py", line 457, in decorated_func
    response = f(request, *args, **kwargs)

  File "/data/www/input.allizom.org/input/fjord/feedback/api_views.py", line 143, in _feedback_api_router
    return public_feedback_api_view(request)

  File "/data/www/input.allizom.org/input/vendor/src/django/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)

  File "/data/www/input.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.38.0.31/newrelic/hooks/component_djangorestframework.py", line 27, in _nr_wrapper_APIView_dispatch_
    return wrapped(*args, **kwargs)

  File "/data/www/input.allizom.org/input/vendor/src/django/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)

  File "/data/www/input.allizom.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 327, in dispatch
    response = self.handle_exception(exc)

  File "/data/www/input.allizom.org/input/vendor/src/django-rest-framework/rest_framework/views.py", line 324, in dispatch
    response = handler(request, *args, **kwargs)

  File "/data/www/input.allizom.org/input/fjord/feedback/api_views.py", line 97, in get
    responses = models.ResponseMappingType.reshape(search[:maximum])

  File "/data/www/input.allizom.org/input/fjord/feedback/models.py", line 423, in reshape
    for result in results

  File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1615, in __iter__
    return iter(self._do_search())

  File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1431, in _do_search
    response = self.raw()

  File "/data/www/input.allizom.org/input/vendor/src/elasticutils/elasticutils/__init__.py", line 1510, in raw
    **extra_search_kwargs)

  File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)

  File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/client/__init__.py", line 440, in search
    params=params, body=body)

  File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/transport.py", line 284, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)

  File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/connection/http_urllib3.py", line 55, in perform_request
    self._raise_error(response.status, raw_data)

  File "/data/www/input.allizom.org/input/vendor/src/elasticsearch-py/elasticsearch/connection/base.py", line 97, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)

RequestError: TransportError(400, u'SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[s8HM99PgT12L_9gKv2RRbg][stage-inputindex][4]: RemoteTransportException[[elasticsearch1_dev_phx1][inet[/10.8.81.181:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][4]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@6ca4ae62>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[l7m1bjB-RC-AbQ4Srp0eLQ][stage-inputindex][3]: SearchParseException[[stage-inputindex][3]: from[-1],size[-1],sort[<custom:"created": org.
 elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@20978f4>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[7P21zPibSNGvmWl952lotA][stage-inputindex][2]: RemoteTransportException[[elasticsearch3_dev_phx1][inet[/10.8.81.183:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][2]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1f07f1e4>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "u
 ser_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[s8HM99PgT12L_9gKv2RRbg][stage-inputindex][1]: RemoteTransportException[[elasticsearch1_dev_phx1][inet[/10.8.81.181:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][1]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@492666b8>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }{[7P21zPibSNGvmWl952lotA][stage-inputindex][0]: RemoteTransport
 Exception[[elasticsearch3_dev_phx1][inet[/10.8.81.183:9300]][search/phase/query]]; nested: SearchParseException[[stage-inputindex][0]: from[-1],size[-1],sort[<custom:"created": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@424f80e>!]: Parse Failure [Failed to parse source [{"sort": [{"created": "desc"}], "fields": [["id", "happy", "api", "url_domain", "has_email", "description", "category", "description_bigrams", "user_agent", "product", "version", "platform", "locale", "source", "campaign", "organic", "created"]], "size": 1000}]]]; nested: ElasticsearchParseException[Expected field name but got END_ARRAY "fields"]; }]')


Need to look into what's going on. Seems like it's having problems with the "sort" portion which might mean it's an ElasticUtils problem.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 1 • 9 years ago

Grabbing this to work on tomorrow.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 2 • 9 years ago

Running the test suite kicks up 17 errors. We can track down the issues using that.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 3 • 9 years ago

Easy fix. In a PR: https://github.com/mozilla/fjord/pull/439

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 4 • 9 years ago

The reason there's all this "depends on"/"does not depends on" stuff in this bug is because I accidentally blocked a bunch of bugs on this bug, but got the bug number wrong.

Landed in: https://github.com/mozilla/fjord/commit/de4621e524f959d5615393a88e494b9b02deca09

Will push on Monday.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 5 • 9 years ago

Had to push today because next week is release week.

Pushed it to production. Verified it works now on -stage (where we have ES 1.2.4) and -prod (where we have ES 0.90).

Marking as FIXED.