Production deployments are rolled back in bug 1621505.

400 with "While firing hook:\n\nTypeError: Cannot read property 'length' of null\n\n---\n\n* method: triggerHook\n* errorCode: InputError\n* statusCode: 400\n* time: 2020-03-11T01:13:50.838Z"

this was seen in both shipit and treeherder.

The error is reported here, meaning it most likely comes from taskcreator.fire.

Here's the logging for that request from the Hooks service:

{
  "jsonPayload": {
    "Timestamp": 1583889230838000000,
    "serviceContext": {
      "service": "hooks",
      "version": "v27.0.0"
    },
    "Severity": 5,
    "Logger": "taskcluster.hooks.api",
    "Fields": {
      "hasAuthed": true,
      "sourceIp": "34.83.197.158",
      "statusCode": 400,
      "resource": "/hooks/project-xpi/in-tree-action-1-release-promotion%2Fc8466d9eff/trigger",
      "name": "triggerHook",
      "clientId": "project/releng/shipit/staging",
      "v": 1,
      "satisfyingScopes": [
        "hooks:trigger-hook:project-xpi/in-tree-action-1-release-promotion/c8466d9eff"
      ],
      "query": {},
      "expires": "3019-10-31T20:49:54.835Z",
      "public": false,
      "apiVersion": "v1",
      "method": "POST",
      "duration": 230.528322
    },
    "Hostname": "taskcluster-hooks-web-c896dcfbc-w5f79",
    "Pid": 1,
    "EnvVersion": "2.0",
    "Type": "monitor.apiMethod"
  },
  "timestamp": "2020-03-11T01:13:50.839358807Z",
}

I can repro in a dev environment at https://dustin.taskcluster-dev.net/hooks/relpromo/test

payload

decision:
  action:
    name: foo
    title: foo
    description: foo
    taskGroupId: "1234"
    cb_name: foo
    symbol: "F"
  push:
    owner: me
    revision: "123"
    pushlog_id: "13"
  repository:
    url: https://foo/bar
    project: try
    level: "1"
  parameters: {}
user:
  input: null
  taskId: null
  taskGroupId: "1234"

It looks like this is an issue in json-e:

const fs = require('fs');
const slugid = require('slugid');
const yaml = require('js-yaml');
const render = require('json-e');                                                                                                                                                                                                                                                                                              

const payload = yaml.load(fs.readFileSync('pay.yaml'));
const hook = yaml.load(fs.readFileSync('hook.yaml'));

const res = render(hook, {payload, taskId: slugid.nice()});
console.log(JSON.stringify(res, null, 2));

with the payload from comment 4 in pay.yaml and the hook body from https://firefox-ci-tc.services.mozilla.com/hooks/project-xpi/in-tree-action-1-release-promotion%2Fc8466d9eff in hook.yaml. This will fail with the same error on json-e 4, but succeed for json-e 3.

https://github.com/taskcluster/taskcluster/pull/2491 to revert JSON-e, and then we can make a 27.1.0 and upgrade everything to that.

This error occurs in v3.0.2
Dustin tried v3.0.0 and observed no error. I tried v3.0.1 and observed no error, but v3.0.2 does give the same error

PR to further revert json-e https://github.com/taskcluster/taskcluster/pull/2493

Thanks to owlish, this is now actually fixed in v27.2.0. [edit: meaning the rollback to JSON-e v3.0.2 was succesful]

(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #8)

Thanks to owlish, this is now actually fixed in v27.2.0. [edit: meaning the rollback to JSON-e v3.0.2 was succesful]

rollback to v3.0.1 you mean?

There's another PR, to return the short-circuiting of boolean logic support https://github.com/taskcluster/taskcluster/pull/2497

rollback to v3.0.1 you mean?

facepalm

So, to summarize: I rolled back the change in json-e that caused trouble, and made a json-e release that supports short-circuiting in boolean logic but doesn't have breaking things from the v3.0.2. The new version is 4.0.1, and Taskcluster will be using it in subsequent releases. The bug in json-e solution for which we had to roll back is re-open in json-e repo.

Closing this now