https://sentry.prod.mozaws.net/operations/taskcluster-community/issues/6374518/?query=is:unresolved

If you can verify that this is, indeed, unset, then I can gpg you a fresh value.

The github config is currently of the following form:

 github:
            bot_username: communitytc-taskcluster
            github_app_id: "(5 digits)"
            github_private_pem: |
                -----BEGIN RSA PRIVATE KEY-----
                (an RSA key)
                -----END RSA PRIVATE KEY-----
            pulse_password: (some string)
            pulse_username: communitytc-taskcluster-github
            taskcluster_access_token: (some string)
            webhook_secret: (some string)

Let's replace that key with a fresh one that you think should be correct, to rule out the possibility that the old one is causing problems.

I sent along the secrets. I sent a client ID and client secret as well but it turns out those aren't needed, so you can ignore them.

New key is deployed. Leaving this open till we've tested whether it's fixed.

https://github.com/taskcluster/taskcluster/pull/1434 to detect this error a bit better.

Per yesterday's troubleshooting, it appears that a correctly formatted PEM value is base64 encoded in the Kubernets secrets object, but the format is incorrect (\n rather than newlines) in the actual env var. Options appear to be to either pass the var as a json object, which appears to succeed with the worker creds, or alter the taskcluster github app to handle the format produced by Kubernetes. https://github.com/taskcluster/taskcluster/pull/1466 does the latter, and is expected to resolve this bug when included in a release and deployed.

Maybe I misunderstood something in our conversation yesterday. The K8s secret value does have newlines (base64-encoded). It's when those are transferred to the running pod that those newlines seem to be escaped.

However, https://github.com/kubernetes/kubernetes/issues/23404 suggests the opposite, that k8s does in fact preserve newlines (to the consternation of people accidentally putting newlines in their env vars).

dustin@dorp ~ $ ( echo hello; echo world ) | base64
aGVsbG8Kd29ybGQK

I edited that into the secret:

kubectl edit secret taskcluster-github

and then restarted the pod and:

dustin@lamport ~ $ kubectl exec taskcluster-github-web-8458f5747b-r59ft  printenv | grep -C3 GITHUB_PRIVATE_PEM                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
TASKCLUSTER_ROOT_URL=https://dustin.taskcluster-dev.net                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
LEVEL=notice
PULSE_PASSWORD=yGO3CHjJSzKSKmhCyxzydA
GITHUB_PRIVATE_PEM=hello
world

GITHUB_APP_ID=???

so I don't think we have solved this -- helm, or something before it, is escaping those newlines.

In the interest of getting started, I'm going to adjust the patch to indicate that this is a workaround for a bug in the code generating the k8s configs, rather than a bug in k8s, and then land it anyway. Should we open a new bug to drill down to why the cloudops machinery is escaping newlines?

This got solved upstream. We'll leave the \n-unescaping in place anyway.