1039170 - Integration Trees closed due Android tests fail Automation Error: Unable to reboot panda-xyz via Relay Board.

Reporter

Description

•

10 years ago

23:26:28     INFO -  07/15/2014 23:26:28: ERROR: Failed to powercycle 15 times, giving up
23:26:28     INFO -  07/15/2014 23:26:28: WARNING: Automation Error: Unable to reboot panda-0733 via Relay Board.

https://tbpl.mozilla.org/php/getParsedLog.php?id=43903939&tree=Mozilla-Inbound

seems a recent problem and hitting several pandas

Carsten Book [:Tomcat]

Reporter

Comment 1

•

10 years ago

affected pandas it seems

panda-0700
panda-0847
panda-0758
panda-0853
panda-0727
panda-0733
panda-0752
panda-0699
panda-0041
panda-0800
panda-0733
panda-0772
panda-0710
panda-0040
panda-0808

Pete Moore [:pmoore][:pete]

Comment 2

•

10 years ago

So my first thoughts are:

*) This is probably related to the most recent panda move from sc1 -> scl3 which I think ran yesterday.
*) There could be problems with the relay board data in https://hg.mozilla.org/build/tools/file/tip/buildfarm/mobile/devices.json -> I can compare this to what we have in https://inventory.mozilla.org/en-US/#inventory
*) I will check if all these pandas are connected to the same panda relay, in case there is a problem with the relay (should not be foopy related - foopies are not involved in the rebooting via panda relay)

Carsten Book [:Tomcat]

Reporter

Comment 3

•

10 years ago

reopened the trees since retrigger on other pandas came back green, will keep monitor the situation

Pete Moore [:pmoore][:pete]

Comment 4

•

10 years ago

Should be safe to disable these pandas in slavealloc until cause has been established

Pete Moore [:pmoore][:pete]

Comment 5

•

10 years ago

Looks like a lot of different relay boards are involved, so not related to a particular relay:

pmoore@Elisandra:~/git/tools/buildfarm/mobile master $ cat badpandas.list | while read panda; do line="$(sed -n "/${panda}/=" devices.json)"; sed -n ${line},$((line+4))p devices.json ; done
    "panda-0700": {
        "foopy": "foopy88",
        "relayhost": "panda-relay-063.p8.releng.scl3.mozilla.com",
        "relayid": "1:3"
    },
    "panda-0847": {
        "foopy": "foopy99",
        "relayhost": "panda-relay-076.p9.releng.scl3.mozilla.com",
        "relayid": "2:3"
    },
    "panda-0758": {
        "foopy": "foopy92",
        "relayhost": "panda-relay-068.p8.releng.scl3.mozilla.com",
        "relayid": "2:2"
    },
    "panda-0853": {
        "foopy": "foopy100",
        "relayhost": "panda-relay-077.p9.releng.scl3.mozilla.com",
        "relayid": "1:2"
    },
    "panda-0727": {
        "foopy": "foopy90",
        "relayhost": "panda-relay-065.p8.releng.scl3.mozilla.com",
        "relayid": "2:4"
    },
    "panda-0733": {
        "foopy": "foopy90",
        "relayhost": "panda-relay-066.p8.releng.scl3.mozilla.com",
        "relayid": "1:3"
    },
    "panda-0752": {
        "foopy": "foopy92",
        "relayhost": "panda-relay-067.p8.releng.scl3.mozilla.com",
        "relayid": "2:7"
    },
    "panda-0699": {
        "foopy": "foopy88",
        "relayhost": "panda-relay-063.p8.releng.scl3.mozilla.com",
        "relayid": "1:2"
    },
    "panda-0041": {
        "foopy": "foopy103",
        "relayhost": "panda-relay-003.p10.releng.scl3.mozilla.com",
        "relayid": "2:4"
    },
    "panda-0800": {
        "foopy": "foopy96",
        "relayhost": "panda-relay-072.p9.releng.scl3.mozilla.com",
        "relayid": "1:4"
    },
    "panda-0733": {
        "foopy": "foopy90",
        "relayhost": "panda-relay-066.p8.releng.scl3.mozilla.com",
        "relayid": "1:3"
    },
    "panda-0772": {
        "foopy": "foopy93",
        "relayhost": "panda-relay-069.p8.releng.scl3.mozilla.com",
        "relayid": "2:5"
    },
    "panda-0710": {
        "foopy": "foopy89",
        "relayhost": "panda-relay-064.p8.releng.scl3.mozilla.com",
        "relayid": "1:2"
    },
    "panda-0040": {
        "foopy": "foopy103",
        "relayhost": "panda-relay-003.p10.releng.scl3.mozilla.com",
        "relayid": "2:3"
    },
    "panda-0808": {
        "foopy": "foopy96",
        "relayhost": "panda-relay-073.p9.releng.scl3.mozilla.com",
        "relayid": "1:1"
    },

Pete Moore [:pmoore][:pete]

Comment 6

•

10 years ago

Panda 3 move details:

https://infra.etherpad.mozilla.org/scl1-panda-train-3-checklist
https://bugzilla.mozilla.org/show_bug.cgi?id=1027839

Pete Moore [:pmoore][:pete]

Comment 7

•

10 years ago

Also releng part (e.g. updates to devices.json): https://bugzilla.mozilla.org/show_bug.cgi?id=1038014

Pete Moore [:pmoore][:pete]

Comment 8

•

10 years ago

I checked inventory against devices.json: the first half all match exactly:

panda-0700: https://inventory.mozilla.org/en-US/systems/show/8172/
panda-0847: https://inventory.mozilla.org/en-US/systems/show/8319/
panda-0758: https://inventory.mozilla.org/en-US/systems/show/8230/
panda-0853: https://inventory.mozilla.org/en-US/systems/show/8325/
panda-0727: https://inventory.mozilla.org/en-US/systems/show/8199/
panda-0733: https://inventory.mozilla.org/en-US/systems/show/8205/
panda-0752: https://inventory.mozilla.org/en-US/systems/show/8224/
panda-0699: https://inventory.mozilla.org/en-US/systems/show/8171/

I didn't check the second half, as this clearly isn't the problem.

In other words, "system.relay.0" in inventory matches the "relayhost" and "relayid" in devices.json for these pandas.

Pete Moore [:pmoore][:pete]

Comment 9

•

10 years ago

Can also ping a random relay host:

pmoore@Elisandra:~/git/mozilla master $ ping -c5 panda-relay-073.p9.releng.scl3.mozilla.com
PING panda-relay-073.p9.releng.scl3.mozilla.com (10.26.136.42): 56 data bytes
64 bytes from 10.26.136.42: icmp_seq=0 ttl=56 time=185.868 ms
64 bytes from 10.26.136.42: icmp_seq=1 ttl=56 time=185.788 ms
64 bytes from 10.26.136.42: icmp_seq=2 ttl=56 time=187.358 ms
64 bytes from 10.26.136.42: icmp_seq=3 ttl=56 time=185.341 ms
64 bytes from 10.26.136.42: icmp_seq=4 ttl=56 time=185.613 ms

--- panda-relay-073.p9.releng.scl3.mozilla.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 185.341/185.994/187.358/0.706 ms

Pete Moore [:pmoore][:pete]

Comment 10

•

10 years ago

Will find code for "Calling PDU powercycle" log lines, to see what it is doing...

Pete Moore [:pmoore][:pete]

Comment 11

•

10 years ago

Was able to pdu powercycle panda-700 manually:

[cltbld@foopy88.p8.releng.scl3.mozilla.com sut_lib]$ python
Python 2.6.6 (r266:84292, Dec  7 2011, 20:48:22) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import relay
>>> relay.powercycle('panda-relay-063.p8.releng.scl3.mozilla.com', 1, 3)                                                                                                                                    
True
>>>

Carsten Book [:Tomcat]

Reporter

Comment 12

•

10 years ago

panda-0710
panda-0759
panda-0846
panda-0733
panda-0852
panda-0788
panda-0028
panda-0827
panda-0029
panda-0046
panda-0746
panda-0790
panda-0715
panda-0805
panda-0846
panda-0805

Amy Rich [:arr] [:arich]

Comment 13

•

10 years ago

dustin: looks like the releng software (which bypasses mozpool) is sometimes having difficulty rebooting the pandas from move train 3.  We didn't experience this with the other two move trains, so I don't know if there's something different about this one. Could you take a look and help diagnose, please?

Pete Moore [:pmoore][:pete]

Comment 14

•

10 years ago

OK I'll disable these train 3 pandas in slavealloc now:

panda-0698.p8.releng.scl3.mozilla.com
panda-0699.p8.releng.scl3.mozilla.com
panda-0700.p8.releng.scl3.mozilla.com
panda-0701.p8.releng.scl3.mozilla.com
panda-0702.p8.releng.scl3.mozilla.com
panda-0703.p8.releng.scl3.mozilla.com
panda-0704.p8.releng.scl3.mozilla.com
panda-0705.p8.releng.scl3.mozilla.com
panda-0706.p8.releng.scl3.mozilla.com
panda-0707.p8.releng.scl3.mozilla.com
panda-0708.p8.releng.scl3.mozilla.com
panda-0709.p8.releng.scl3.mozilla.com
panda-0710.p8.releng.scl3.mozilla.com
panda-0711.p8.releng.scl3.mozilla.com
panda-0712.p8.releng.scl3.mozilla.com
panda-0713.p8.releng.scl3.mozilla.com
panda-0714.p8.releng.scl3.mozilla.com
panda-0715.p8.releng.scl3.mozilla.com
panda-0716.p8.releng.scl3.mozilla.com
panda-0717.p8.releng.scl3.mozilla.com
panda-0718.p8.releng.scl3.mozilla.com
panda-0719.p8.releng.scl3.mozilla.com
panda-0721.p8.releng.scl3.mozilla.com
panda-0722.p8.releng.scl3.mozilla.com
panda-0723.p8.releng.scl3.mozilla.com
panda-0724.p8.releng.scl3.mozilla.com
panda-0725.p8.releng.scl3.mozilla.com
panda-0726.p8.releng.scl3.mozilla.com
panda-0727.p8.releng.scl3.mozilla.com
panda-0728.p8.releng.scl3.mozilla.com
panda-0729.p8.releng.scl3.mozilla.com
panda-0730.p8.releng.scl3.mozilla.com
panda-0731.p8.releng.scl3.mozilla.com
panda-0732.p8.releng.scl3.mozilla.com
panda-0733.p8.releng.scl3.mozilla.com
panda-0734.p8.releng.scl3.mozilla.com
panda-0735.p8.releng.scl3.mozilla.com
panda-0736.p8.releng.scl3.mozilla.com
panda-0737.p8.releng.scl3.mozilla.com
panda-0738.p8.releng.scl3.mozilla.com
panda-0739.p8.releng.scl3.mozilla.com
panda-0740.p8.releng.scl3.mozilla.com
panda-0741.p8.releng.scl3.mozilla.com
panda-0742.p8.releng.scl3.mozilla.com
panda-0743.p8.releng.scl3.mozilla.com
panda-0744.p8.releng.scl3.mozilla.com
panda-0745.p8.releng.scl3.mozilla.com
panda-0746.p8.releng.scl3.mozilla.com
panda-0747.p8.releng.scl3.mozilla.com
panda-0748.p8.releng.scl3.mozilla.com
panda-0749.p8.releng.scl3.mozilla.com
panda-0750.p8.releng.scl3.mozilla.com
panda-0751.p8.releng.scl3.mozilla.com
panda-0752.p8.releng.scl3.mozilla.com
panda-0753.p8.releng.scl3.mozilla.com
panda-0754.p8.releng.scl3.mozilla.com
panda-0755.p8.releng.scl3.mozilla.com
panda-0756.p8.releng.scl3.mozilla.com
panda-0757.p8.releng.scl3.mozilla.com
panda-0758.p8.releng.scl3.mozilla.com
panda-0759.p8.releng.scl3.mozilla.com
panda-0760.p8.releng.scl3.mozilla.com
panda-0761.p8.releng.scl3.mozilla.com
panda-0762.p8.releng.scl3.mozilla.com
panda-0763.p8.releng.scl3.mozilla.com
panda-0764.p8.releng.scl3.mozilla.com
panda-0765.p8.releng.scl3.mozilla.com
panda-0766.p8.releng.scl3.mozilla.com
panda-0767.p8.releng.scl3.mozilla.com
panda-0768.p8.releng.scl3.mozilla.com
panda-0769.p8.releng.scl3.mozilla.com
panda-0770.p8.releng.scl3.mozilla.com
panda-0771.p8.releng.scl3.mozilla.com
panda-0772.p8.releng.scl3.mozilla.com
panda-0773.p8.releng.scl3.mozilla.com
panda-0774.p8.releng.scl3.mozilla.com
panda-0775.p8.releng.scl3.mozilla.com
panda-0776.p8.releng.scl3.mozilla.com
panda-0777.p8.releng.scl3.mozilla.com
panda-0778.p8.releng.scl3.mozilla.com
panda-0779.p8.releng.scl3.mozilla.com
panda-0780.p8.releng.scl3.mozilla.com
panda-0782.p8.releng.scl3.mozilla.com
panda-0783.p8.releng.scl3.mozilla.com
panda-0784.p8.releng.scl3.mozilla.com
panda-0785.p8.releng.scl3.mozilla.com
panda-0885.p8.releng.scl3.mozilla.com

panda-0786.p9.releng.scl3.mozilla.com
panda-0787.p9.releng.scl3.mozilla.com
panda-0788.p9.releng.scl3.mozilla.com
panda-0789.p9.releng.scl3.mozilla.com
panda-0790.p9.releng.scl3.mozilla.com
panda-0791.p9.releng.scl3.mozilla.com
panda-0792.p9.releng.scl3.mozilla.com
panda-0794.p9.releng.scl3.mozilla.com
panda-0795.p9.releng.scl3.mozilla.com
panda-0796.p9.releng.scl3.mozilla.com
panda-0797.p9.releng.scl3.mozilla.com
panda-0798.p9.releng.scl3.mozilla.com
panda-0799.p9.releng.scl3.mozilla.com
panda-0800.p9.releng.scl3.mozilla.com
panda-0801.p9.releng.scl3.mozilla.com
panda-0802.p9.releng.scl3.mozilla.com
panda-0803.p9.releng.scl3.mozilla.com
panda-0804.p9.releng.scl3.mozilla.com
panda-0805.p9.releng.scl3.mozilla.com
panda-0806.p9.releng.scl3.mozilla.com
panda-0807.p9.releng.scl3.mozilla.com
panda-0808.p9.releng.scl3.mozilla.com
panda-0809.p9.releng.scl3.mozilla.com
panda-0810.p9.releng.scl3.mozilla.com
panda-0811.p9.releng.scl3.mozilla.com
panda-0812.p9.releng.scl3.mozilla.com
panda-0813.p9.releng.scl3.mozilla.com
panda-0814.p9.releng.scl3.mozilla.com
panda-0815.p9.releng.scl3.mozilla.com
panda-0816.p9.releng.scl3.mozilla.com
panda-0817.p9.releng.scl3.mozilla.com
panda-0818.p9.releng.scl3.mozilla.com
panda-0819.p9.releng.scl3.mozilla.com
panda-0820.p9.releng.scl3.mozilla.com
panda-0821.p9.releng.scl3.mozilla.com
panda-0822.p9.releng.scl3.mozilla.com
panda-0823.p9.releng.scl3.mozilla.com
panda-0824.p9.releng.scl3.mozilla.com
panda-0825.p9.releng.scl3.mozilla.com
panda-0826.p9.releng.scl3.mozilla.com
panda-0827.p9.releng.scl3.mozilla.com
panda-0828.p9.releng.scl3.mozilla.com
panda-0829.p9.releng.scl3.mozilla.com
panda-0830.p9.releng.scl3.mozilla.com
panda-0831.p9.releng.scl3.mozilla.com
panda-0832.p9.releng.scl3.mozilla.com
panda-0833.p9.releng.scl3.mozilla.com
panda-0834.p9.releng.scl3.mozilla.com
panda-0835.p9.releng.scl3.mozilla.com
panda-0836.p9.releng.scl3.mozilla.com
panda-0837.p9.releng.scl3.mozilla.com
panda-0838.p9.releng.scl3.mozilla.com
panda-0839.p9.releng.scl3.mozilla.com
panda-0840.p9.releng.scl3.mozilla.com
panda-0841.p9.releng.scl3.mozilla.com
panda-0842.p9.releng.scl3.mozilla.com
panda-0843.p9.releng.scl3.mozilla.com
panda-0844.p9.releng.scl3.mozilla.com
panda-0845.p9.releng.scl3.mozilla.com
panda-0846.p9.releng.scl3.mozilla.com
panda-0847.p9.releng.scl3.mozilla.com
panda-0848.p9.releng.scl3.mozilla.com
panda-0849.p9.releng.scl3.mozilla.com
panda-0850.p9.releng.scl3.mozilla.com
panda-0851.p9.releng.scl3.mozilla.com
panda-0852.p9.releng.scl3.mozilla.com
panda-0853.p9.releng.scl3.mozilla.com
panda-0854.p9.releng.scl3.mozilla.com
panda-0855.p9.releng.scl3.mozilla.com
panda-0856.p9.releng.scl3.mozilla.com
panda-0857.p9.releng.scl3.mozilla.com
panda-0858.p9.releng.scl3.mozilla.com
panda-0859.p9.releng.scl3.mozilla.com
panda-0860.p9.releng.scl3.mozilla.com
panda-0861.p9.releng.scl3.mozilla.com
panda-0862.p9.releng.scl3.mozilla.com
panda-0863.p9.releng.scl3.mozilla.com
panda-0864.p9.releng.scl3.mozilla.com
panda-0865.p9.releng.scl3.mozilla.com
panda-0866.p9.releng.scl3.mozilla.com
panda-0867.p9.releng.scl3.mozilla.com
panda-0868.p9.releng.scl3.mozilla.com
panda-0869.p9.releng.scl3.mozilla.com
panda-0870.p9.releng.scl3.mozilla.com
panda-0871.p9.releng.scl3.mozilla.com
panda-0872.p9.releng.scl3.mozilla.com
panda-0873.p9.releng.scl3.mozilla.com
panda-0886.p9.releng.scl3.mozilla.com

panda-0022.p10.releng.scl3.mozilla.com
panda-0023.p10.releng.scl3.mozilla.com
panda-0024.p10.releng.scl3.mozilla.com
panda-0025.p10.releng.scl3.mozilla.com
panda-0026.p10.releng.scl3.mozilla.com
panda-0027.p10.releng.scl3.mozilla.com
panda-0028.p10.releng.scl3.mozilla.com
panda-0029.p10.releng.scl3.mozilla.com
panda-0030.p10.releng.scl3.mozilla.com
panda-0031.p10.releng.scl3.mozilla.com
panda-0032.p10.releng.scl3.mozilla.com
panda-0034.p10.releng.scl3.mozilla.com
panda-0035.p10.releng.scl3.mozilla.com
panda-0037.p10.releng.scl3.mozilla.com
panda-0038.p10.releng.scl3.mozilla.com
panda-0039.p10.releng.scl3.mozilla.com
panda-0040.p10.releng.scl3.mozilla.com
panda-0041.p10.releng.scl3.mozilla.com
panda-0042.p10.releng.scl3.mozilla.com
panda-0043.p10.releng.scl3.mozilla.com
panda-0046.p10.releng.scl3.mozilla.com
panda-0047.p10.releng.scl3.mozilla.com
panda-0048.p10.releng.scl3.mozilla.com
panda-0049.p10.releng.scl3.mozilla.com
panda-0050.p10.releng.scl3.mozilla.com
panda-0051.p10.releng.scl3.mozilla.com
panda-0052.p10.releng.scl3.mozilla.com
panda-0053.p10.releng.scl3.mozilla.com
panda-0054.p10.releng.scl3.mozilla.com
panda-0055.p10.releng.scl3.mozilla.com
panda-0056.p10.releng.scl3.mozilla.com
panda-0058.p10.releng.scl3.mozilla.com
panda-0059.p10.releng.scl3.mozilla.com
panda-0060.p10.releng.scl3.mozilla.com
panda-0061.p10.releng.scl3.mozilla.com
panda-0062.p10.releng.scl3.mozilla.com
panda-0063.p10.releng.scl3.mozilla.com
panda-0064.p10.releng.scl3.mozilla.com
panda-0065.p10.releng.scl3.mozilla.com
panda-0066.p10.releng.scl3.mozilla.com
panda-0067.p10.releng.scl3.mozilla.com
panda-0068.p10.releng.scl3.mozilla.com
panda-0070.p10.releng.scl3.mozilla.com
panda-0071.p10.releng.scl3.mozilla.com
panda-0072.p10.releng.scl3.mozilla.com
panda-0073.p10.releng.scl3.mozilla.com
panda-0074.p10.releng.scl3.mozilla.com
panda-0075.p10.releng.scl3.mozilla.com
panda-0076.p10.releng.scl3.mozilla.com
panda-0077.p10.releng.scl3.mozilla.com
panda-0078.p10.releng.scl3.mozilla.com
panda-0079.p10.releng.scl3.mozilla.com
panda-0080.p10.releng.scl3.mozilla.com
panda-0874.p10.releng.scl3.mozilla.com
panda-0875.p10.releng.scl3.mozilla.com
panda-0876.p10.releng.scl3.mozilla.com
panda-0877.p10.releng.scl3.mozilla.com
panda-0878.p10.releng.scl3.mozilla.com
panda-0879.p10.releng.scl3.mozilla.com
panda-0880.p10.releng.scl3.mozilla.com
panda-0881.p10.releng.scl3.mozilla.com
panda-0882.p10.releng.scl3.mozilla.com
panda-0883.p10.releng.scl3.mozilla.com
panda-0884.p10.releng.scl3.mozilla.com
panda-0887.p10.releng.scl3.mozilla.com
panda-0888.p10.releng.scl3.mozilla.com
panda-0889.p10.releng.scl3.mozilla.com
panda-0890.p10.releng.scl3.mozilla.com
panda-0891.p10.releng.scl3.mozilla.com
panda-0892.p10.releng.scl3.mozilla.com
panda-0893.p10.releng.scl3.mozilla.com
panda-0894.p10.releng.scl3.mozilla.com
panda-0895.p10.releng.scl3.mozilla.com
panda-0896.p10.releng.scl3.mozilla.com
panda-0897.p10.releng.scl3.mozilla.com
panda-0898.p10.releng.scl3.mozilla.com
panda-0899.p10.releng.scl3.mozilla.com
panda-0900.p10.releng.scl3.mozilla.com
panda-0901.p10.releng.scl3.mozilla.com
panda-0902.p10.releng.scl3.mozilla.com
panda-0903.p10.releng.scl3.mozilla.com
panda-0904.p10.releng.scl3.mozilla.com
panda-0905.p10.releng.scl3.mozilla.com
panda-0906.p10.releng.scl3.mozilla.com
panda-0907.p10.releng.scl3.mozilla.com
panda-0908.p10.releng.scl3.mozilla.com
panda-0909.p10.releng.scl3.mozilla.com

Pete Moore [:pmoore][:pete]

Comment 15

•

10 years ago

Using http://mozilla-slaveapi.readthedocs.org/en/latest/api/#slaves-slave-actions-disable ...

Pete Moore [:pmoore][:pete]

Comment 16

•

10 years ago

I cannot see a way to specify a comment via this REST API, so I am having to do it without. I also don't see a way to query the current enabled/disabled status - but maybe I missed this.

Running:

cat naughtypandas | sed 's/\..*//' | while read panda; do curl -u "${MY_LDAP_USER}:${MY_LDAP_PASSWORD}" -dfoo=bar "https://secure.pub.build.mozilla.org/slaveapi/slaves/${panda}/actions/disable"; done

(where naughtypandas is a sorted copy and paste of the list of pandas above, with blank lines removed)

Pete Moore [:pmoore][:pete]

Comment 17

•

10 years ago

Dustin found problem! devices.json was out of date on the foopies - needed to reconfig. On foopies we had e.g. https://hg.mozilla.org/build/tools/file/408f66b7fd6e/buildfarm/mobile/devices.json

Kim is doing a reconfig now, this should fix it.

Amy Rich [:arr] [:arich]

Updated

•

10 years ago

Blocks: 1038014

Pete Moore [:pmoore][:pete]

Comment 18

•

10 years ago

We split it so I updated foopies, Mim did rest of reconfig. manage_foopies.py hung, but seems all foopies were updated, except foopy101 which has known problems (bug 1038452):

(fabric)pmoore@Elisandra:~/git/tools master $ buildfarm/maintenance/manage_foopies.py -f buildfarm/mobile/devices.json -H all -j 16 show_revision
foopy115       82fb9d827b24
foopy114       82fb9d827b24
foopy105       82fb9d827b24
foopy107       82fb9d827b24
foopy102       82fb9d827b24
foopy104       82fb9d827b24
foopy100       82fb9d827b24
foopy103       82fb9d827b24
foopy106       82fb9d827b24
foopy108       82fb9d827b24
foopy110       82fb9d827b24
foopy111       82fb9d827b24
foopy113       82fb9d827b24
foopy109       82fb9d827b24
foopy112       82fb9d827b24
foopy117       82fb9d827b24
foopy123       82fb9d827b24
foopy116       82fb9d827b24
foopy120       82fb9d827b24
foopy119       82fb9d827b24
foopy122       82fb9d827b24
foopy121       82fb9d827b24
foopy124       82fb9d827b24
foopy125       82fb9d827b24
foopy126       82fb9d827b24
foopy28        82fb9d827b24
foopy27        82fb9d827b24
foopy128       82fb9d827b24
foopy29        82fb9d827b24
foopy30        82fb9d827b24
foopy31        82fb9d827b24
foopy32        82fb9d827b24
foopy39        82fb9d827b24
foopy43        82fb9d827b24
foopy41        82fb9d827b24
foopy42        82fb9d827b24
foopy44        82fb9d827b24
foopy47        82fb9d827b24
foopy45        82fb9d827b24
foopy46        82fb9d827b24
foopy40        82fb9d827b24
foopy48        82fb9d827b24
foopy49        82fb9d827b24
foopy50        82fb9d827b24
Failed to run show_revision on foopy101
Traceback (most recent call last):
  File "buildfarm/maintenance/manage_foopies.py", line 68, in run_action_on_foopy
    action_func(foopy)
  File "/Users/pmoore/git/tools/buildfarm/maintenance/foopy_fabric.py", line 29, in show_revision
    tools_rev = run('hg -R /builds/tools ident -i')
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 535, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/operations.py", line 1017, in run
    warn_only=warn_only, stdout=stdout, stderr=stderr, timeout=timeout)
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/operations.py", line 891, in _run_command
    channel=default_channel(), command=wrapped_command, pty=pty,
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/state.py", line 380, in default_channel
    chan = _open_session()
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/state.py", line 372, in _open_session
    return connections[env.host_string].get_transport().open_session()
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 109, in __getitem__
    self.connect(key)
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 101, in connect
    self[key] = connect(user, host, port, sock)
  File "/Users/pmoore/fabric/lib/python2.7/site-packages/fabric/network.py", line 461, in connect
    raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to foopy101.build.mozilla.org (tried 1 time)

foopy101 FAILED
foopy51        82fb9d827b24
foopy52        82fb9d827b24
foopy53        82fb9d827b24
foopy54        82fb9d827b24
foopy57        82fb9d827b24
foopy56        82fb9d827b24
foopy55        82fb9d827b24
foopy60        82fb9d827b24
foopy58        82fb9d827b24
foopy59        82fb9d827b24
foopy61        82fb9d827b24
foopy63        82fb9d827b24
foopy62        82fb9d827b24
foopy65        82fb9d827b24
foopy66        82fb9d827b24
foopy67        82fb9d827b24
foopy68        82fb9d827b24
foopy69        82fb9d827b24
foopy70        82fb9d827b24
foopy72        82fb9d827b24
foopy74        82fb9d827b24
foopy73        82fb9d827b24
foopy76        82fb9d827b24
foopy77        82fb9d827b24
foopy75        82fb9d827b24
foopy71        82fb9d827b24
foopy79        82fb9d827b24
foopy78        82fb9d827b24
foopy80        82fb9d827b24
foopy81        82fb9d827b24
foopy64        82fb9d827b24
foopy82        82fb9d827b24
foopy83        82fb9d827b24
foopy84        82fb9d827b24
foopy85        82fb9d827b24
foopy91        82fb9d827b24
foopy89        82fb9d827b24
foopy93        82fb9d827b24
foopy86        82fb9d827b24
foopy88        82fb9d827b24
foopy92        82fb9d827b24
foopy90        82fb9d827b24
foopy95        82fb9d827b24
foopy87        82fb9d827b24
foopy94        82fb9d827b24
foopy96        82fb9d827b24
foopy97        82fb9d827b24
foopy98        82fb9d827b24
foopy99        82fb9d827b24

Pete Moore [:pmoore][:pete]

Comment 19

•

10 years ago

s/Mim/Kim/ (of course!)

Will now re-enable train 3 pandas in slavealloc...

Pete Moore [:pmoore][:pete]

Comment 20

•

10 years ago

Running from cruncher to re-enable them all:

(tools_venv)[pmoore@cruncher.srv.releng.scl3 tools]$ cat naughtypandas | while read panda; do slavealloc enable ${panda}; done

(could not run locally as firewall seemed to block)

In a couple of minutes, all should be enabled again (together with any from train 3 that were previously intentionally disabled for another reason - hopefully not).

Pete Moore [:pmoore][:pete]

Updated

•

10 years ago

Status: NEW → RESOLVED

Closed: 10 years ago

Resolution: --- → FIXED

BMO Automation

Updated

•

6 years ago

Product: Release Engineering → Infrastructure & Operations

BMO Automation

Updated

•

4 years ago

Product: Infrastructure & Operations → Infrastructure & Operations Graveyard