Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Description • 9 years ago

Similar errors have appeared multiple times in the try test pool:

14:29:19     INFO - Running command: ['c:\\mozilla-build\\python27\\python.exe', 'c:\\mozilla-build\\buildbotve\\scripts\\buildbot', 'sendchange', '--master', 'buildbot-master81.build.mozilla.org:9301', '--username', 'sendchange-unittest', '--branch', 'try-win32-debug-unittest', '-r', u'eacce28c98b6e2be79f5f36e3487d52db0b23f13', '--username', u'msreckovic@mozilla.com', '--comments', 'try: -b d -p win32,win64 -u all -t none', '--property', 'buildid:20150616132629', '--property', 'pgo_build:False', '--property', 'builduid:328ab5a737234a1a9909135c1f903c62', u'https://queue.taskcluster.net/v1/task/B4nnGoSiTe-6FAF6kveL2A/artifacts/public/build/firefox-41.0a1.en-US.win32.zip', u'https://queue.taskcluster.net/v1/task/B4nnGoSiTe-6FAF6kveL2A/artifacts/public/build/test_packages.json']
14:29:19     INFO - Copy/paste: c:\mozilla-build\python27\python.exe c:\mozilla-build\buildbotve\scripts\buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange-unittest --branch try-win32-debug-unittest -r eacce28c98b6e2be79f5f36e3487d52db0b23f13 --username msreckovic@mozilla.com --comments "try: -b d -p win32,win64 -u all -t none" --property buildid:20150616132629 --property pgo_build:False --property builduid:328ab5a737234a1a9909135c1f903c62 https://queue.taskcluster.net/v1/task/B4nnGoSiTe-6FAF6kveL2A/artifacts/public/build/firefox-41.0a1.en-US.win32.zip https://queue.taskcluster.net/v1/task/B4nnGoSiTe-6FAF6kveL2A/artifacts/public/build/test_packages.json
14:29:19     INFO -  c:\mozilla-build\python27\python.exe: can't open file 'c:\mozilla-build\buildbotve\scripts\buildbot': [Errno 2] No such file or directory
14:29:19    ERROR - Return code: 2
14:29:19     INFO - The sendchange failed but we don't want to turn the build orange: 2


Even though send change failed the jobs returned green.

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 1 • 9 years ago

Looks like the same issue of mozharness assuming c:\mozilla-build\buildbotve is a virtualenv with the buildbot server code installed in it.  I forgot how we solved this last time, though..

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 2 • 9 years ago

We solved that with a symbolic link in the startbuildbot.bat 

REM A temporary terrible hack to get pass hardcoding in mozharness for a deprecated path 
REM Once that is addressed this should be removed
REM Ref Bug 1162230
REM the hack is here because the link command needs to be rerun on each boot

if not exist C:\mozilla-build\buildbotve mkdir C:\mozilla-build\buildbotve
ln -s C:\mozilla-build\virtualenv.py C:\mozilla-build\buildbotve\virtualenv.py

	Chris AtLee [:catlee]
	Comment 3 • 9 years ago

I've disabled these machines in slavealloc until we can get this fixed

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 4 • 9 years ago

Ah, ok -- I wonder if we could modify mozharness to install buildbot in its virtualenv, instead.  Jordan?

	Jordan Lund (:jlund) Assignee
	Comment 5 • 9 years ago

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Ah, ok -- I wonder if we could modify mozharness to install buildbot in its
> virtualenv, instead.  Jordan?

could do. Though, we currently don't create a venv for building ff (we create one for uploading TC artifacts and recording influxdb stats). As it is right now, we call the actual mh build script (and the 'mach build') subprocess with the system wide py2.7

I previously was trying to install buildbot into a venv but ran into problems on linux because of mock[1]. So either we would have to address that or make a windows edge case. Both don't sound optimal.

How about we config more than one path for the buildbot exe that supports both pre and post puppet windows and then search for one that exists before calling it. Even as a stopgap until we are 100% on puppet with new file paths.

[1] https://bugzil.la/1087102

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 6 • 9 years ago

That works.  The path has a version # in it, so when we upgraded Buildbot we'd need to make sure that version number changed, too, but that doesn't seem like such a big deal and is unlikely to happen too often in the next year or so.

	Jordan Lund (:jlund) Assignee
	Comment 7 • 9 years ago

discussed over irc, I can pick this up next week in a hack session @ whistler :)

/me steals from markco. feel free to steal back if you really want it

	Jordan Lund (:jlund) Assignee
	Comment 8 • 9 years ago

hacking on this today.

so it looks like exes supports a list already for something else. This introduces a dict where key==reason_or_name_of_path, value=path

# negative testing
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9534be641a87

while these tests run, I'll look at addressing the buildbot fix for this. Unfortunetly, since we are not using mh for beta/release, we will need to fix buildbot (MozillaBuildFactory)

	Jordan Lund (:jlund) Assignee
	Comment 9 • 9 years ago

this adds the new puppetized path for all windows jobs. I am putting it in front of the old/current buildbotve path.

My assumption here is 'if the windows slave has a desired exe from the new path (buildbot-0.8.4-pre-moz6\scripts\*'), use that. Which would include the buildbot exe

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 10 • 9 years ago

Comment on attachment 8626290 [details] [diff] [review]
150625_win_puppet_sendchange-bbot-cfgs.patch

This looks correct.  Do you think a comment would be helpful, or will hg blame be sufficient?

	Jordan Lund (:jlund) Assignee
	Comment 11 • 9 years ago

took a couple iterations but https://treeherder.mozilla.org/#/jobs?repo=try&revision=9534be641a87 is finally happy

logic still applies from https://bugzilla.mozilla.org/show_bug.cgi?id=1175701#c8

	Michael Shal [:mshal]
	Comment 12 • 9 years ago

Comment on attachment 8627372 [details] [diff] [review]
150629_windows_sendchange_query_exe_paths_support-mh.patch

This looks fine - the only thing I see that might cause trouble is that we do the '% repl_dict' substitution after the os.path.exists() checks. So if we used %(abs_work_dir) or whatever in the path, I think it would fail. It doesn't look like we actually use that support in any of the exe paths, however.

	Jordan Lund (:jlund) Assignee
	Comment 13 • 9 years ago

(In reply to Michael Shal [:mshal] from comment #12)
> Comment on attachment 8627372 [details] [diff] [review]
> 150629_windows_sendchange_query_exe_paths_support-mh.patch
> 
> This looks fine - the only thing I see that might cause trouble is that we
> do the '% repl_dict' substitution after the os.path.exists() checks

as mentioned on irc, tx for the catch. we should not remove support for this expansion if the list, tuple, and string case can use it already.

I've respun a new patch and will attach for r? once try runs are green

	Jordan Lund (:jlund) Assignee
	Comment 14 • 9 years ago

same try rev as before, retriggers look good with latest mh pin:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9534be641a87

interdiff:
http://people.mozilla.org/~jlund/150630_windows_sendchange_query_exe-interdiff.patch

	Michael Shal [:mshal]
	Comment 15 • 9 years ago

Comment on attachment 8628023 [details] [diff] [review]
150630_windows_sendchange_query_exe.patch

Looks good!

	Jordan Lund (:jlund) Assignee
	Comment 16 • 9 years ago

Comment on attachment 8628023 [details] [diff] [review]
150630_windows_sendchange_query_exe.patch

thanks, on default

remote:   https://hg.mozilla.org/build/mozharness/rev/af321e004210

	Jordan Lund (:jlund) Assignee
	Comment 17 • 9 years ago

Comment on attachment 8626290 [details] [diff] [review]
150625_win_puppet_sendchange-bbot-cfgs.patch

thanks, I added bug # and comment above each path

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/fdeaeb10ee77

	Jordan Lund (:jlund) Assignee
	Comment 18 • 9 years ago

In production: https://hg.mozilla.org/build/buildbot-configs/rev/fdeaeb10ee77

	Jordan Lund (:jlund) Assignee
	Comment 19 • 9 years ago

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

	Jordan Lund (:jlund) Assignee
	Comment 20 • 9 years ago

markco: feel free to resolve this bug once you are able to test it again. Let me know if I can help at all :D

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 21 • 9 years ago

I am going to resolve this. If this proves to be an issue after I get passed a few blockers I will reopen.

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 22 • 9 years ago

It seems we are having some failures. 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=103193c1cd13

9:43:16     INFO -  Traceback (most recent call last):
09:43:16     INFO -    File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\buildbot", line 3, in <module>
09:43:16     INFO -      from buildbot.scripts import runner
09:43:16     INFO -  ImportError: No module named buildbot.scripts
09:43:16    ERROR - Return code: 1
09:43:16     INFO - The sendchange failed but we don't want to turn the build orange: 1
09:43:16     INFO - Running command: ['c:\\mozilla-build\\python27\

	Jordan Lund (:jlund) Assignee
	Comment 23 • 9 years ago

so the patch seems to be working as expected:

if searches the possible 'buildbot' exe paths and

# if the machine is puppet based, it runs: 
'c:\\mozilla-build\\python27\\python.exe', 'c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\scripts\\buildbot' .. etc

if the machine is gpo based, it runs:
'c:\\mozilla-build\\python27\\python.exe', 'c:\\mozilla-build\\buildbotve\\scripts\\buildbot' etc

however, the puppet one is failing because it doesn't have 'buildbot' installed in the interpreter: 

puppet:
cltbld@B-2008-IX-0175 ~
$ cat /c/mozilla-build/buildbot-0.8.4-pre-moz6/Scripts/buildbot
#!c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python2.7.exe

from buildbot.scripts import runner
runner.run()

cltbld@B-2008-IX-0175 ~
$ /c/mozilla-build/python27/python.exe
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from buildbot.scripts import runner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named buildbot.scripts

gpo:
cltbld@B-2008-IX-0033 ~
$ cat /c/mozilla-build/buildbotve/scripts/buildbot
#!c:\mozilla-build\buildbotve\Scripts\python.exe

from buildbot.scripts import runner
runner.run()

cltbld@B-2008-IX-0033 ~
$ /c/mozilla-build/python27/python.exe
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from buildbot.scripts import runner
>>>

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 24 • 9 years ago

The module is available from within the virtualenv directory:

C:\Users\cltbld>C:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python.exe
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> from buildbot.scripts import runner
>>>

Would this be a viable option for the Puppet configured machines?

	Jordan Lund (:jlund) Assignee
	Comment 25 • 9 years ago

I don't see why we can't use it :)

	Pulsebot
	Comment 26 • 9 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/1413ee039b51

	Carsten Book [:Tomcat]
	Comment 27 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/1413ee039b51

	Chris Cooper [:coop] (he/him)
	Comment 28 • 9 years ago

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/2da0b7595468

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 29 • 9 years ago

I think we are good: 
12:18:53     INFO - #####
12:18:53     INFO - ##### Running sendchange step.
12:18:53     INFO - #####
12:18:53     INFO - Running pre-action listener: influxdb_recording_pre_action
12:18:53     INFO - Running main action method: sendchange
12:18:53     INFO - 'enable_talos_sendchange' is false; skipping
12:18:53     INFO - Running command: ['c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\Scripts\\python.exe', 'c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\scripts\\buildbot', 'sendchange', '--master', 'buildbot-master81.build.mozilla.org:9301', '--username', 'sendchange-unittest', '--branch', 'try-win32-debug-unittest', '-r', u'56272fbd3fb5a400eb778bda1f3cc208d79becc0', '--username', u'nfitzgerald@mozilla.com', '--comments', 'try: -b do -p all -u cppunit -t none', '--property', 'buildid:20150724110756', '--property', 'pgo_build:False', '--property', 'builduid:8267d8745bbb4bb0bd8de95607be9936', u'https://queue.taskcluster.net/v1/task/DBeC18NBSO6J2eKeFw9N1w/artifacts/public/build/firefox-42.0a1.en-US.win32.zip', u'https://queue.taskcluster.net/v1/task/DBeC18NBSO6J2eKeFw9N1w/artifacts/public/build/test_packages.json']
12:18:53     INFO - Copy/paste: c:\mozilla-build\buildbot-0.8.4-pre-moz6\Scripts\python.exe c:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange-unittest --branch try-win32-debug-unittest -r 56272fbd3fb5a400eb778bda1f3cc208d79becc0 --username nfitzgerald@mozilla.com --comments "try: -b do -p all -u cppunit -t none" --property buildid:20150724110756 --property pgo_build:False --property builduid:8267d8745bbb4bb0bd8de95607be9936 https://queue.taskcluster.net/v1/task/DBeC18NBSO6J2eKeFw9N1w/artifacts/public/build/firefox-42.0a1.en-US.win32.zip https://queue.taskcluster.net/v1/task/DBeC18NBSO6J2eKeFw9N1w/artifacts/public/build/test_packages.json
12:18:59     INFO -  NOTE: --username/-u is deprecated: use --who/-W'
12:18:59     INFO -  change sent successfully
12:18:59     INFO - Return code: 0
12:18:59     INFO - Running post-action listener: influxdb_recording_post_action
12:18:59     INFO - Starting new HTTP connection (1): goldiewilson-onepointtwentyone-1.c.influxdb.com

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 30 • 9 years ago

Looked like sendchange has failed in a different build: 

19:18:54     INFO - ##### Running sendchange step.
19:18:54     INFO - #####
19:18:54     INFO - Running pre-action listener: influxdb_recording_pre_action
19:18:54     INFO - Running main action method: sendchange
19:18:54     INFO - Running command: ['c:\\mozilla-build\\python27\\python.exe', 'c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\scripts\\buildbot', 'sendchange', '--master', 'buildbot-master81.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'try-win32-talos', '-r', u'ad632449c494bea3eb0f12ea8c92fa4fa2ed37b4', '--username', u'smokey101stair@gmail.com', '--comments', 'try: -b do -p linux,linux64,macosx64,win32,win64 -u mochitest-bc,mochitest-e10s-browser-chrome-1,mochitest-e10s-browser-chrome-2,mochitest-e10s-browser-chrome-3 -t none', '--property', 'buildid:20150724182541', '--property', 'pgo_build:False', '--property', 'builduid:66abb0400c82453d96fc2e5b10c68293', u'https://queue.taskcluster.net/v1/task/rhJcCXZETfm5_AFPzJgxyw/artifacts/public/build/firefox-42.0a1.en-US.win32.zip']
19:18:54     INFO - Copy/paste: c:\mozilla-build\python27\python.exe c:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\buildbot sendchange --master buildbot-master81.build.mozilla.org:9301 --username sendchange --branch try-win32-talos -r ad632449c494bea3eb0f12ea8c92fa4fa2ed37b4 --username smokey101stair@gmail.com --comments "try: -b do -p linux,linux64,macosx64,win32,win64 -u mochitest-bc,mochitest-e10s-browser-chrome-1,mochitest-e10s-browser-chrome-2,mochitest-e10s-browser-chrome-3 -t none" --property buildid:20150724182541 --property pgo_build:False --property builduid:66abb0400c82453d96fc2e5b10c68293 https://queue.taskcluster.net/v1/task/rhJcCXZETfm5_AFPzJgxyw/artifacts/public/build/firefox-42.0a1.en-US.win32.zip
19:18:54     INFO -  Traceback (most recent call last):
19:18:54     INFO -    File "c:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\buildbot", line 3, in <module>
19:18:54     INFO -      from buildbot.scripts import runner
19:18:54     INFO -  ImportError: No module named buildbot.scripts
19:18:54    ERROR - Return code: 1
19:18:54     INFO - The sendchange failed but we don't want to turn the build orange: 1

	Jordan Lund (:jlund) Assignee
	Comment 31 • 9 years ago

(In reply to Mark Cornmesser [:markco] from comment #30)
> Looked like sendchange has failed in a different build: 

this is likely due to a try job being pushed that is out of date (prior to https://bugzilla.mozilla.org/show_bug.cgi?id=1175701#c27) with m-c or it's based on an older branch where this change doesn't exist.

The best we can do here is either wait a few days until all devs have merged in m-c or just let the jobs not trigger sendchanges and expect that devs will update to latest m-c.

We should probably merge this sendchange patch across all branches though so we don't break say esr jobs

	Amy Rich [:arr] [:arich]
	Comment 32 • 9 years ago

jlund: has this change been merged everywhere at this point? If so, are we good to r/f?

	Jordan Lund (:jlund) Assignee
	Comment 33 • 9 years ago

(In reply to Amy Rich [:arr] [:arich] from comment #32)
> jlund: has this change been merged everywhere at this point? If so, are we
> good to r/f?

I'm assuming that this patch has helped and we are no longer seeing the issue in comment 30/31. I will uplift it everywhere tomorrow.

	Jordan Lund (:jlund) Assignee
	Comment 34 • 9 years ago

this bug's patches has been uplifted to m-a but due to current releases and merge week, I have to postpone uploading beta/release/esr till friday or monday

	Jordan Lund (:jlund) Assignee
	Comment 35 • 9 years ago

(In reply to Jordan Lund (:jlund) from comment #34)
> this bug's patches has been uplifted to m-a but due to current releases and
> merge week, I have to postpone uploading beta/release/esr till friday or
> monday

we are actually good here as mozharness is not used to build beta/release/esr yet. those branches use buildbot factories/build-steps only.

dustin: a while ago we patched buildbot to support the new path sendchange path[1]. however, in the morharness case that alone wasn't enough as we also had to tell it to use a specific interpreter[2] because of this[3]. Looks like in the buildbot case we use SendChangeStep. Do you think we will need to put `${MOZILLABUILD}python27` after `${MOZILLABUILD}buildbot-0.8.4-pre-moz6\\scripts` in [1]? I worry that will cause spurious results if you think we need to.

[1] https://bugzilla.mozilla.org/attachment.cgi?id=8626290
[2] https://bug1175701.bmoattachments.org/attachment.cgi?id=8638119
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1175701#c23

ni: dustin via irc

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 36 • 9 years ago

I don't understand the situation well enough to make any concrete suggestions.  Changing the order of PATH seems a pretty drastic step, though, and is likely to have other consequences.  I suspect it would be better to modify SendChangeStep to use the proper interpreter.

I brought up the possibility, recently, of reverting support for installing multiple versions of Buildbot side-by-side.  We've had this in puppet for a while because, on Linux and Mac OS X, it's easy (yay symlinks).  The expectation was that we'd be upgrading Buildbot frequently, and that it would be nice to be able to have the "old" and "new" versions installed side-by-side and just flip between them with slavealloc.  This functionality was never developed or used.  On Windows, this has been the source of a lot of headache.  Maybe we should just go back to installing one version of Buildbot in a very predictable path on all systems, not just Windows.

	Jordan Lund (:jlund) Assignee
	Comment 37 • 9 years ago

here is how we are calling sendchange via mozharness currently (gpo):
    22:38:42     INFO - Running command: ['c:\\mozilla-build\\python27\\python.exe', 'c:\\mozilla-build\\buildbotve\\scripts\\buildbot', 'sendchange',

and this is buildbot factory based equivalent (gpo):
    'python' 'c:/builds/moz2_slave/m-beta-w32-0000000000000000000/tools/buildfarm/utils/retry.py' '-s' '5' '-t' '1800' '-r' '5', 'buildbot' 'sendchange'
    # with important path bits: PATH=C:\mozilla-build\python27;C:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts;C:\mozilla-build\buildbotve\scripts;


for the puppet win case we have configured the following in mozharness:
    22:38:42     INFO - Running command: ['c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\Scripts\\python.exe', 'c:\\mozilla-build\\buildbot-0.8.4-pre-moz6\\scripts\\buildbot', 'sendchange',


so for buildbot factory win puppet case, with our current PATH above I think we would end up using the `c:\\mozilla-build\\python27\\python.exe` interpreter to call `C:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\buildbot` which is bad as iiuc we don't have buildbot installed in `c:\\mozilla-build\\python27\\python.exe` on the windows puppet scenario.

So there are two options as I see it:

1) patch SendChangeStep to take in a configured path for the interpreter. iiuc SendChangeStep is part of buildbot itself, not buildbotcustom.

2) install buildbot in `c:\\mozilla-build\\python27\\python.exe` on windows puppet machines.

markco, how hard would 2 be. it would mean we don't need any extra conditions or hacks on the automation side.

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 38 • 9 years ago

#1 shouldn't be bad at all -- it's in buildbotcustom:
  https://dxr.mozilla.org/build_buildbotcustom/source/steps/misc.py#74
It should be easy enough to replace `'python'` with a configurable path at
  https://dxr.mozilla.org/build_buildbotcustom/source/steps/misc.py#146

c:\\mozilla-build\\python27 isn't a virtualenv, so installing Buildbot there is not a great idea.

	Jordan Lund (:jlund) Assignee
	Comment 39 • 9 years ago

(In reply to Dustin J. Mitchell [:dustin] from comment #38)
> #1 shouldn't be bad at all -- it's in buildbotcustom:
>   https://dxr.mozilla.org/build_buildbotcustom/source/steps/misc.py#74
> It should be easy enough to replace `'python'` with a configurable path at
>   https://dxr.mozilla.org/build_buildbotcustom/source/steps/misc.py#146
> 

ah, not sure why I thought SendChangeStep was part of buildbot. Okay, the one issue here is buildbot is not really aware about how the windows machines are configured (whether they are gpo based or puppet based). So we can't add say the following in buildbot-configs:

['win32']['sendchange_interpreter'] = 'C:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\python.exe'

as that will fail for the current gpo case. Mozharness handles this by saying something like:

if os.path.exists('C:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\python.exe'):
    # change the sendchange python interpreter as this is a puppet machine
    python = 'C:\mozilla-build\buildbot-0.8.4-pre-moz6\scripts\python.exe'

But in buildbot land, we can't really have runtime conditions like that pre-configured. How do we normally support two different slave configured states within the same platform in buildbot world?

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 40 • 9 years ago

That is tricky.. Adding code to call a `stat` on the slave would be a more substantial change to the step.

Stepping back for a minute to note how many "in-transition" things we have going on right now: some builds use mozharness, some don't.  Some builds use puppet-configured hosts, some don't.  Some builds use Buildbot, some don't.  Yikes!

Given all of this, your PATH modifiction suggestion from comment 35 is sounding more attractive.  On non-puppet machines, ${MOZILLABUILD}buildbot-0.8.4-pre-moz6 doesn't exist at all, so adding it won't hurt.  On puppetized machines, use of `python` in the build process will find the virtualenv python, but I *think* that will be OK, as it still has pywin32 and all that suff available.  Would it be possible to amend the path only in the SendChangeStep?

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 41 • 9 years ago

It looks like we are going to make a change on the Puppet configured Windows machines to use the generic buildbotve directory for the active version of buildbot.

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 42 • 9 years ago

These changes can be reverted now. "c:\mozilla-build\buildbotve\scripts\buildbot" now exists within the Puppet configuration.

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 43 • 9 years ago

jlund: Could you revert the changes made for the puppet machines, please? Everything should now be going to the buildbotve directory. The specific version buildbot directories no longer exists.

	Jordan Lund (:jlund) Assignee
	Comment 44 • 9 years ago

Comment on attachment 8626290 [details] [diff] [review]
150625_win_puppet_sendchange-bbot-cfgs.patch

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/175d149df600
remote:   https://hg.mozilla.org/build/buildbot-configs/rev/3c677267d1ee

	Jordan Lund (:jlund) Assignee
	Comment 45 • 9 years ago

backing out of mh is kind of awkward as it was part of the large mh->gecko_tree migration.

I had to do this patch by hand. I'll land tomorrow as trees are closed right now and I'm not 100% it won't cause bustage. I am keeping the script core changes in case we want to have multiple search options in exes again in the future.

keeping needinfo to self as reminder

	Jordan Lund (:jlund) Assignee
	Comment 46 • 9 years ago

backed out on https://hg.mozilla.org/integration/mozilla-inbound/rev/c05e13ac8ce0

aurora to go

	Carsten Book [:Tomcat]
	Comment 47 • 9 years ago

https://hg.mozilla.org/mozilla-central/rev/c05e13ac8ce0

	Jordan Lund (:jlund) Assignee
	Comment 48 • 9 years ago

this has been uplifted. the patch has been reverted

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 49 • 9 years ago

I kinda feel bad reopening this, but we should now point sendchange to use  "C:\mozilla-build\buildbotve\Scripts\python.exe" . This directory should be static and has all the needed modules needed for sendchange to work available to python within this directory. This is not urgent. I presently have a hack to work around it.

	Amy Rich [:arr] [:arich]
	Comment 51 • 8 years ago

Did this ever get resolved?

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 52 • 8 years ago

As far as I know there has been no change to this.

	Jordan Lund (:jlund) Assignee
	Comment 53 • 8 years ago

missed comment 49. will push a patch to try with requested value for python exe

	Jordan Lund (:jlund) Assignee
	Comment 54 • 8 years ago

https://treeherder.mozilla.org/#/jobs?repo=try&revision=76d4bf83d0e2

	Jordan Lund (:jlund) Assignee
	Comment 55 • 8 years ago

Review commit: https://reviewboard.mozilla.org/r/65584/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65584/

	Jordan Lund (:jlund) Assignee
	Comment 56 • 8 years ago

(In reply to Jordan Lund (:jlund) from comment #54)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=76d4bf83d0e2

looks sane.

(In reply to Jordan Lund (:jlund) from comment #55)
> Created attachment 8772885 [details]
> Bug 1175701 - change windows python interpreter for buildbot sendchange,
> 
> Review commit: https://reviewboard.mozilla.org/r/65584/diff/#index_header

this ^ shouldn't affect taskcluster builds

	Jordan Lund (:jlund) Assignee
	Comment 57 • 8 years ago

markco: is what I said below accurate?

11:05:17 <mshal> jlund: so running buildbot/sendchange has to use a specific python exe?
11:05:41 <mshal> I don't quite follow what exactly it's supposed to fix
11:07:12 <jlund> iiuc - sys.executable is resolving to a path on our windows machines that is a link to the real python.exe that we want
11:07:26 <jlund> or at least that's my guess at markco's 'hack' mentioned here: https://bugzilla.mozilla.org/show_bug.cgi?id=1175701#c49
11:08:02 <jlund> so it is working. but he wants to remove the hack and use a static path for buildbot sendchange call
11:08:17 <mshal> oic

	Jordan Lund (:jlund) Assignee
	Comment 58 • 8 years ago

also of note:

11:39:02 <Callek> jlund: arr: yea my concern is with removing the hack during the heavy tme-sens releases (last betas, rc/release/beta1) -- specifically both incase a failure to sendchange causes a build bustage and incase something in the build is relying on a side-effect of something installed with the buildbot python and that path....


since there is nothing busted, I'll wait till markco gets back from PTO and chats with callek over comment 57

the patch seems fine and is r+ so I'll land once there are no further concerns and we can remove the temp fix on win machines at the same time.

	Mark Cornmesser [:markco] OOO 2024/04/15 Reporter
	Comment 59 • 8 years ago

With the pending move to task cluster, my opinion is to just let sit as is. Though it is not pretty it is functioning.