Closed
Bug 1087102
Opened 10 years ago
Closed 10 years ago
buildbot executable is not reachable by mock
Categories
(Release Engineering :: Applications: MozharnessCore, defect)
Release Engineering
Applications: MozharnessCore
Tracking
(Not tracked)
RESOLVED
WONTFIX
People
(Reporter: jlund, Unassigned)
References
Details
mach runs inside mock and now needs to do sendchanges(bug 1084163) however it can't reach the buildbot exec as mock doesn't know about '/tools' I am experimenting with creating a virtualenv for linux builds and installing a pip version of buildbot. This requires uploading to our pypi server and configuring mozharness to install a custom venv.
|
Reporter | |
Comment 1•10 years ago
|
||
> ./utils/publish_package_our_pypi.sh ~/Downloads/buildbot-0.8.4-pre-moz4.tar.gz Testing http://pypi.pub.build.mozilla.org/pub/buildbot-0.8.4-pre-moz4.tar.gz ... HTTP/1.1 200 OK Date: Wed, 22 Oct 2014 01:01:38 GMT Server: Apache X-Backend-Server: web2.releng.webapp.scl3.mozilla.com Last-Modified: Wed, 22 Oct 2014 01:01:37 GMT ETag: W/"3e6428" Accept-Ranges: bytes Content-Length: 4088872 Content-Type: application/x-gzip
|
|
Reporter | |
Comment 2•10 years ago
|
||
from buildbot-0.8.4-pre-moz4.tar.gz, it looks like it expects:
## dependencies
setup_args['install_requires'] = [
'twisted >= 8.0.0',
'Jinja2 >= 2.1',
'sqlalchemy >= 0.6',
# buildbot depends on sqlalchemy internals. See buildbot.db.model.
'sqlalchemy-migrate == 0.6',
]
but our pypi server doesn't have sqlalchemy-migrate == 0.6 (it has 0.7.2). I've uploaded 0.6 to see how much farther I can progress.
testing http://pypi.pub.build.mozilla.org/pub/sqlalchemy-migrate-0.6.tar.gz ...
HTTP/1.1 200 OK
Date: Wed, 22 Oct 2014 01:43:47 GMT
Server: Apache
X-Backend-Server: web1.releng.webapp.scl3.mozilla.com
Last-Modified: Wed, 22 Oct 2014 01:43:46 GMT
ETag: W/"17f26"
Accept-Ranges: bytes
Content-Length: 98086
Content-Type: application/x-gzip|
|
Reporter | |
Comment 3•10 years ago
|
||
installing buildbot via pip requires gcc. gcc on our linux slaves is only available through mock environment (mock being the initial reason for needing this bug). We can continue down the rabbit hole by installing gcc or trying to find the bare minimum packages needed for a sendchange but this is already been a large time sink for an unclear gain. We are reverting the sendchange logic and subsequently the check-test target back into mozharness so we don't get hit with the end2end time increase. check-tests are be planned to be stripped out into their own job anyway. Closing for said reasons. We may re-open if we want sendchanges to come from mach in the future.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
|
||
Comment 4•10 years ago
|
||
Where did you see gcc as a dependency? This is all I needed to install buildbot (the same version as releng) in a venv: https://hg.mozilla.org/build/braindump/file/d0241ce318e8/buildbot-related/list_builder_differences.sh#l63 https://hg.mozilla.org/build/braindump/file/d0241ce318e8/buildbot-related/pre_buildbot_master_requirements.txt
|
|
Reporter | |
Comment 5•10 years ago
|
||
perhaps you have gcc available on your linux box? It is not a 'requirement' in the pip module sense but something needs it for the way 'pip' installs twistd. At least on my slave it complained for me:
snippet:
18:50:16 INFO - Installing buildbot==0.8.4-pre-moz4 into virtualenv /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv
18:50:16 INFO - retry: Calling <bound method FxDesktopBuild.run_command of <__main__.FxDesktopBuild object at 0xcd9310>> with args: [['/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/bin/pip', 'install', '--download-cache', '/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/cache', '--timeout', '120', 'buildbot==0.8.4-pre-moz4']], kwargs: {'error_level': 'warning', 'error_list': [{'substr': 'not found or a compiler error:', 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0xddb490>, 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0xb63810>, 'level': 'warning'}, {'regex': <_sre.SRE_Pattern object at 0xbc33f0>, 'level': 'debug'}, {'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0xdb8100>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0xdce390>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0xca2670>, 'level': 'critical'}], 'cwd': '/builds/slave/m-cen-lx-ntly-0000000000000000/build'}, attempt #1
18:50:16 INFO - Running command: ['/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/bin/pip', 'install', '--download-cache', '/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/cache', '--timeout', '120', 'buildbot==0.8.4-pre-moz4'] in /builds/slave/m-cen-lx-ntly-0000000000000000/build
18:50:16 INFO - Copy/paste: /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/bin/pip install --download-cache /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/cache --timeout 120 buildbot==0.8.4-pre-moz4
18:50:16 INFO - Ignoring indexes: https://pypi.python.org/simple/
18:50:16 INFO - Downloading/unpacking buildbot==0.8.4-pre-moz4
18:50:16 INFO - http://pypi.pvt.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pvt.build.mozilla.org has it available
18:50:16 INFO - http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
...
...
...
18:50:27 INFO - Running setup.py install for twisted
18:50:28 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c conftest.c -o conftest.o
18:50:28 INFO - unable to execute gcc: No such file or directory
18:50:28 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c conftest.c -o conftest.o
18:50:28 INFO - unable to execute gcc: No such file or directory
18:50:28 INFO - building 'twisted.test.raiser' extension
18:50:28 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c twisted/test/raiser.c -o build/temp.linux-x86_64-2.7/twisted/test/raiser.o
18:50:28 INFO - unable to execute gcc: No such file or directory
18:50:28 INFO - error: command 'gcc' failed with exit status 1
18:50:28 INFO - Complete output from command /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/bin/python -c "import setuptools, tokenize;__file__='/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/build/twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-771FU3-record/install-record.txt --single-version-externally-managed --compile --install-headers /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/include/site/python2.7:
18:50:28 INFO - running install
18:50:28 INFO - running build
18:50:28 INFO - running build_py
18:50:28 INFO - creating build
18:50:28 INFO - creating build/lib.linux-x86_64-2.7
18:50:28 INFO - creating build/lib.linux-x86_64-2.7/twisted
18:50:28 INFO - copying twisted/plugin.py -> build/lib.linux-x86_64-2.7/twisted
18:50:28 INFO - copying twisted/_version.py -> build/lib.linux-x86_64-2.7/twisted
18:50:28 INFO - copying twisted/__init__.py -> build/lib.linux-x86_64-2.7/twisted
18:50:28 INFO - copying twisted/copyright.py -> build/lib.linux-x86_64-2.7/twisted
18:50:28 INFO - creating build/lib.linux-x86_64-2.7/twisted/mail
18:50:28 INFO - copying twisted/mail/mail.py -> build/lib.linux-x86_64-2.7/twisted/mail
....
.... repeated
....
19:02:53 INFO - copying twisted/internet/iocpreactor/iocpsupport/iocpsupport.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport
19:02:53 INFO - copying twisted/internet/iocpreactor/iocpsupport/winsock_pointers.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport
19:02:53 INFO - copying twisted/runner/portmap.c -> build/lib.linux-x86_64-2.7/twisted/runner
19:02:53 INFO - copying twisted/python/_epoll.c -> build/lib.linux-x86_64-2.7/twisted/python
19:02:53 INFO - copying twisted/python/sendmsg.c -> build/lib.linux-x86_64-2.7/twisted/python
19:02:53 INFO - running build_ext
19:02:53 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c conftest.c -o conftest.o
19:02:53 INFO - unable to execute gcc: No such file or directory
19:02:53 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c conftest.c -o conftest.o
19:02:53 INFO - unable to execute gcc: No such file or directory
19:02:53 INFO - building 'twisted.test.raiser' extension
19:02:53 INFO - creating build/temp.linux-x86_64-2.7
19:02:53 INFO - creating build/temp.linux-x86_64-2.7/twisted
19:02:53 INFO - creating build/temp.linux-x86_64-2.7/twisted/test
19:02:53 INFO - gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tools/python27/include/python2.7 -c twisted/test/raiser.c -o build/temp.linux-x86_64-2.7/twisted/test/raiser.o
19:02:53 INFO - unable to execute gcc: No such file or directory
19:02:53 INFO - error: command 'gcc' failed with exit status 1
19:02:53 INFO - ----------------------------------------
19:02:53 INFO - Cleaning up...
19:02:53 INFO - Command /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/bin/python -c "import setuptools, tokenize;__file__='/builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/build/twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-pplpGl-record/install-record.txt --single-version-externally-managed --compile --install-headers /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/include/site/python2.7 failed with error code 1 in /builds/slave/m-cen-lx-ntly-0000000000000000/build/venv/build/twisted
19:02:53 INFO - Storing debug log for failure in /home/cltbld/.pip/pip.log
19:02:53 WARNING - Return code: 1|
|
||
Comment 6•10 years ago
|
||
That kind of sucks! Something that would be worth trying is to hit the requirements file that I pointed out and only then try to install buildbot. I think the problem is it might try to be creating a newer version of twisted rather than the fix version that I point on that file [1]. I think I recall hitting this issue in the past. [1] https://hg.mozilla.org/build/braindump/file/d0241ce318e8/buildbot-related/pre_buildbot_master_requirements.txt#l3
You need to log in
before you can comment on or make changes to this bug.
Description
•