948426 - Random failed transactions for http://hg.mozilla.org/

Reporter

Description

•

11 years ago

In an effort to benchmark hg.mozilla.org for an upcoming blog post, I've ran siege against hg.mozilla.org. Surprisingly, we have some transient failed transactions for the static URL http://hg.mozilla.org/mozilla-central

The 7 failed transactions are what concern me. These could be the cause of all the transient unreplicatable 'hg is broken' issues we have so many bugs open about.

bkero@bkero-general:~$ siege http://hg.mozilla.org/mozilla-central
** SIEGE 3.0.1
** Preparing 15 concurrent users for battle.
The server is now under siege...^C
Lifting the server siege...      done.
 
Transactions:                    712 hits
Availability:                  99.03 %
Elapsed time:                  32.41 secs
Data transferred:               3.35 MB
Response time:                  0.20 secs
Transaction rate:              21.97 trans/sec
Throughput:                     0.10 MB/sec
Concurrency:                    4.29
Successful transactions:         712
Failed transactions:               7
Longest transaction:            0.46
Shortest transaction:           0.10

Ben Kero [:bkero]

Reporter

Comment 1

•

11 years ago

Running siege directly against an hgweb head shows we cannot find this problem.

Perhaps it is a problem somewhere with Zeus?

[bkero@boris bin]$ ./siege -H "Host: hg.mozilla.org" http://hgweb4.dmz.scl3.mozilla.com/mozilla-central
** SIEGE 2.72
** Preparing 15 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:		        1464 hits
Availability:		      100.00 %
Elapsed time:		       66.38 secs
Data transferred:	        6.88 MB
Response time:		        0.19 secs
Transaction rate:	       22.05 trans/sec
Throughput:		        0.10 MB/sec
Concurrency:		        4.17
Successful transactions:        1464
Failed transactions:	           0
Longest transaction:	        3.17
Shortest transaction:	        0.12

Ben Kero [:bkero]

Reporter

Comment 2

•

11 years ago

After running siege tests against hg.mozilla.org for different parts of the day, this seems to only be happening at certain times.

Sometimes I will run siege against the service with no failed transactions for 15 minutes, and sometimes it will result in hundreds of errors.

Tomcat notes on IRC that releng experienced this again today.

Ben Kero [:bkero]

Reporter

Comment 3

•

11 years ago

Tailing all the webheads for errors yields some such as this in apache error logs:

[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212] mod_wsgi (pid=21381): Exception occurred processing WSGI script '/repo/hg/webroot_wsgi/hgweb.wsgi'.
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212] Traceback (most recent call last):
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 153, in __call__
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     return self.run_wsgi(req)
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 217, in run_wsgi
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     return hgweb(repo).run_wsgi(req)
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgweb_mod.py", line 240, in run_wsgi
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     content = getattr(webcommands, cmd)(self, req, tmpl)
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/webcommands.py", line 479, in summary
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     i = reversed(web.repo.tagslist())
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 607, in tagslist
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     for t, n in self.tags().iteritems():
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 553, in tags
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     self.changelog.rev(v)
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/changelog.py", line 184, in rev
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     r = super(changelog, self).rev(node)
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 290, in rev
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212]     return self._nodecache[node]
[Wed Dec 18 07:00:49 2013] [error] [client 10.22.74.212] RevlogError

Ben Kero [:bkero]

Reporter

Comment 4

•

11 years ago

and another:

[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212] mod_wsgi (pid=19430): Exception occurred processing WSGI script '/repo/hg/webroot_wsgi/integration/hgweb.wsgi'.
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212] Traceback (most recent call last):
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 153, in __call__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return self.run_wsgi(req)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 217, in run_wsgi
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return hgweb(repo).run_wsgi(req)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgweb_mod.py", line 240, in run_wsgi
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     content = getattr(webcommands, cmd)(self, req, tmpl)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/repo/hg/extensions/pushlog-feed.py", line 469, in pushes
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return tmpl('pushes', data=pushes_worker(query, 'full' in req.form and web))
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/repo/hg/extensions/pushlog-feed.py", line 446, in pushes_worker
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     ctx = web.repo.changectx(node)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 700, in changectx
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return self[changeid]
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 385, in __getitem__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return context.changectx(self, changeid)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 76, in __init__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     self._rev = repo.changelog.rev(self._node)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/repoview.py", line 170, in changelog
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     revs = filterrevs(unfi, self.filtername)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/repoview.py", line 117, in filterrevs
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     repo.filteredrevcache[filtername] = func(repo.unfiltered())
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/repoview.py", line 48, in computeunserved
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     if phases.hassecret(repo):
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/phases.py", line 400, in hassecret
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return bool(repo._phasecache.phaseroots[2])
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 27, in __get__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return super(repofilecache, self).__get__(repo.unfiltered(), type)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/scmutil.py", line 970, in __get__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     entry = filecacheentry(path)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/scmutil.py", line 876, in __init__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     self.cachestat = filecacheentry.stat(self.path)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/scmutil.py", line 919, in stat
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     return util.cachestat(path)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]   File "/usr/lib64/python2.6/site-packages/mercurial/posix.py", line 489, in __init__
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212]     self.stat = os.stat(path)
[Wed Dec 18 07:12:29 2013] [error] [client 10.22.74.212] OSError: [Errno 116] Stale file handle: '/repo/hg/mozilla/integration/mozilla-inbound/.hg/store/phaseroots'

Ben Kero [:bkero]

Reporter

Updated

•

11 years ago

Blocks: 819405

Ben Kero [:bkero]

Reporter

Comment 5

•

10 years ago

I'm at the mercurial hackathon and have spoken with one of the mercurial developers (brodie) at Atlassian. He confirmed he's seen this "Stale file handle" problem before, and it was fixed by upgrading the client kernel.

His currently running kernel (with the fix) is 2.6.32-431.1.2.0.1.el6.x86_64

Our currently running kernel is 2.6.32-358.6.2.el6.x86_64

At this point I'd like to failover to the hgssh2 box, then upgrade and reboot. There should be no visible outage to any users.

Matt Mackall

Comment 6

•

10 years ago

The first trace here is caused by a bad tag while browsing hgweb and should not be correlated with pull failures. Also should be fixed?

http://bz.selenic.com/show_bug.cgi?id=3971

Hal Wine [:hwine] use NI!

Assignee

Updated

•

10 years ago

Assignee: server-ops-devservices → hwine

Blocks: 945383

Hal Wine [:hwine] use NI!

Assignee

Comment 7

•

10 years ago

I'm confused here: comment 1 implies this replicates only when access via load balancer, comment 3 and comment 4 imply issue is visible in logs. What's the current hypothesis?

If this is visible on a node, can we replicate in staging and demonstrate the kernel upgrade fixes things?

Also, this bug was referenced in #releng today with respect to intermittent ISE 500 errors observed while fetching bzipped archives for the FFx 30 release tag by :glandium. Can we add that data to this bug, or open another bug with that data?

Flags: needinfo?(bkero)

Ben Kero [:bkero]

Reporter

Comment 8

•

10 years ago

Initial investigations led me to believe it was on the load balancer. Further investigation revealed that it was happening on the hgweb hosts themselves.

I've been able to replicate this issue using siege (with the incantation ./siege -v http://hostname:8000/FIREFOX_31_0_RELEASE.tar.bz2 ) on several different hosts across RHEL5/6 and across Mercurial versions 2.2.2, 2.5.4, and 3.0.1 with tracebacks to follow. I posted to the #mercurial channel yesterday evening and a dev suggested that I file an upstream bug about it.

All HTTP service was performed with 'hg serve'

(FROM RHEL6 AND HG v3.0.1)
    10.2.10.5 - - [11/Jun/2014 00:47:24] Exception happened during processing request '/FIREFOX_31_0_RELEASE.tar.bz2':
    Traceback (most recent call last):
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 77, in do_POST
        self.do_write()
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 70, in do_write
        self.do_hgweb()
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 137, in do_hgweb
        for chunk in self.server.application(env, self._start_response):
      File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 327, in increasingchunks
        for chunk in source:
      File "/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 315, in _flatten
        for j in _flatten(i):
      File "/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 308, in _flatten
        for i in thing:
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/webcommands.py", line 532, in changelist
        author=ctx.user(),
      File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 191, in user
        return self._changeset[1]
      File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 246, in __get__
        result = self.func(obj)
      File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 146, in _changeset
        return self._repo.changelog.read(self.rev())
      File "/usr/lib64/python2.6/site-packages/mercurial/changelog.py", line 282, in read
        text = self.revision(node)
      File "/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 919, in revision
        text = self._cache[2]
    TypeError: 'NoneType' object is unsubscriptable 


(FROM PRODUCTION HGWEB HOST USING HG v3.0.1)
    10.2.10.5 - - [11/Jun/2014 01:01:00] Exception happened during processing request '/FIREFOX_31_0_RELEASE.tar.bz2':
    Traceback (most recent call last):
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 77, in do_POST
        self.do_write()
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 70, in do_write
        self.do_hgweb()
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 137, in do_hgweb
        for chunk in self.server.application(env, self._start_response):
      File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 327, in increasingchunks
        for chunk in source:
      File "/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 315, in _flatten
        for j in _flatten(i):
      File "/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 308, in _flatten
        for i in thing:
      File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/webcommands.py", line 532, in changelist
        author=ctx.user(),
      File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 191, in user
        return self._changeset[1]
      File "/usr/lib64/python2.6/site-packages/mercurial/util.py", line 246, in __get__
        result = self.func(obj)
      File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 146, in _changeset
        return self._repo.changelog.read(self.rev())
      File "/usr/lib64/python2.6/site-packages/mercurial/changelog.py", line 282, in read
        text = self.revision(node)
      File "/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 931, in revision
        text = self._checkhash(text, node, rev)
      File "/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 940, in _checkhash
        % (self.indexfile, rev))
    RevlogError: integrity check failed on 00changelog.i:2205 


(Version 2.2.2 on RHEL5)
    127.0.0.1 - - [10/Jun/2014 18:03:09] Exception happened during processing request '/FIREFOX_31_0_RELEASE.tar.bz2':
    Traceback (most recent call last):
      File "/usr/lib/python2.4/site-packages/mercurial/hgweb/server.py", line 77, in do_POST
        self.do_write()
      File "/usr/lib/python2.4/site-packages/mercurial/hgweb/server.py", line 70, in do_write
        self.do_hgweb()
      File "/usr/lib/python2.4/site-packages/mercurial/hgweb/server.py", line 137, in do_hgweb
        for chunk in self.server.application(env, self._start_response):
      File "/usr/lib/python2.4/site-packages/mercurial/util.py", line 315, in increasingchunks
        for chunk in source:
      File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 217, in _flatten
        for j in _flatten(i):
      File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 210, in _flatten
        for i in thing:
      File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 161, in runmap
        for i in d:
      File "/usr/lib/python2.4/site-packages/mercurial/hgweb/webcommands.py", line 204, in changelist
        l.insert(0, {"parity": parity.next(),
      File "/usr/lib/python2.4/site-packages/mercurial/hgweb/webutil.py", line 110, in nodeinbranch
        if branch != 'default' and repo.branchtags().get(branch) != ctx.node():
      File "/usr/lib/python2.4/site-packages/mercurial/localrepo.py", line 520, in branchtags
        for bn, heads in self.branchmap().iteritems():
    AttributeError: 'NoneType' object has no attribute 'iteritems' 


(FROM PRODUCTION HGWEB HOST USING HG v3.0.1)
    Traceback (most recent call last):
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 80, in do_POST
        self.do_write()
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 73, in do_write
        self.do_hgweb()
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/hgweb/server.py", line 140, in do_hgweb
        for chunk in self.server.application(env, self._start_response):
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/util.py", line 369, in increasingchunks
        for chunk in source:
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 517, in _flatten
        for j in _flatten(i):
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 510, in _flatten
        for i in thing:
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/templater.py", line 193, in runmap
        for i in d:
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/hgweb/webcommands.py", line 606, in branches
        'date': web.repo[n].date()}
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/context.py", line 357, in date
        return self._changeset[2]
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/util.py", line 287, in __get__
        result = self.func(obj)
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/context.py", line 332, in _changeset
        return self._repo.changelog.read(self.rev())
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/changelog.py", line 284, in read
        text = self.revision(node)
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 1012, in revision
        bins = self._chunks(chain)
      File "/root/hg/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 934, in _chunks
        ladd(decompress(buffer(data, chunkstart - offset, chunklength)))
    ValueError: offset must be zero or positive


(FROM PRODUCTION HGWEB INSTALLATIONS, v2.5.4, RHEL6)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5] mod_wsgi (pid=5686): Exception occurred processing WSGI script '/repo_local/mozilla/webroot_wsgi/releases/l10n/mozilla-release/hgweb.wsgi'.
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5] Traceback (most recent call last):
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 153, in __call__
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     return self.run_wsgi(req)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 217, in run_wsgi
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     return hgweb(repo).run_wsgi(req)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/hgweb_mod.py", line 240, in run_wsgi
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     content = getattr(webcommands, cmd)(self, req, tmpl)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/hgweb/webcommands.py", line 804, in archive
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     cnode = web.repo.lookup(key)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 662, in lookup
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     return self[key].node()
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/localrepo.py", line 385, in __getitem__
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     return context.changectx(self, changeid)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/context.py", line 53, in __init__
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     self._rev = repo.changelog.rev(changeid)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/changelog.py", line 184, in rev
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     r = super(changelog, self).rev(node)
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]   File "/usr/lib64/python2.6/site-packages/mercurial/revlog.py", line 290, in rev
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5]     return self._nodecache[node]
    [Wed Jun 11 00:30:57 2014] [error] [client 10.2.10.5] RevlogError

Flags: needinfo?(bkero)

Ben Kero [:bkero]

Reporter

Comment 9

•

10 years ago

The preceding tests were performed in diagnosis of the ISE 500 issues glandium was experiencing yesterday. They are copied verbatim from pastebin, which in turn was copied verbatim from terminals running 'hg serve' on the mentioned hosts.

Some of these tests were done on reviewboard1.dev.webapp.scl3.mozilla.com as it was a convenient non-production host that had hg version 3.0.1 installed, and some of them done on Boris, which has rhel5 and hg v2.2.2.

Ben Kero [:bkero]

Reporter

Comment 10

•

10 years ago

I replicated the issue on my workstation and received yet another traceback (this time using hg's own mercurial-repo instead of nb-NO).

127.0.0.1 - - [11/Jun/2014 11:42:52] Exception happened during processing request '/FIREFOX_31_0_RELEASE.tar.bz2':
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/server.py", line 80, in do_POST
    self.do_write()
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/server.py", line 73, in do_write
    self.do_hgweb()
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/server.py", line 140, in do_hgweb
    for chunk in self.server.application(env, self._start_response):
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/hgweb_mod.py", line 136, in __call__
    return self.run_wsgi(req)
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/hgweb_mod.py", line 249, in run_wsgi
    content = getattr(webcommands, cmd)(self, req, tmpl)
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/webcommands.py", line 347, in shortlog
    return changelog(web, req, tmpl, shortlog=True)
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/webcommands.py", line 331, in changelog
    entries = list(changelist())
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/webcommands.py", line 307, in changelist
    "inbranch": webutil.nodeinbranch(web.repo, ctx),
  File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/webutil.py", line 175, in nodeinbranch
    branchnode = repo.branchtip(branch)
  File "/usr/lib/python2.7/dist-packages/mercurial/localrepo.py", line 671, in branchtip
    return self._branchtip(self.branchmap()[branch])
  File "/usr/lib/python2.7/dist-packages/mercurial/localrepo.py", line 662, in _branchtip
    if not self[h].closesbranch():
  File "/usr/lib/python2.7/dist-packages/mercurial/context.py", line 347, in closesbranch
    return 'close' in self._changeset[5]
  File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 281, in __get__
    result = self.func(obj)
  File "/usr/lib/python2.7/dist-packages/mercurial/context.py", line 314, in _changeset
    return self._repo.changelog.read(self.rev())
  File "/usr/lib/python2.7/dist-packages/mercurial/changelog.py", line 283, in read
    text = self.revision(node)
  File "/usr/lib/python2.7/dist-packages/mercurial/revlog.py", line 924, in revision
    cachedrev = self._cache[1]
TypeError: 'NoneType' object has no attribute '__getitem__'

Ed Morley [:emorley]

Comment 11

•

10 years ago

Is this an instance of it?
https://tbpl.mozilla.org/php/getParsedLog.php?id=41599552&tree=B2g-Inbound

Ben Kero [:bkero]

Reporter

Comment 12

•

10 years ago

Upstream bugs filed here:
http://bz.selenic.com/show_bug.cgi?id=4273
http://bz.selenic.com/show_bug.cgi?id=4274
http://bz.selenic.com/show_bug.cgi?id=4275
http://bz.selenic.com/show_bug.cgi?id=4276
http://bz.selenic.com/show_bug.cgi?id=4277
http://bz.selenic.com/show_bug.cgi?id=4278
http://bz.selenic.com/show_bug.cgi?id=4280

Hal Wine [:hwine] use NI!

Assignee

Updated

•

10 years ago

See Also: → http://bz.selenic.com/show_bug.cgi?id=4273, http://bz.selenic.com/show_bug.cgi?id=4274, http://bz.selenic.com/show_bug.cgi?id=4275, http://bz.selenic.com/show_bug.cgi?id=4276, http://bz.selenic.com/show_bug.cgi?id=4277, http://bz.selenic.com/show_bug.cgi?id=4278, http://bz.selenic.com/show_bug.cgi?id=4280

Kendall Libby [:fubar] (he/him)

Updated

•

10 years ago

Component: Server Operations: Developer Services → Mercurial: hg.mozilla.org

Product: mozilla.org → Developer Services

:kanban-engops

Updated

•

10 years ago

Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/61]

:kanban-engops

Updated

•

10 years ago

Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/61] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/871] [kanban:engops:https://kanbanize.com/ctrl_board/6/61]

:kanban-engops

Updated

•

10 years ago

Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/871] [kanban:engops:https://kanbanize.com/ctrl_board/6/61] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/875] [kanban:engops:https://kanbanize.com/ctrl_board/6/61]

:kanban-engops

Updated

•

10 years ago

Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/875] [kanban:engops:https://kanbanize.com/ctrl_board/6/61] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/876] [kanban:engops:https://kanbanize.com/ctrl_board/6/61]

Nobody; OK to take it and work on it

Updated

•

10 years ago

Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/876] [kanban:engops:https://kanbanize.com/ctrl_board/6/61] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/876]

Ben Kero [:bkero]

Reporter

Comment 13

•

9 years ago

All bugs fixed upstream.

Status: NEW → RESOLVED

Closed: 9 years ago

Resolution: --- → FIXED

Bugzilla

Quick Search

Random failed transactions for http://hg.mozilla.org/

Categories

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Tracking

(Not tracked)

People

(Reporter: bkero, Assigned: hwine)

References

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/876] )

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Comment 4

Updated

Comment 5

Comment 6

Updated

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Updated

Updated

Updated

Updated

Updated

Updated

Updated

Comment 13