Closed Bug 1134767 Opened 9 years ago Closed 9 years ago

try closed - corrupt entries buildbot_schedulers database

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: jlund, Unassigned)

References

Details

Jordan Lund (:jlund)

Reporter

Description

•

9 years ago

      No description provided.

Jordan Lund (:jlund)

Reporter

Comment 1

•

9 years ago

2015-02-19 11:18:26-0800 [-] Unhandled Error
      2     Traceback (most recent call last):
      3       File "/tools/python27/lib/python2.7/threading.py", line 524, in __bootstrap
      4         self.__bootstrap_inner()
      5       File "/tools/python27/lib/python2.7/threading.py", line 551, in __bootstrap_inner
      6         self.run()
      7       File "/tools/python27/lib/python2.7/threading.py", line 504, in run
      8         self.__target(*self.__args, **self.__kwargs)
      9     --- <exception caught here> ---
     10       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/threadpool.py", line 207, in _worker
     11         result = context.call(ctx, function, *args, **kwargs)
     12       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/context.py", line 59, in callWithContext
     13         return self.currentContext().callWithContext(ctx, func, *args, **kw)
     14       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/context.py", line 37, in callWithContext
     15         return func(*args,**kw)
     16       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/enterprise/adbapi.py", line 429, in _runInteraction
     17         result = interaction(trans, *args, **kw)
     18       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/builder.py", line 513, in _claim_bu        ildreqs
     19         t)
     20       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 839, in get_unclaime        d_buildrequests
     21         for (brid,) in t.fetchall()]
     22       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 784, in getBuildRequ        estWithNumber
     23         br = self._txn_getBuildRequestWithNumber(t, brid)
     24       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 801, in _txn_getBuil        dRequestWithNumber
     25         ss = self.getSourceStampNumberedNow(ssid, t)
     26       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 534, in getSourceSta        mpNumberedNow
     27         ss = self._txn_getSourceStampNumbered(t, ssid)
     28       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 575, in _txn_getSour        ceStampNumbered
     29         for (changeid,) in r]
     30       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 392, in getChangeNum        beredNow
     31         c = self._txn_getChangeNumberedNow(t, changeid)
     32       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 425, in _txn_getChan        geNumberedNow
     33         changeid, t)
     34       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 584, in get_properti        es_from_db
     35         return self._txn_get_properties_from_db(t, tablename, idname, id)
     36       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 597, in _txn_get_pro        perties_from_db
     37         value, source = json.loads(valuepair)
     38       File "/tools/python27/lib/python2.7/json/__init__.py", line 326, in loads
     39         return _default_decoder.decode(s)
2015-02-19 11:18:26-0800 [-] Unhandled Error
      2     Traceback (most recent call last):
      3       File "/tools/python27/lib/python2.7/threading.py", line 524, in __bootstrap
      4         self.__bootstrap_inner()
      5       File "/tools/python27/lib/python2.7/threading.py", line 551, in __bootstrap_inner
      6         self.run()
      7       File "/tools/python27/lib/python2.7/threading.py", line 504, in run
      8         self.__target(*self.__args, **self.__kwargs)
      9     --- <exception caught here> ---
     10       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/threadpool.py", line 207, in _worker
     11         result = context.call(ctx, function, *args, **kwargs)
     12       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/context.py", line 59, in callWithContext
     13         return self.currentContext().callWithContext(ctx, func, *args, **kw)
     14       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/python/context.py", line 37, in callWithContext
     15         return func(*args,**kw)
     16       File "/builds/buildbot/try1/lib/python2.7/site-packages/twisted/enterprise/adbapi.py", line 429, in _runInteraction
     17         result = interaction(trans, *args, **kw)
     18       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/process/builder.py", line 513, in _claim_bu        ildreqs
     19         t)
     20       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 839, in get_unclaime        d_buildrequests
     21         for (brid,) in t.fetchall()]
     22       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 784, in getBuildRequ        estWithNumber
     23         br = self._txn_getBuildRequestWithNumber(t, brid)
     24       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 801, in _txn_getBuil        dRequestWithNumber
     25         ss = self.getSourceStampNumberedNow(ssid, t)
     26       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 534, in getSourceSta        mpNumberedNow
     27         ss = self._txn_getSourceStampNumbered(t, ssid)
     28       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 575, in _txn_getSour        ceStampNumbered
     29         for (changeid,) in r]
     30       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 392, in getChangeNum        beredNow
     31         c = self._txn_getChangeNumberedNow(t, changeid)
     32       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 425, in _txn_getChan        geNumberedNow
     33         changeid, t)
     34       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 584, in get_properti        es_from_db
     35         return self._txn_get_properties_from_db(t, tablename, idname, id)
     36       File "/builds/buildbot/try1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/db/connector.py", line 597, in _txn_get_pro        perties_from_db
     37         value, source = json.loads(valuepair)
     38       File "/tools/python27/lib/python2.7/json/__init__.py", line 326, in loads
     39         return _default_decoder.decode(s)

interestingly this is on build try masters. I haven't seen any errors on bm81 (scheduler) yet.

Ryan VanderMeulen [:RyanVM]

Comment 2

•

9 years ago

All trees are failing to schedule jobs. All are closed now.

Gregory Szorc [:gps]

Comment 3

•

9 years ago

This is my doing. I push merge commits to try in batches of 40 commits to avoid this issue. However, one of my pushes failed and the subsequent push brought in missed commits from the failed push, pushing us over the limit that will overflow the db column and result in truncated JSON, which makes buildbot barf.

I'll adjust my batch push script to not do this any more.

Jordan Lund (:jlund)

Reporter

Comment 4

•

9 years ago

I think we can resolve this jobs are pending and triggering agin.

will leave comment of what the 'fix' was after lunch.

Status: NEW → RESOLVED

Closed: 9 years ago

Resolution: --- → FIXED

Jordan Lund (:jlund)

Reporter

Comment 5

•

9 years ago

this happened again from four changeids between 16:32 -> 16:40 PT while gps was merging try

here was the fix:

get on to r/w buildbot scheduler db: https://wiki.mozilla.org/ReleaseEngineering/Buildbot_Database_Schemas

# sanity check limits:
mysql> describe change_properties;
+----------------+---------------+------+-----+---------+-------+
| Field          | Type          | Null | Key | Default | Extra |
+----------------+---------------+------+-----+---------+-------+
| changeid       | int(11)       | NO   | MUL | NULL    |       |
| property_name  | varchar(256)  | NO   |     | NULL    |       |
| property_value | varchar(1024) | NO   |     | NULL    |       |
+----------------+---------------+------+-----+---------+-------+

# see if we have commit titles that exceed that ^ 1024 limit
mysql> select changeid from change_properties where length(property_value) = 1024;
+----------+
| changeid |
+----------+
|  5355181 |
|  5355190 |
|  5355232 |
|  5355241 |
+----------+
4 rows in set (6.27 sec)

# let's see what is wrong with one of those change_properties
mysql> select * from change_properties where changeid in (5355181);
| changeid | property_name | property_value
5355181 | commit_titles | [["Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Me
rge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "
Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head",
 "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head
", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try he
ad", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try head", "Merge try hea

# oh noes, our commit title was truncated and now we have invalid syntax. We normally want the commit titles to be like [["try: -b none -t none", "Merge try head"], "BaseHgPoller"]

# let's fix up this by reducing all the commit titles to len 1 for ones that exceed 1024
mysql> update change_properties set property_value = '[["Merge try head"], "BaseHgPoller"]' where property_name = 'commit_titles' and changeid in (5355181, 5355190, 5355232, 5355241);
Query OK, 4 rows affected (0.01 sec)
Rows matched: 4  Changed: 4  Warnings: 0


# sanity check the world didn't blow up
mysql> select changeid from change_properties where length(property_value) = 1024;
Empty set (6.25 sec)
mysql> select * from change_properties where changeid in (5355181, 5355190, 5355232, 5355241);
+----------+---------------+--------------------------------+
| changeid | property_name | property_value                 |
+----------+---------------+--------------------------------+
|  5355181 | commit_titles | [["Merge try head"], "Change"] |
|  5355190 | commit_titles | [["Merge try head"], "Change"] |
|  5355232 | commit_titles | [["Merge try head"], "Change"] |
|  5355241 | commit_titles | [["Merge try head"], "Change"] |
+----------+---------------+--------------------------------+
4 rows in set (0.00 sec)

# this might not be needed but the schedulers can get into a bit of a funk, so let's kick (restart) the build and test scheduler
[reconfig-virtual-env]jlund@Hastings163:~BUC/tools/buildfarm/maintenance
> python manage_masters.py -f production-masters.json -R scheduler restart

BMO Automation

Updated

•

6 years ago

Product: Release Engineering → Infrastructure & Operations

BMO Automation

Updated

•

4 years ago

Product: Infrastructure & Operations → Infrastructure & Operations Graveyard

You need to log in before you can comment on or make changes to this bug.

Bugzilla

Quick Search

try closed - corrupt entries buildbot_schedulers database

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Tracking

(Not tracked)

People

(Reporter: jlund, Unassigned)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Updated

Updated