John Hopkins (:jhopkins) Reporter
	Description • 12 years ago

To avoid or lessen the need for try repo resets, we should schedule a daily or weekly merge of try repo heads.
The basic merge script is at https://bug733070.bugzilla.mozilla.org/attachment.cgi?id=604040.  One caveat is it did munge .hgtags in one commit which I had to undo, but otherwise worked fine.

	Chris AtLee [:catlee]
	Comment 1 • 12 years ago

nice work!

Before deploying this, I feel like we should get some more feedback to see if this will cause anybody problems. Can you reach out to other developers, maybe via a newsgroup or blog post, to check if there are any objections to doing this?

	John Hopkins (:jhopkins) Reporter
	Comment 2 • 12 years ago

This updated version detects ancestor heads which can't be merged but increased the runtime noticeably.  There may be a better way to handle this case.

	(not currently active) Ted Mielczarek
	Comment 3 • 12 years ago

Does simply closing the old heads have the same benefit?

	(no longer active)
	Comment 4 • 12 years ago

(In reply to Ted Mielczarek [:ted] from comment #3)
> Does simply closing the old heads have the same benefit?

IIRC, no.  I once tested it.  hg is just broken with the way it handles heads when pulling/cloning, and closing a branch doesn't mean that it cannot be pulled/cloned.

	Jesse Ruderman
	Comment 5 • 12 years ago

Can we hide these merges from TBPL?

	(no longer active)
	Comment 6 • 12 years ago

(In reply to Jesse Ruderman from comment #5)
> Can we hide these merges from TBPL?

If we denote them somehow using a tag or something, for sure.  If we use a tag, we should use a naming pattern and not a single name.  Something like mergeheads-n.

	Steve Fink [:sfink] [:s:]
	Comment 7 • 12 years ago

I chased this through a little bit. I don't know exactly where these slow things down, so I don't know if simply hiding them on tbpl is good enough. It would be good to avoid these are far back in the pipeline as possible, but I don't know all the layers involved.

How about this: in PushlogJSONParser.js, do

  var url = "https://hg.mozilla.org/" + repoName + "/json-pushes?full=1&ignore_user=try-decapitator@mozilla.org";

and then implement an ignore_user parameter in hgpoller.py. (I briefly looked at its source, and perhaps you'd need to push it back further into the RSS feed generator it seems to sit atop, but I won't go there.)

Then of course, you'd need to commit the merges as the try-decapitator user.

	(no longer active)
	Comment 8 • 12 years ago

(In reply to Steve Fink [:sfink] from comment #7)
> I chased this through a little bit. I don't know exactly where these slow
> things down, so I don't know if simply hiding them on tbpl is good enough.
> It would be good to avoid these are far back in the pipeline as possible,
> but I don't know all the layers involved.

hg sucks at handling trees with lots of heads when pulling/pushing, because it looks at all of them in order to determine what needs to be pulled/pushed.

> How about this: in PushlogJSONParser.js, do
> 
>   var url = "https://hg.mozilla.org/" + repoName +
> "/json-pushes?full=1&ignore_user=try-decapitator@mozilla.org";
> 
> and then implement an ignore_user parameter in hgpoller.py. (I briefly
> looked at its source, and perhaps you'd need to push it back further into
> the RSS feed generator it seems to sit atop, but I won't go there.)
> 
> Then of course, you'd need to commit the merges as the try-decapitator user.

That seems like a much better idea than mine!

	John Hopkins (:jhopkins) Reporter
	Comment 9 • 12 years ago

(In reply to Ehsan Akhgari [:ehsan] from comment #8)
> hg sucks at handling trees with lots of heads when pulling/pushing, because
> it looks at all of them in order to determine what needs to be pulled/pushed.

Also worth noting that hg spends a non-trivial amount of time doing that processing.  In my tests, cloning a 2500 head try repo took 15m but when I shrunk that same repo down to 1500 heads it took only 9m (to/from local disk).

Another approach to hiding the head merges is to omit checkins that don't result in any builds.  This may be better, in fact, since if the try-decapitator user ever _did_ kick off builds I'd want to know about it.

	Steve Fink [:sfink] [:s:]
	Comment 10 • 12 years ago

(In reply to Ehsan Akhgari [:ehsan] from comment #8)
> (In reply to Steve Fink [:sfink] from comment #7)
> > I chased this through a little bit. I don't know exactly where these slow
> > things down, so I don't know if simply hiding them on tbpl is good enough.
> > It would be good to avoid these are far back in the pipeline as possible,
> > but I don't know all the layers involved.
> 
> hg sucks at handling trees with lots of heads when pulling/pushing, because
> it looks at all of them in order to determine what needs to be pulled/pushed.

Oh, sorry. I didn't mean to imply that we should leave the heads in the repo and just hide things from tbpl, though rereading what I wrote makes it sound like that. I am assuming that the head merging will be done. I'm talking about what really should be a followup bug, to prevent problems when tbpl users view a time window including one of these merge pushes with a kajillion changesets in it. I don't know if it's ok to fetch all those changesets and then drop them, or if we need to prevent them from ever being rendered, transmitted, and parsed.

(In reply to John Hopkins (:jhopkins) from comment #9)
> Another approach to hiding the head merges is to omit checkins that don't
> result in any builds.  This may be better, in fact, since if the
> try-decapitator user ever _did_ kick off builds I'd want to know about it.

Do we know about jobs early enough (eg in hgpoller.py)? I was trying to avoid parsing the hgpoller.py json at all.

	John Hopkins (:jhopkins) Reporter
	Comment 11 • 12 years ago

(In reply to Steve Fink [:sfink] from comment #10)

Agreed - let's treat hiding of the merge changes as a separate, followup bug.  Since nobody has flagged this as a blocker, I believe we are ok to proceed with the work to periodically merge try repo heads.  Thanks, all!

	Steve Fink [:sfink] [:s:]
	Comment 12 • 12 years ago

The only part still relevant to this bug is the suggestion that having a specific user (eg try-decapitator) for merging the heads would be useful for the followup, and agreement on the exact name of that user could be considered to block the followup.

	Axel Hecht
	Comment 13 • 12 years ago

Curious. Would it help if the changeset we merge to was on another branch than default?

	Chris AtLee [:catlee]
	Comment 14 • 12 years ago

(In reply to Axel Hecht from comment #13)
> Curious. Would it help if the changeset we merge to was on another branch
> than default?

help how? we build off-default branches on try currently, although we could explicitly ignore a certain branch...or even pushes by a certain user for that matter.

	John Hopkins (:jhopkins) Reporter
	Comment 15 • 12 years ago

fyi, I am running a manual head merge (using the attached script) and will push it tonight or tomorrow when the script is done.

	Hal Wine [:hwine] use NI! Assignee
	Comment 16 • 12 years ago

Some random notes from the manual merge in bug 761780:
 - when hg repo really big, must do pulls via ssh
 - head merge removing 2208 heads took about 14 hours on a fast box (dedicated CPU, SSD)

	Hal Wine [:hwine] use NI! Assignee
	Comment 17 • 12 years ago

Remove approx 320 heads on Wed Jun 13, however that seems not to have helped much:
(In reply to Boris Zbarsky (:bz) from bug 761780 comment #6)
> Hmm.  So I just tried timing a try push today (for the first time since this
> bug was fixed), and it took 4m45s wall-clock...  :(  Not sure whether we
> have lots of heads again or whether something else is up.

	Boris Zbarsky [:bzbarsky]
	Comment 18 • 12 years ago

Is it possible that the time for my push was being gated by something other than number of heads?

	Hal Wine [:hwine] use NI! Assignee
	Comment 19 • 12 years ago

Yes, but we don't currently collect any metrics on anything else, which is why I reopened bug 691459 to obtain additional information.

	Dustin J. Mitchell [:dustin] (he/him)
	Comment 20 • 12 years ago

Ben, any thoughts here?

	Hal Wine [:hwine] use NI! Assignee
	Comment 21 • 12 years ago

(In reply to Steve Fink [:sfink] from comment #12)
> The only part still relevant to this bug is the suggestion that having a
> specific user (eg try-decapitator) for merging the heads would be useful for
> the followup, and agreement on the exact name of that user could be
> considered to block the followup.

We now have such a user: trylander@mozilla.com (cf bug 763601; bug 765396)

	Mark Banner (:standard8)
	Comment 22 • 12 years ago

Please can this be done for the try-comm-central repository as well as the try repo. Same rules apply to both, so it doesn't seem to need a separate bug.

	Hal Wine [:hwine] use NI! Assignee
	Comment 24 • 12 years ago

The auto merge of try heads was stopped after the most recent reset of try (bug 778062), as the merges aggravated the "hg web can't display summary of deep heads" bug (discussed in another bug thread I can't find at the moment).

I.e. the "auto merge" made things worse than the occasional manual reset of the try repo.

We have all the code to do it, should deploying a newer version of hg make an auto prune a winning pay off again.