Ben Bucksch (:BenB) Assignee
	Description • 10 years ago

For a certain mail checker, we need to allow the end user to log in to webmail.

Scenario:
1. User enters username and password at mail check setup
2. Mail checker polls POP3 or IMAP
3. New mail arrives. mail checker notifies user.
4. User wants to go to the new mail.
5. Mailchecker wants to offer a "go to webmail" button that automatically logs the user in, with the already known credentials

For this, we need to know the webmail login: URL and how to submit username/password.

We have collected URLs for most providers in our ISPDB.
I would like to add them to the ISPDB XML files.

	Ben Bucksch (:BenB) Assignee
	Comment 1 • 10 years ago

This was compiled by a colleague of mine.

	Ben Bucksch (:BenB) Assignee
	Comment 2 • 10 years ago

I've compiled the biggest ones that we need first.
This also contains the login field IDs, so that we can fill it in automatically with the information we already have.

	Ben Bucksch (:BenB) Assignee
	Comment 3 • 10 years ago

Blake, Gozer, what do you think about this?

I could understand when you say this has nothing to do with Thunderbird.

I could also understand when you say that some admins might not like this, say that this is a security problem. I don't think it is, the problem is when I have the password, the form is public anyway (and I got this information merely by going to their website), but it wouldn't surprise me when some people started to FUSS and FUD about this.

On the positive side, the ISPDB was supposed to be a generic database, not just for Thunderbird, and this is definitely useful information. The mail checker uses the ISPDB with success, and having the information in there would be useful and convenient.

	Andrew Sutherland [:asuth] (he/him)
	Comment 4 • 10 years ago

From the perspective of the Gaia e-mail app, at least the login URL aspect could be potentially useful.  Specifically, there's a family of scenarios that go like this:
- User is in a new part of the world
- Mail provider is fancy and has a "suspicious login" detector, finds IP suspicious, requires user to login via webmail to prove it's them, possibly by using a more complicated login process.  Of course, often time the user just logs in with the same credentials, so it's silly, but hey.
 
This definitely happens for gmail, but they also include a URL in the error message that includes some explanation of what's going on, so it's also not that hard to just regexp that out and/or show the entire error message with auto-linkification enabled.

Being able to send the user to a login URL as a fallback seems potentially useful, especially in the more nebulous failure cases.  We could say "Something bad seems to be going on, we have no idea, maybe try logging in via webmail."

The automatic DOM stuff seems more likely to break.  An argument against that would depend on whether or not the unspecified mail checker (can you specify it?) polls the ISPDB server and at what rate and what efforts are made to avoid emergent DoS attacks on the server infrastructure.  Like if the app checked every hour on the hour, that would be a big/bad change for the ISPDB.  And even if the certain checker didn't do it, hypothetically it would encourage other checkers to do so in a bad way.  The counter-argument is eh, if we find checkers doing that, just tell them to only hit the ISPDB if they ever have a login fail, and have a back-off on that to determine if that changes.

	Blake Winton (:bwinton) (:☕️)
	Comment 5 • 10 years ago

I don't object to the idea, but I think we should bikeshed a little about the format of the data in the XML.

Do we want to restrict ourselves to only adding pages that have ids for their username/password/loginButton, or should that be a css selector?

What does it mean to have both a usernameFieldID and a usernameFieldName?

	Blake Winton (:bwinton) (:☕️)
	Comment 6 • 10 years ago

(In reply to Blake Winton (:bwinton) from comment #5)
> I don't object to the idea, but I think we should bikeshed a little about
> the format of the data in the XML.
> 
> Do we want to restrict ourselves to only adding pages that have ids for
> their username/password/loginButton, or should that be a css selector?
> 
> What does it mean to have both a usernameFieldID and a usernameFieldName?

I'm also a little concerned about keeping it up to date, since I suspect login pages will change more frequently than server configurations…

	Ben Bucksch (:BenB) Assignee
	Comment 7 • 10 years ago

Test server:
​http://www.mara.beonex.com/ispdb/
E.g.
​http://www.mara.beonex.com/ispdb/t-online.de
​http://www.mara.beonex.com/ispdb/hotmail.com
​http://www.mara.beonex.com/ispdb/yahoo.com
​http://www.mara.beonex.com/ispdb/googlemail.com
(no secondary domains)

	Ben Bucksch (:BenB) Assignee
	Comment 8 • 10 years ago

> Do we want to restrict ourselves to only adding pages that have ids for their username/password/loginButton, or should that be a css selector?
> What does it mean to have both a usernameFieldID and a usernameFieldName?

I'm weary about CSS selectors, because the XML comes from the network, and when you feed this unchecked to jQuery, you might run code. An ID or name is much easier to check, esp. given that all I've seen so far match [a-zA-Z0-9\-\_]+, no spaces.

I've came across 2 sites have that no ID on the fields, but had a name="" attribute. hotmail has an ID that looks generated and a name that looks manually chosen. I'd try the ID first, and if that fails, try the name.

I threw in the CSS selector just in case some site is nasty. We currently don't need it.

Blake, if you have a better proposal for the XML format, without overengineering it, please suggest it. We could make it:
<loginAutomaticDOM url="https://hotmail.com">
  <usernameForm>%EMAILADDRESS%</usernameForm>
  <usernameField id="username" name="username" />
  <passwordField name="passwd" />
  <loginButton id="submit" name="login" />
</loginAutomaticDOM>
I'd like that. Would probably be even easier to program when using it.

	Blake Winton (:bwinton) (:☕️)
	Comment 9 • 10 years ago

I'm pretty sure we won't run code if we use document.querySelector instead of jQuery.
Also, how were you _thinking_ of getting to the element from the ID?  Some sort of regex on the page content?!?  ;)

	Ben Bucksch (:BenB) Assignee
	Comment 10 • 10 years ago

> on whether or not the unspecified mail checker ... polls the ISPDB server and at what rate and what efforts are made to avoid emergent DoS attacks on the server infrastructure. Like if the app checked every hour on the hour, that would be a big/bad change for the ISPDB.

It does the same as Thunderbird, fetches the config at setup and then saves it locally in prefs.

> if we find checkers doing that, just tell them to only hit the ISPDB if they ever have a login fail, and have a back-off on that to determine if that changes.

Right. Good idea.

	Ben Bucksch (:BenB) Assignee
	Comment 11 • 10 years ago

>   <usernameForm>%EMAILADDRESS%</usernameForm>

Actually, make that plain <username>, because this matches the <incomingServer> section.
I think the field access is something different, so that should be in a different element.

	Ben Bucksch (:BenB) Assignee
	Comment 12 • 10 years ago

> I'm pretty sure we won't run code if we use document.querySelector instead of jQuery.
> Also, how were you _thinking_ of getting to the element from the ID?

getElementByID()
Even if I use querySelector(), I'd still check the input first that it only contains the above characters. It's a lot harder to protect when [ and ' and = and whatnot are allowed inputs.

	Philippe M. Chiasson (:gozer)
	Comment 13 • 10 years ago

(In reply to Ben Bucksch (:BenB) from comment #3)
> Blake, Gozer, what do you think about this?

Could you explain a little bit the implication of an entry having a loginPage but *no* loginAutomaticDOM?

> I could understand when you say this has nothing to do with Thunderbird.

But it has a lot to do about e-mail ISP configuration, so I think it arguably
belongs here.

> I could also understand when you say that some admins might not like this,
> say that this is a security problem. I don't think it is, the problem is
> when I have the password, the form is public anyway (and I got this
> information merely by going to their website), but it wouldn't surprise me
> when some people started to FUSS and FUD about this.

We could *only* approve https:// URLs as a starting policy, for instance.

> On the positive side, the ISPDB was supposed to be a generic database, not
> just for Thunderbird, and this is definitely useful information. The mail
> checker uses the ISPDB with success, and having the information in there
> would be useful and convenient.

I agree.

	Ben Bucksch (:BenB) Assignee
	Comment 14 • 10 years ago

> Could you explain a little bit the implication of an entry having a loginPage
> but *no* loginAutomaticDOM?

The application would load loginPage in the browser, where the user has to manually enter the username and password.

> We could *only* approve https:// URLs as a starting policy, for instance.

Yes. In fact, I'd add that to the spec, explicitly stating that only https is valid and http is not. I thought I did that. I did it now.

	Ben Bucksch (:BenB) Assignee
	Comment 15 • 10 years ago

I fixed the format based on bwinton's suggestions (at least parts of them).
Example:
    <webMail>
      <loginPage url="https://mail.example.com/login/" />
      <loginAutomaticDOM url="https://mail.example.com/login/">
        <username>%EMAILADDRESS%</username>
        <usernameField id="email_field" name="email" />
        <passwordField name="password" />
        <loginButton id="submit_button" name="login"/>
      </loginAutomaticDOM>
    </webMail>
For spec, see https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat#XML

	Ben Bucksch (:BenB) Assignee
	Comment 16 • 10 years ago

<username> is now identical to <username> within <incomingServer>

	Ben Bucksch (:BenB) Assignee
	Comment 18 • 10 years ago

bwinton: ping

Correction: Gmail URL should be:
​https://accounts.google.com/ServiceLogin?service=mail&continue=http://mail.google.com/mail/

	Ben Bucksch (:BenB) Assignee
	Comment 19 • 10 years ago

Gmail URL fixed

	Blake Winton (:bwinton) (:☕️)
	Comment 20 • 10 years ago

Comment on attachment 8377281 [details] [diff] [review]
As XML, v2 - with new elements format

Review of attachment 8377281 [details] [diff] [review]:
-----------------------------------------------------------------

(r=me, but see comments below.)

	Blake Winton (:bwinton) (:☕️)
	Comment 21 • 10 years ago

Comment on attachment 8386450 [details] [diff] [review]
As XML, v3 - with new elements format

Okay, so r-me, but "loginAutomaticDOM" looks strange, so I propose we change it to "autoLoginInfo", or "loginInfo", or "loginPageInfo" (whichever you prefer).

	Ben Bucksch (:BenB) Assignee
	Comment 22 • 10 years ago

<loginPageInfo> it is, then.

	Ben Bucksch (:BenB) Assignee
	Comment 23 • 10 years ago

As requested by bwinton in comment 21 + 22, renamed <loginAutomaticDOM> to <loginPageInfo>.

Also, we found that login.yahoo.com doesn't work - login works, but then we're on the homepage. mail.yahoo.com gets us to webmail. Makes sense. I fixed the URL.

Commited as SVN Revision 130691
FIXED

Thanks, all!

	Ben Bucksch (:BenB) Assignee
	Comment 24 • 10 years ago

Info is live https://autoconfig.thunderbird.net/autoconfig/v1.1/yahoo.com

	dol
	Comment 25 • 10 years ago

(In reply to Ben Bucksch (:BenB) from comment #22)
> <loginPageInfo> it is, then.

@BenB Can you reflect this change also in the corresponding wiki page ( https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat#XML ). Thank you.

	Ben Bucksch (:BenB) Assignee
	Comment 26 • 10 years ago

(In reply to dol from comment #25)
> (In reply to Ben Bucksch (:BenB) from comment #22)
> > <loginPageInfo> it is, then.
> 
> @BenB Can you reflect this change also in the corresponding wiki page (
> https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat#XML
> ). Thank you.

Done