Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Description • 7 years ago

Python 2 is going away. It behooves us to think about this now so that we can figure out our library situation, rewrite and rearchitect things, and generally have time to switch to Python 3 leisurely.

This tracker covers all the work required to switch to Python 3.

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 1 • 7 years ago

I'm thinking this is a good plan of attack:

1. make a list of requirements which don't support Python 3; write up a bug for each one

2. deal with all those bugs: remove code, rewrite code, update libraries, switch to different libraries, etc

3. update socorro code and Docker infrastructure to work with Python 3


Maybe this is a 2018q1 project? In the meantime, as we make changes, we should be asking ourselves, "Will this work with Python 3?"

	Lonnen :lonnen
	Comment 2 • 7 years ago

Step 1 is probably a good 2017q4 goal

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 3 • 6 years ago

We're all set with requirements now--all the requirements support Python 3.

We have a rough infrastructure for running tests in Python 3. This will let us work through updating code and tests incrementally.

We should fix linting with flake8 in Python 3 first. Once that's done, one way to break up the work in chunks is this:

1. socorro
   * socorro.lib
   * socorro.external
   * socorro.external.boto
   * socorro.external.postgres
   * socorro.external.es
2. crontabber
3. processor
4. webapp

Helpful resources:

Python porting guide:
   https://docs.python.org/3/howto/pyporting.html
   particularly this section: https://docs.python.org/3/howto/pyporting.html#update-your-code

Cheat sheet: Writing Python 2-3 compatible code
   http://python-future.org/compatible_idioms.html#essential-syntax-differences

Supporting Python 3: An in-depth guide
   http://python3porting.com/

Six documentation:
   https://pythonhosted.org/six/

	Will Kahn-Greene [:willkg] ET needinfo? me Assignee
	Comment 4 • 6 years ago

Just so I don't forget where I wrote it again, I wrote up some docs on iteration steps here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1460035#c2

	cmiller
	Comment 5 • 6 years ago

A big change in python 2 and python 3 is the base string data type. ascii(python 2) and unicode(python 3). Many errors will come from pivoting the desired behavior of these data types. Many of the test failed because of the discrepancies in data types for what these test were expecting. Majority of the task will be finding where the data changed to fix it accordingly and modifying or removing test. Code isn't commented very much so stay in constant contact with the lead dev asking about the stack traces from error output and each individual unit test if need be. Steps to solving I used were:

If you don't know python start here:
    https://github.com/gregmalcolm/python_koans (learn the language as much as possible)
    - Will help sparingly because the code is written in sophisticated way beyond any of these test suites but it still helps


Problem solving:

1) Check python 2/3 cheat sheet
      http://python-future.org/compatible_idioms.html#essential-syntax-differences

2) check six library for desired import functionality

3) Stack trace
     - Check the code in working env. Trace functions looking for value modifications as far as possible
     - If web services are involved look up what you can about whats going on with the code as it pertains to the web service
     - Consider using conditional statements for doing something if its a Python 2 env or python 3 env.
     - talk to lead dev on what you can assume about the test in possibly changing the test in the testing and expected values.

There is more commentary on the specific problems I faced on the work I did. They haven't all been the same but the methods I used in solving them were similar. 

https://github.com/pulls?q=is%3Apr+author%3Aceddy-cedd+archived%3Afalse+is%3Aclosed

	cmiller
	Comment 6 • 6 years ago

(In reply to cmiller from comment #5)
> A big change in python 2 and python 3 is the base string data type.
> ascii(python 2) and unicode(python 3). Many errors will come from pivoting
> the desired behavior of these data types. Many of the test failed because of
> the discrepancies in data types for what these test were expecting. Majority
> of the task will be finding where the data changed to fix it accordingly and
> modifying or removing test. Code isn't commented very much so stay in
> constant contact with the lead dev asking about the stack traces from error
> output and each individual unit test if need be. Steps to solving I used
> were:
> 
> If you don't know python start here:
>     https://github.com/gregmalcolm/python_koans (learn the language as much
> as possible)
>     - Will help sparingly because the code is written in sophisticated way
> beyond any of these test suites but it still helps
> 
> 
> Problem solving:
> 
> 1) Check python 2/3 cheat sheet
>      
> http://python-future.org/compatible_idioms.html#essential-syntax-differences
> 
> 2) check six library for desired import functionality
> 
> 3) Stack trace
>      - Check the code in working env. Trace functions looking for value
> modifications as far as possible
>      - If web services are involved look up what you can about whats going
> on with the code as it pertains to the web service
>      - Consider using conditional statements for doing something if its a
> Python 2 env or python 3 env.
>      - talk to lead dev on what you can assume about the test in possibly
> changing the test in the testing and expected values.
> 
> There is more commentary on the specific problems I faced on the work I did.
> They haven't all been the same but the methods I used in solving them were
> similar. 
> 
https://github.com/mozilla-services/socorro/pull/4487
https://github.com/mozilla-services/socorro/pull/4508

	[github robot]
	Comment 8 • 6 years ago

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/3308852985f0d8ec3234bfa5eab0b9fa7cd45252
Merge pull request #4600 from willkg/cleanup-makefile

bug 1406703: Cleanup makefile

	[github robot]
	Comment 10 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/e27372eac6fccc8e13c8be8cd2a550194f216d16
bug 1406703: add webapp tests to python3 test scaffolding

https://github.com/mozilla-services/socorro/commit/bece727d51d4b70e63a00fa4c3ce540f71fcd282
Merge pull request #4651 from willkg/1406703-python3-webapp

bug 1406703: add webapp tests to python3 test scaffolding

	[github robot]
	Comment 12 • 6 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/b249ef6684346f3e2f4f523bc0762e26867b002e
bug 1406703: add comments for what's remaining

This adds commented out lines for what remains to get working in Python 3.
This will make it easier to make sure we've got everything.

https://github.com/mozilla-services/socorro/commit/3f6d58554bd442ac58b661a6210ffffed54f83ad
Merge pull request #4697 from willkg/1406703-python3

bug 1406703: add comments for what's remaining

	[github robot]
	Comment 14 • 5 years ago

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/6d0fa71f14e4bcc8ad8fb66ca5cc72ccb5971505
bug 1406703: redo image with Python 3 and remove Python 2-isms

https://github.com/mozilla-services/socorro/commit/76faea5b078a69feb936d516479567d3ae4fa7e4
bug 1406703: fix socorro_aws_s3.sh script

This was using print statement which doesn't exist in Python 3.

https://github.com/mozilla-services/socorro/commit/45defda60cc2eb4ee7b09de3039ad3c284fdfa2a
bug 1406703: fix fetch_crash_data.py

This was trying to serialize a dictkeys--this fixes that.

https://github.com/mozilla-services/socorro/commit/340c9137ffdba2996bb76813c0dff1b4bc314dc5
bug 1406703: fix rabbitmq crashstorage

The body from a channel.basic_get is a bytes. Since that always consists
of a crash id and that's always a string, this adjusts the code to decode
it first.

https://github.com/mozilla-services/socorro/commit/d2670a82f5b82a9993ac443bd81f323969e20372
fix bug 1406703: update Python version in metadata

https://github.com/mozilla-services/socorro/commit/90214cd165fdb972d15f84934f2ce0efc2bf4f24
fix bug 1406703: Remove commented out code in .circleci/config.yml

https://github.com/mozilla-services/socorro/commit/eec0c4ed742e3e9802d46abb49eb9171f9460ed4
fix bug 1406703: switch to python:3.6.7-slim-jessie

https://github.com/mozilla-services/socorro/commit/6ec93f01df1ef387c3f60bcd70537a0a1d535385
Merge pull request #4740 from willkg/1406703-python3

fix bug 1406703: migrate to python 3