Python 3 Status in OpenStack Liberty

news

Author: Victor Stinner
Source: Planet OpenStack

The Python 3 support in OpenStack Liberty made huge visible progress. Blocking libraries have been ported. Six OpenStack applications are now compatible with Python3: Aodh, Ceilometer, Gnocchi, Ironic, Rally and Sahara. Thanks to voting python34 check jobs, the Python 3 support can only increase, Python 2 only code cannot be reintroduced by mistake in tested code.

Progress made in OpenStack Juno and Kilo

The OpenStack port to Python 3 started slowly in 2013 during the development of OpenStack Havana. For the previous Python 3 status, read again Status of the OpenStack port to Python 3 (Cyril Roelandt, February 2014), see also: Why should OpenStack move to Python 3 right now? (Victor Stinner, December 2013).

The year 2013 was focused on porting third-party libraries and Oslo Incubator. The year 2014 was more focused on porting OpenStack clients and OpenStack common libraries. During the year 2015, we reached the interesting point: start porting OpenStack libraries and OpenStack applications. The Moving Apps to Python 3 etherpad prepared the work for the Liberty cycle.

While Python 3.3 was targeted in 2014, the new Python target version is 3.4. In parallel, Python 2.6 was dropped in more and more OpenStack client and libraries which also simplifies the port to Python 3.

setuptools, pbr and environment markers

It was not possible to have dependencies specific to Python 2 and others specific to Python 3. While dependencies like eventlet or MySQL-Python were incompatible with Python 3, it was annoying to run tests, even manually. Some dependencies were also installed on Python 2.7 whereas they were only neeeded on Python 2.6.

A first step (workaround) was to support requirements-py2.txt and requirements-py3.txt for dependencies specific to Python 2 / Python 3, and test-requirements-py2.txt and test-requirements-py3.txt for test requirements.

A lot of work was done in pip (ex: support environment markers in requirements) and pbr to support environment markers. Environment markers come from the PEP 426 (Metadata for Python Software Packages 2.0): see the Environment markers section. Examples of environment markers: sys_platform == 'win32' (detect Windows) and python_version=='2.7' (Python 2.7).

requirements-py2.txt and requirements-py3.txt were merged again into one unique requirements.txt file, and same for test requirements. The advantage of having environment markers is that it becomes possible to publish a universal wheel package which has different dependencies depending on the Python version.

OpenStack Common Libraries

Before Liberty, the port of almost all OpenStack applications was blocked by dependencies, OpenStack libraries and third-party libraries incompatible with Python 3. First, the root project oslo-incubator project was slowly ported to Python 3. In the meanwhile, this project was splitted into real libraries like oslo.config and oslo.utils. When a new library was created, we tried to ensure that the first release directly supports Python 3.

eventlet

The eventlet library is used by all OpenStack applications, but it was not compatible with Python 3. Hopefully most OpenStack clients don’t use eventlet and so could be ported to Python 3.

The eventlet project was slowly ported since 2014.  In 2015, it’s now fully compatible with Python 3, since the version 0.17.3.

MySQL

The mysql-python library also blocked almost all OpenStack applications.

We tried porting it to Python 3 at first, but the maintainers were not really reactive, so we thought it might be easier to just replace it with another library.

There was a discussion to replace mysql-python with mysqlclient, a fork adding Python 3 support, fixing various bugs and adding some features.

It was decided to replace mysql-python with PyMySQL instead. PyMySQL is a completely new MySQL driver written fully in Python. The main advantage is that it can be monkey-patched by eventlet to become asynchronous.

OpenStack libraries and other libraries

We ported or helped to port the following third-party libraries to Python 3 because they blocked OpenStack applications. We also pushed for releases. We also ported “OpenStack libraries”: libraries written for a specific application, like glance_store for Glance. Releases including Python 3 fixes:

  • ecdsa 0.10
  • glance_store 0.7.0: library written for Glance
  • netifaces 0.10.4
  • nose-exclude 0.4: blocked Horizon unit tests
  • os-brick 0.3.2: library written for Cinder
  • PyEClib 1.0.9
  • python-memcached 1.56: blocked keystone middleware (so all applications)
  • routes 2.2: blocked many applications (cinder, glance, keystone, neutron, etc.)
  • websockify 0.7.0

suds was replaced with suds-jurko: suds blocked cinder, oslo.vmware and nova. suds is no more maintained, whereas suds-jurko is maintained and supports Python 3.

Python 3 Status in OpenStack Liberty

Python 3 Status of OpenStack Common Libraries

All OpenStack common libraries are now compatible with Python 3: the entire test suite can run with Python 3.4. There are currently 20 libraries:

  • cliff
  • oslo.concurrency
  • oslo-incubator
  • oslo.config
  • oslo.context
  • oslo.db
  • oslo.i18n
  • oslo.log
  • oslo.messaging (*)
  • oslo.middleware
  • oslo.rootwrap
  • oslo.serialization
  • oslosphinx
  • oslotest
  • oslo.versionedobjects
  • oslo.vmware
  • oslo.utils
  • pylockfile
  • stevedore
  • taskflow

(*) The Qpid and AMQP transports of oslo.messaging are not yet compatible with Python 3. The legacy Qpid driver is deprecated, buggy and unmaintained. The AMQP driver is being ported to Python 3, it’s almost done.

Python 3 Status of OpenStack Clients

17 clients are fully compatible with Python 3, the entire test suite can run with Python 3.4:

  • keystonemiddleware
  • python-barbicanclient
  • python-ceilometerclient
  • python-cinderclient
  • python-glanceclient
  • python-heatclient
  • python-ironicclient
  • python-keystoneclient
  • python-manilaclient
  • python-marconiclient
  • python-neutronclient
  • python-novaclient
  • python-openstackclient
  • python-saharaclient
  • python-swiftclient
  • python-troveclient
  • python-tuskarclient

2 clients are being ported to Python 3:

  • python-designateclient
  • python-fuelclient (StackForge project)

Python 3 Status of OpenStack Applications

6 applications are already fully compatible with Python 3! All unit tests pass on Python 3.4. Congratulations to their maintainers.

  • Aodh
  • Ceilometer
  • Gnocchi
  • Ironic
  • Rally
  • Sahara

6 applications are being ported to Python 3 with a voting python34 check job to avoid Python 3 regressions. To be able to port the code incrementally, a subset of tests are run on Python 3 to have a working check job. More and more tests are added to the subset each time that more code is ported to Python3. Applications partially ported:

  • Neutron: 99.9% ! (7541 tests/7551, only 10 tests remain)
  • Heat: 86% (4386 tests/5119)
  • Cinder: 38% (2532 tests/6659)
  • Nova: 19% (2746 tests/14358)
  • Glance: 18% (512 tests/2779)
  • Horizon: 15% (238 tests/1605)
  • Keystone: 12% (524 tests/4318)

Keystone is still blocked by python-ldap and ldaptool. python-ldap may be replaced with pyldap, a fork compatible with Python 3.

3 applications are still at an early stage of the Python 3 port:

  • Designate
  • Manila
  • Swift

No more Python 3 regressions

In previous cycles, it was common to reintroduce code incompatible with Python 3 in files which were already ported to Python 3. We tried to avoid this issue by running unit tests on Python 3 on voting gates. When a test is run on Python 3, Python 2 only code cannot by introduced anymore in the tested code.

Some projects like Nova load more tests than the subset of tests executed on Python 3, to prevent people from adding Python 3 incompatible syntax and imports.

The Python 3 support is now also required to add a new dependency to OpenStack global requirements.

In general, more and more developers are aware of Python 3 and take it into account in their development.

What’s Next? How can you help?

Since most unit tests of Neutron and Heat are already ported to Python 3, we can expect a full support during the next Mitaka cycle. The port of other applications will also continue in parallel.

The next major task is to run functional tests on Python 3. Doug Hellmann wrote a specification Enabling Python 3 for Application Integration Tests. The specification was approved by the PTLs of the different projects. Now it’s time to implement the specification. The first step will be to add an option to DevStack to run clients and some applications on Python 3 (while other applications will still run on Python 2).

You can help to port OpenStack to Python 3 by reviewing patches, writing new patches and testing clients and applications on Python 3. The Python 3 wiki page is the central place to collaborate on this project.

Thanks

We would like to thank:

  • Andrey Kurilin (Rally)
  • Cyril Roelandt (Neutron)
  • Davanum Srinivas aka dims (Nova)
  • David Stanek (Keystone)
  • George Peristerakis (Horizon)
  • Ihar Hrachyshka (Neutron)
  • Jaivish Kothari aka janonymous (Swift)
  • Jeremy Stanley who coordinated efforts on switching from MySQL-python to PyMySQL
  • Pradeep Kumar Singh (Designate)
  • Sirushti Murugesan (Heat)
  • Victor Sergeyev (Ironic)
  • Victor Stinner aka haypo (Aodh, Ceilometer, Cinder, Glance, Horizon, Nova, Swift)

There are much more developers involved in the Python 3 porting effort. Thanks to all of them!

This article was written by Cyril Roelandt and Victor Stinner.

Powered by WPeMatico