Converting Python2 to Python3 with metaclasses resulted in a wrong flow

Question

I have a very large Python 2.7.6 project which I need to convert to Python 3.4. I used 2to3 script but 'metaclass' processing seems to be broken.

I filtered the code to shorten and pinpoint the problem. The following fragment works well with Python 2.7.6:

class Base(object):
    class __metaclass__(type):
        def __new__(cls, classname, bases, dict):
            new = type.__new__(cls, classname, bases, dict)
            new.classname = classname
            print ("Base::__metaclass__::new. Called.")
            return new                 

class Heir(Base):
    class __metaclass__(Base.__metaclass__):
        def __new__(self, *args):
            new = Base.__metaclass__.__new__(self, *args)
            print ("Heir::__metaclass__::new. Called.")
            return new

    @classmethod
    def define(cls, nexttype):
        print ("Heir::define. Called.")

class HeirOfHeir(Heir):
    pass

Heir.define(HeirOfHeir)

The code prints as expected:

Base::__metaclass__::new. Called.
Base::__metaclass__::new. Called.
Heir::__metaclass__::new. Called.
Base::__metaclass__::new. Called.
Heir::__metaclass__::new. Called.
Heir::define. Called.

But when running code with Python 3.4 I have only the last print:

Heir::define. Called.

Either 2to3 miscalculated or there is some manual work required. I have little experience with metaclasses unfortunately.

Answer 1

Your original code uses the fact that it is the name __metaclass__ in the class body is used as the meta class, but the 2to3 fixer only looks for straight assignments:

__metaclass__ = MetaClassName

rather than a class __metaclass__ statement or other manner of defining the name (from somemodule import MetaClassName as __metaclass__ would work in a Python 2 class body and 2to3 would miss that too).

You can fix this by moving the meta classes to separate class definitions:

class BaseMeta(type):
    def __new__(cls, classname, bases, dict):
        new = type.__new__(cls, classname, bases, dict)
        new.classname = classname
        print ("BaseMeta::new. Called.")
        return new                 

class Base(object):
    __metaclass__ = BaseMeta

class HeirMeta(BaseMeta):
    def __new__(self, *args):
        new = BaseMeta.__new__(self, *args)
        print ("HeirMeta::new. Called.")
        return new

class Heir(Base):    
    __metaclass__ = HeirMeta

    @classmethod
    def define(cls, nexttype):
        print ("Heir::define. Called.")

class HeirOfHeir(Heir):
    pass

Heir.define(HeirOfHeir)

You'll have to do this to define metaclasses in Python 3 anyway, as the mechanism to define metaclasses was changed to determining the metaclass before the class body is run rather than during (so that a metaclass can influence that step too).

Now 2to3 will correctly detect that there is a __metaclass__ attribute on your classes and rewrite those to use the new Python 3 syntax:

stackoverflow-2.7 $ bin/python -m lib2to3 fixed.py 
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored fixed.py
--- fixed.py    (original)
+++ fixed.py    (refactored)
@@ -5,8 +5,8 @@
         print ("BaseMeta::new. Called.")
         return new                 

-class Base(object):
-    __metaclass__ = BaseMeta
+class Base(object, metaclass=BaseMeta):
+    pass

 class HeirMeta(BaseMeta):
     def __new__(self, *args):
@@ -14,9 +14,7 @@
         print ("HeirMeta::new. Called.")
         return new

-class Heir(Base):    
-    __metaclass__ = HeirMeta
-
+class Heir(Base, metaclass=HeirMeta):    
     @classmethod
     def define(cls, nexttype):
         print ("Heir::define. Called.")
RefactoringTool: Files that need to be modified:
RefactoringTool: fixed.py

and the refactored code works as expected:

stackoverflow-2.7 $ bin/python -m lib2to3 -o ../stackoverflow-3.4 -nw --no-diffs fixed.py 
lib2to3.main: Output in '../stackoverflow-3.4' will mirror the input directory '' layout.
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored fixed.py
RefactoringTool: Writing converted fixed.py to ../stackoverflow-3.4/fixed.py.
RefactoringTool: Files that were modified:
RefactoringTool: fixed.py
stackoverflow-2.7 $ cd ../stackoverflow-3.4
stackoverflow-3.4 $ bin/python -V
Python 3.4.2
stackoverflow-3.4 $ bin/python fixed.py 
BaseMeta::new. Called.
BaseMeta::new. Called.
HeirMeta::new. Called.
BaseMeta::new. Called.
HeirMeta::new. Called.
Heir::define. Called.