How to parse/extract data from a mediawiki marked-up article via python

Question

Source Mediawiki markup

Right now I'm using a variety of regexes to "parse" the data in the mediawiki mark-up into lists/dictionaries, so that elements within the article can be used.

This is hardly the best method, as the number of cases that have to be made are large.

How would one parse an article's mediawiki markup into a variety of python objects so that the data within can be used?

Example being:

• Extract all headlines to a dictionary, hashing it with its section.
• Grab all interwiki links, and stick them into a list (I know
  this can be done from the API but I'd rather only have one API call to
  reduce bandwidth use).
• Extract all image names and hash them with their sections

A variety of regexes can achieve the above, but I'm finding the number I have to make rather large.

Here's the mediawiki unofficial specification (I don't find their official specification as useful).

Answer 1

mwlib - MediaWiki parser and utility library

pediapress/mwlib:

mwlib provides a library for parsing MediaWiki articles and converting them to different output formats. mwlib is used by wikipedia's "Print/export" feature in order to generate PDF documents from wikipedia articles.

Here's the documentation page. The older doc page used have a one-liner example:

from mwlib.uparser import simpleparse
simpleparse("=h1=\n*item 1\n*item2\n==h2==\nsome [[Link|caption]] there\n")

If you want to see how it's used in action, see the test cases that come with the code. (mwlib/tests/test_parser.py from git repository):

from mwlib import parser, expander, uparser
from mwlib.expander import DictDB
from mwlib.xfail import xfail
from mwlib.dummydb import DummyDB
from mwlib.refine import util, core

parse = uparser.simpleparse

def test_headings():
    r=parse(u"""
= 1 =
== 2 ==
= 3 =
""")

    sections = [x.children[0].asText().strip() for x in r.children if isinstance(x, parser.Section)]
    assert sections == [u"1", u"3"]

Also see Markup spec and Alternative parsers for more information.