Extract/decode Javascript variables from html into python

Question

I'm trying to extract, with python, some javascript variables from an HTML site:

<script>
var nData = new Array();
var Data = "5b7b......";
nData = CallInit(Data);
...
...
</script>

I can see the content of "nData" in firebug (DOM Panel) without problem:

[Object { height="532",  width="1280",  url="https://example.org...8EDA4F3F5F395B9&key=lh1",  more...}, Object { height="266",  width="640",  url="https://example.org...8EDA4F3F5F395B9&key=lh1",  more...}]

The content of nData is an URL. How can i parse/extract the content of nData to python? It's possible?

Thanks

Answer 1

With the help of the python library Ghost.py it should be possible to get a dynamic variable out of executed Javascript code.

I just tried it out with some small test site and got a Javascript variable named a which I use on that page as a python object. I did the following:

1. Install Ghost.py with pip install Ghost.py.

2. Install PySide (it's a prerequisite for Ghost.py) with pip install PySide.

3. Use the following python code:

   from ghost import Ghost
   ghost = Ghost()
   ghost.open('https://dl.dropboxusercontent.com/u/13991899/test/index.html')
   js_variable, _ = ghost.evaluate('a', expect_loading=True)
   print js_variable
   

You should be able to get your variable nData into the python variable js_variable by opening your site with ghost.open and then call ghost.evaluate('nData').