Parsing Google Search Response Json Python requests. Regex Multiline

Question

I want to transform the original response into a valid JSON, which I am able to do, but in a bit sloppy way.

This is the original response:

// API callback
google.search.Search.apiary2387({
 "cursor": {
  "currentPageIndex": 0,
  "estimatedResultCount": "4490",
  "moreResultsUrl": "http://www.google.com/cse?oe=utf8&ie=utf8&source=uds&q=ssh&start=0&sort=&cx=013305635491195529773:0ufpuq-fpt0",
  "resultCount": "4,490",
  "searchResultTime": "0.22",
  "pages": [
   {
    "label": 1,
    "start": "0"
   },
   {
    "label": 2,
    "start": "1"
   },
   {
    "label": 3,
    "start": "2"
   },
   {
    "label": 4,
    "start": "3"
   },
   {
    "label": 5,
    "start": "4"
   },
   {
    "label": 6,
    "start": "5"
   },
   {
    "label": 7,
    "start": "6"
   },
   {
    "label": 8,
    "start": "7"
   },
   {
    "label": 9,
    "start": "8"
   },
   {
    "label": 10,
    "start": "9"
   }
  ]
 },
 "context": {
  "title": "Pastebin Active",
  "total_results": "0",
  "facets": []
 },
 "results": [
  {
   "GsearchResultClass": "GwebSearch",
   "cacheUrl": "http://www.google.com/search?q=cache:PBL2A25kpZoJ:pastebin.com",
   "clicktrackUrl": "https://www.google.com/url?q=http://pastebin.com/u/ssh&sa=U&ved=0ahUKEwiO4fjNpovMAhWBPxoKHYJXAS4QFggEMAA&client=internal-uds-cse&usg=AFQjCNHczEhDXdcUnRZhpArEeSiHfjwMJA",
   "content": "BitBucket - Backup your code in the cloud! Host unlimited private projects, for free\n. SIGN UP takes 10 seconds, and it's free! Guest ...",
   "contentNoFormatting": "BitBucket - Backup your code in the cloud! Host unlimited private projects, for free\n. SIGN UP takes 10 seconds, and it's free! Guest ...",
   "formattedUrl": "pastebin.com/u/\u003cb\u003essh\u003c/b\u003e",
   "title": "\u003cb\u003eSsh's\u003c/b\u003e Pastebin - Pastebin.com",
   "titleNoFormatting": "Ssh's Pastebin - Pastebin.com",
   "unescapedUrl": "http://pastebin.com/u/ssh",
   "url": "http://pastebin.com/u/ssh",
   "visibleUrl": "pastebin.com",
   "richSnippet": {
    "cseImage": {
     "src": "http://pastebin.com/i/facebook.png"
    },
    "metatags": {
     "fbAppId": "231493360234820",
     "ogTitle": "Ssh's Pastebin - Pastebin.com",
     "ogType": "article",
     "ogUrl": "http://pastebin.com/u/ssh",
     "ogImage": "http://pastebin.com/i/facebook.png",
     "ogSiteName": "Pastebin",
     "viewport": "width=device-width, maximum-scale=1.0, user-scalable=no"
    }
   }
  }
 ]
}
);

And in order to extract the valid JSON I have to remove the JavaScript call, so I remove everything until the first ( and I remove at the end, the ).

This is how I think it would work:

import requests
import re
import json

url = 'https://www.googleapis.com/customsearch/v1element?key=AIzaSyCVAXiUzRYsML1Pv6RwSG1gunmMikTzQqY&rsz=filtered_cse&num=1&hl=en&prettyPrint=true&source=gcsc&gss=.com&sig=432dd570d1a386253361f581254f9ca1&start=0&cx=013305635491195529773:0ufpuq-fpt0&q=ssh&sort=&googlehost=www.google.com&callback=google.search.Search.apiary2387'

resp = requests.get(url)

content = resp.content

formatted = re.sub(r'(.*\(|\);$)','', content , re.I|re.M|re.DOTALL)

formatted_json = json.loads(formatted)
for i, result in enumerate(formatted_json['results']):
    print formatted_json['results'][i]['url']

And this is what I had to add in order to make it work:

formatted = re.sub(r'// API callback', '', content)

I don't know why, since I am removing everything until I find the (, why doesn't it apply to all lines if I use the flag re.M ¿?

You can see (r'(.*\(|\);$)','', content , re.DOTALL) should work:

https://regex101.com/r/uN2wV4/3

(the option /s means . is also \n like DOTALL)

Answer 1

Easiest way: just removing this parameter from the request:

callback=google.search.Search.apiary2387

And the response is a valid Json.