I'm trying to backup a code of mine from sololearn site. I could copy/paste it of course, but because i would like to repeat it for other codes, and also for learning purpose, i'd like to do it with a python code, and only using the standard library if possible.
I present here the more basic try. I also have been struggling with HTMLParser, html.entities, xml.etree, i've tried to decode the response as "utf-8", to pass it through html.unescape(). The result is always dirty. this kind of dirty: \u003c!DOCTYPE html\u003e\r\n\u003chtml\u003e\r\n\u003c!--\r\ sometimes less, but never clean
from urllib.request import urlopen
import re
url = "https://code.sololearn.com/************/#"
with urlopen(url) as response:
page = str(response.read())
code = re.search(r'window.code = "(.*)";.*window.cssCode',page).group(1)
print(code)
The goal is to backup my files, writing them into files in a clean functional form, the codes can be html+css+js, python, c, etc... I also tried to work on the dirty results with regex modifications, but i think it's impossible, because the codes may contain on purpose elements like "\r\n" that should not be modified.
Seems that you got JSON encoded string. You can use ast.literal_eval() (doc) to decode the string:
from ast import literal_eval
from urllib.request import urlopen
import re
url = "https://code.sololearn.com/************/#"
with urlopen(url) as response:
page = response.read().decode('utf-8')
code = re.search(r'window.code = "(.*)";.*window.cssCode',page, flags=re.DOTALL).group(1)
print(literal_eval('"' + code + '"'))
Prints:
<!DOCTYPE html>
<html>
<!--
If you're interested in the tools used here:
to display a partition:
http://www.vexflow.com/
to make it sound:
https://tonejs.github.io/
-->
<head>
<link href="https://fonts.googleapis.com/css?family=Annie+Use+Your+Telescope&display=swap" rel="stylesheet">
<script src="https://cdnjs.cloudflare.com/ajax/libs/tone/13.8.12/Tone.js"></script>
<script src="https://unpkg.com/vexflow/releases/vexflow-min.js"></script>
<title>Melody Generator</title>
</head>
<body>
<div id="wrapper">
<div id="popup">
<div id="description">description gonna be here</div>
<div id="choice"></div>
</div>
<div id="input" class="blur">
<div id="melody">
<h1>Melody</h1>
<textarea id="melo_num" class="text_input" placeholder="Enter two words..."></textarea>
<p id="melo_rebased"></p>
</div>
<div id="rhythm">
<h1>Rhythm</h1>
<textarea id="rhyt_num" class="text_input" placeholder="...hear some magic !"></textarea>
<p id="rhyt_rebased"></p>
</div>
</div>
<div id="partition" class="blur"></div>
<div id="controls" class="blur">
<div id="back" class="control">back</div>
<div id="play" class="control">play</div>
<div id="stop" class="control">stop</div>
</div>
<div id="current" class="blur"></div>
<p></p>
<div id="settings" class="blur">
<div id="loop" class="blur">loop
<div class="twinkle lamp" id="loop_lamp"></div>
</div>
<div id="root" class="blur">root
<div class="lamp" id="root_lamp"></div>
</div>
<div id="mode" class="blur">mode
<div class="lamp" id="mode_lamp"></div>
</div>
<div id="range" class="blur">range
<div class="lamp" id="range_lamp"></div>
</div>
<div id="rhythm" class="blur">rhythm
<div class="lamp" id="rhythm_lamp"></div>
</div>
<div id="convert" class="blur">convert
<div class="lamp" id="convert_lamp"></div>
</div>
<div id="volume" class="blur slider_box">
volume
<input id="sound_vol" class="slider" type="range" min="-50" max="0" value="-10">
</div>
<div id="speed" class="blur slider_box">
speed
<input id="speed_lvl" class="slider" type="range" min="0" max="200" value="100">
</div>
<div id="sustain" class="blur slider_box">
sustain
<input id="sustain_lvl" class="slider" type="range" min="0" max="200" value="100">
</div>
<div id="demo" class="blur">demo
<div class="lamp" id="demo_lamp"></div>
</div>
</div>
<p></p>
</div>
</body>
</html>
Or use json.loads() (doc):
import json
print(json.loads('"' + code + '"'))