Having difficulty extracting some data I need from a web page.
This is the type of thing I'm attempting
from bs4 import BeautifulSoup
import urllib
url = 'http://www.myfxbook.com/community/outlook'
htmlfile = urllib.urlopen(url)
soup = BeautifulSoup(htmlfile)
for tag in soup.findAll('td'):
print tag
(this gives thousands of lines of code as an output)
This is where the data that I need is held
I don't know how to make it specific enough
I need the '5169 Lots' inside the 'td' tags, and the same thing for the '2879 Lots' down a bit further.
So what I'm essentially after is the two numbers that show how many lots are Long and Short on EURUSD.
Any help would be greatly appreciated.
I am using Python 2.7
from sys import exit
import lxml.html
import requests
url = 'http://www.myfxbook.com/community/outlook'
doc = requests.get(url)
root = lxml.html.fromstring(doc.text)
all_id_inputs = root.xpath("//input[starts-with(@id,'outlookTip')]/@id")
for ids in all_id_inputs:
xpath_value = '//*[@id="{}"]/@value'.format(ids)
table_value = root.xpath(xpath_value)
root_table = lxml.html.fromstring(table_value[0])
# print table_value[0]
# print
firt_value = root_table.xpath("//table/tr[2]/td[3]/text()")
second_value = root_table.xpath("//table/tr[3]/td[3]/text()")
third_value = root_table.xpath("//div/text()")[1]
print firt_value
print second_value
print third_value
print
Output:
['4227.84 Lots']
['7968.21 Lots']
41% of traders are currently trading EURUSD.
['1330.00 Lots']
['3910.07 Lots']
17% of traders are currently trading GBPUSD.
['772.90 Lots']
['1339.60 Lots']
12% of traders are currently trading USDJPY.
['204.63 Lots']
['402.83 Lots']
6% of traders are currently trading GBPJPY.
['633.87 Lots']
['883.68 Lots']
8% of traders are currently trading USDCAD.
['172.53 Lots']
['555.24 Lots']
5% of traders are currently trading EURAUD.
['328.70 Lots']
['981.80 Lots']
8% of traders are currently trading EURJPY.
['747.65 Lots']
['361.27 Lots']
4% of traders are currently trading AUDCAD.
['197.44 Lots']
['139.89 Lots']
3% of traders are currently trading AUDJPY.
['285.17 Lots']
['329.60 Lots']
2% of traders are currently trading AUDNZD.
['1186.19 Lots']
['566.72 Lots']
9% of traders are currently trading AUDUSD.
['112.03 Lots']
['156.68 Lots']
1% of traders are currently trading CADJPY.
['146.43 Lots']
['236.74 Lots']
3% of traders are currently trading EURCAD.
['205.32 Lots']
['190.43 Lots']
1% of traders are currently trading EURCHF.
['0.00 Lots']
['0.00 Lots']
0% of traders are currently trading EURCZK.
['826.11 Lots']
['313.21 Lots']
7% of traders are currently trading EURGBP.
['3.67 Lots']
['3.01 Lots']
0% of traders are currently trading EURNOK.
['36.12 Lots']
['150.40 Lots']
2% of traders are currently trading EURNZD.
['5.74 Lots']
['1.92 Lots']
0% of traders are currently trading EURPLN.
['3.13 Lots']
['0.76 Lots']
0% of traders are currently trading EURSEK.
['18.04 Lots']
['4.85 Lots']
0% of traders are currently trading EURTRY.
['39.89 Lots']
['434.25 Lots']
3% of traders are currently trading GBPCAD.
['38.58 Lots']
['124.30 Lots']
1% of traders are currently trading GBPCHF.
['207.33 Lots']
['146.33 Lots']
1% of traders are currently trading NZDCAD.
['86.32 Lots']
['61.68 Lots']
1% of traders are currently trading NZDJPY.
['580.56 Lots']
['266.71 Lots']
5% of traders are currently trading NZDUSD.
['802.76 Lots']
['243.32 Lots']
4% of traders are currently trading USDCHF.
['0.00 Lots']
['0.22 Lots']
0% of traders are currently trading USDCZK.
['0.20 Lots']
['5.23 Lots']
0% of traders are currently trading USDHKD.
['9.18 Lots']
['7.99 Lots']
0% of traders are currently trading USDMXN.
['1.75 Lots']
['1.92 Lots']
0% of traders are currently trading USDNOK.
['2.32 Lots']
['0.41 Lots']
0% of traders are currently trading USDPLN.
['1.66 Lots']
['3.91 Lots']
0% of traders are currently trading USDSEK.
['31.71 Lots']
['11.61 Lots']
0% of traders are currently trading USDSGD.
['0.00 Lots']
['0.70 Lots']
0% of traders are currently trading USDTHB.
['13.18 Lots']
['3.78 Lots']
0% of traders are currently trading USDTRY.
['21.88 Lots']
['24.87 Lots']
0% of traders are currently trading USDZAR.
['28.33 Lots']
['76.17 Lots']
0% of traders are currently trading CHFJPY.
['160.82 Lots']
['35.39 Lots']
1% of traders are currently trading AUDCHF.
['156.37 Lots']
['337.38 Lots']
2% of traders are currently trading GBPNZD.
['70.59 Lots']
['9.52 Lots']
0% of traders are currently trading NZDCHF.
['7.56 Lots']
['44.56 Lots']
0% of traders are currently trading XAGUSD.
['347.22 Lots']
['372.50 Lots']
3% of traders are currently trading XAUUSD.
['85.77 Lots']
['48.52 Lots']
0% of traders are currently trading CADCHF.
['96.59 Lots']
['1021.96 Lots']
5% of traders are currently trading GBPAUD.
['2.00 Lots']
['0.00 Lots']
0% of traders are currently trading SPA35.
['10.22 Lots']
['14.55 Lots']
0% of traders are currently trading SGDJPY.
['0.42 Lots']
['0.00 Lots']
0% of traders are currently trading GBPNOK.
['29.20 Lots']
['24.30 Lots']
0% of traders are currently trading US30.
['4.15 Lots']
['3.30 Lots']
0% of traders are currently trading EURZAR.
['0.37 Lots']
['2.32 Lots']
0% of traders are currently trading AUDSGD.
['0.39 Lots']
['0.00 Lots']
0% of traders are currently trading GBPSEK.
['0.02 Lots']
['0.01 Lots']
0% of traders are currently trading CHFSGD.
['0.65 Lots']
['32.99 Lots']
0% of traders are currently trading EURSGD.
['0.24 Lots']
['0.25 Lots']
0% of traders are currently trading GBPSGD.
['173.06 Lots']
['14.01 Lots']
0% of traders are currently trading US500.
['10.80 Lots']
['9.10 Lots']
0% of traders are currently trading UK100.
['0.40 Lots']
['10.62 Lots']
0% of traders are currently trading NOKJPY.
['2.63 Lots']
['8.80 Lots']
0% of traders are currently trading ZARJPY.
['0.16 Lots']
['0.00 Lots']
0% of traders are currently trading GBPTRY.
['16.20 Lots']
['1.98 Lots']
0% of traders are currently trading USDRUB.
['0.20 Lots']
['0.22 Lots']
0% of traders are currently trading SEKJPY.
['0.02 Lots']
['0.05 Lots']
0% of traders are currently trading XAUAUD.
['0.00 Lots']
['0.48 Lots']
0% of traders are currently trading XAUEUR.
['0.01 Lots']
['3.40 Lots']
0% of traders are currently trading NAS100.
['0.30 Lots']
['0.02 Lots']
0% of traders are currently trading US2000.
['36.00 Lots']
['88.00 Lots']
0% of traders are currently trading FRA40.
['132.73 Lots']
['79.14 Lots']
0% of traders are currently trading GER30.
['1.00 Lots']
['0.50 Lots']
0% of traders are currently trading AUS200.
['0.00 Lots']
['0.00 Lots']
0% of traders are currently trading JPN225.
['0.00 Lots']
['0.06 Lots']
0% of traders are currently trading NOKSEK.
['0.06 Lots']
['0.28 Lots']
0% of traders are currently trading XAGEUR.
['0.00 Lots']
['0.03 Lots']
0% of traders are currently trading XPTUSD.
['0.00 Lots']
['0.01 Lots']
0% of traders are currently trading XPDUSD.
['0.89 Lots']
['4.13 Lots']
0% of traders are currently trading USDCNH.
['0.00 Lots']
['0.00 Lots']
0% of traders are currently trading IT40.
['1.10 Lots']
['3.42 Lots']
0% of traders are currently trading XTIUSD.
['0.55 Lots']
['0.82 Lots']
0% of traders are currently trading XBRUSD.
['18.00 Lots']
['18.00 Lots']
0% of traders are currently trading HK50.
['0.61 Lots']
['0.00 Lots']
0% of traders are currently trading XNGUSD.