Search code examples
pythonpandasbinance

Pandas json_load misses decimals

I am struggling to display all the decimals coming from a json feed when I use pandas to convert the data. The code is the following.

import pandas as pd

url = 'https://api.binance.com/api/v1/klines?interval=1m&symbol=VETBTC'
df = pd.read_json(url, orient='columns', precise_float=True)

df.columns = ["Open_time","Open","High","Low","Close","Volume","Close_time","Quote_AV","TradesNo","Taker_base","Taker_quote","Ignore"]
df['Open_time'] = pd.to_datetime(df['Open_time'],unit='ms')
df['Close_time'] = pd.to_datetime(df['Close_time'],unit='ms')

print df.head(10000)

The output is:

             Open_time      Open      High       Low     Close   Volume              Close_time  Quote_AV  TradesNo  Taker_base  Taker_quote  Ignore
0   2018-11-21 02:53:00  0.000001  0.000001  0.000001  0.000001    64166 2018-11-21 02:53:59.999  0.077268         6       44229     0.053344       0
1   2018-11-21 02:54:00  0.000001  0.000001  0.000001  0.000001     5030 2018-11-21 02:54:59.999  0.005996         2        1010     0.001212       0
2   2018-11-21 02:55:00  0.000001  0.000001  0.000001  0.000001    61463 2018-11-21 02:55:59.999  0.073756         2       61463     0.073756       0
3   2018-11-21 02:56:00  0.000001  0.000001  0.000001  0.000001   106492 2018-11-21 02:56:59.999  0.127790         2      106492     0.127790       0
4   2018-11-21 02:57:00  0.000001  0.000001  0.000001  0.000001    13215 2018-11-21 02:57:59.999  0.015858         1       13215     0.015858       0
5   2018-11-21 02:58:00  0.000001  0.000001  0.000001  0.000001    25991 2018-11-21 02:58:59.999  0.031181         2       25142     0.030170       0
6   2018-11-21 02:59:00  0.000001  0.000001  0.000001  0.000001  2024424 2018-11-21 02:59:59.999  2.429309        14     1157504     1.389005       0
7   2018-11-21 03:00:00  0.000001  0.000001  0.000001  0.000001     6500 2018-11-21 03:00:59.999  0.007865         1        6500     0.007865       0
8   2018-11-21 03:01:00  0.000001  0.000001  0.000001  0.000001    24128 2018-11-21 03:01:59.999  0.028954         2           0     0.000000       0
9   2018-11-21 03:02:00  0.000001  0.000001  0.000001  0.000001  1126289 2018-11-21 03:02:59.999  1.351547         2           0     0.000000       0
10  2018-11-21 03:03:00  0.000001  0.000001  0.000001  0.000001    91099 2018-11-21 03:03:59.999  0.109695         6       37571     0.045461       0
11  2018-11-21 03:04:00  0.000001  0.000001  0.000001  0.000001    71152 2018-11-21 03:04:59.999  0.086094         1       71152     0.086094       0
12  2018-11-21 03:05:00  0.000001  0.000001  0.000001  0.000001    12222 2018-11-21 03:05:59.999  0.014789         2       12222     0.014789       0

While the json feed has values with more decimals just like:

0 1542768840000 1 "0.00000119" 2 "0.00000120" 3 "0.00000119" 4 "0.00000120" 5 "5030.00000000" 6 1542768899999 7 "0.00599580" 8 2 9 "1010.00000000" 10 "0.00121200" 11 "0"

I tried using the precise_float option but it doesn't seem to do what it is supposed to. Any help would be highly appreciated.

Solution 
  • import pandas as pd
import ssl
ssl._create_default_https_context = ssl._create_unverified_context

url = 'https://api.binance.com/api/v1/klines?interval=1m&symbol=VETBTC'
df = pd.read_json(url, orient='columns', precise_float=True)

df.columns = ["Open_time","Open","High","Low","Close","Volume","Close_time","Quote_AV","TradesNo","Taker_base","Taker_quote","Ignore"]
df['Open_time'] = pd.to_datetime(df['Open_time'],unit='ms')
df['Close_time'] = pd.to_datetime(df['Close_time'],unit='ms')

print(df.head())

    OutPut:

                Open_time      Open      High       Low     Close  Volume  \
0 2018-11-21 03:17:00  0.000001  0.000001  0.000001  0.000001  960188   
1 2018-11-21 03:18:00  0.000001  0.000001  0.000001  0.000001   89803   
2 2018-11-21 03:19:00  0.000001  0.000001  0.000001  0.000001       0   
3 2018-11-21 03:20:00  0.000001  0.000001  0.000001  0.000001       0   
4 2018-11-21 03:21:00  0.000001  0.000001  0.000001  0.000001  438661   

               Close_time  Quote_AV  TradesNo  Taker_base  Taker_quote  Ignore  
0 2018-11-21 03:17:59.999  1.152354         5       12795     0.015482       0  
1 2018-11-21 03:18:59.999  0.108186         6       42283     0.051162       0  
2 2018-11-21 03:19:59.999  0.000000         0           0     0.000000       0  
3 2018-11-21 03:20:59.999  0.000000         0           0     0.000000       0  
4 2018-11-21 03:21:59.999  0.526410         8        1714     0.002074       0  

​

    Setting precision:

    pd.set_option('precision', 15)
print(df.head())

    Output:

                Open_time        Open        High         Low       Close  Volume  \
0 2018-11-21 03:13:00  0.00000121  0.00000121  0.00000121  0.00000121    7231   
1 2018-11-21 03:14:00  0.00000121  0.00000121  0.00000121  0.00000121   22162   
2 2018-11-21 03:15:00  0.00000120  0.00000120  0.00000120  0.00000120    1000   
3 2018-11-21 03:16:00  0.00000121  0.00000121  0.00000120  0.00000120   83247   
4 2018-11-21 03:17:00  0.00000120  0.00000121  0.00000120  0.00000121  960188   

               Close_time    Quote_AV  TradesNo  Taker_base  Taker_quote  \
0 2018-11-21 03:13:59.999  0.00874951         1        7231   0.00874951   
1 2018-11-21 03:14:59.999  0.02681602         3       22162   0.02681602   
2 2018-11-21 03:15:59.999  0.00120000         1           0   0.00000000   
3 2018-11-21 03:16:59.999  0.10062838         7       73198   0.08856958   
4 2018-11-21 03:17:59.999  1.15235355         5       12795   0.01548195   

   Ignore  
0       0  
1       0  
2       0  
3       0  
4       0

    Reference: https://pandas.pydata.org/pandas-docs/stable/options.html#setting-startup-options-in-python-ipython-environment