Search code examples
web-servicesforex

How to get Intraday backfilled FOREX data programmatically?

I've been looking for real time intraday quotes data for major FOREX pairs, backfilled to several days. I want to use it programmatically for Android or web application, therefore a CSV or XML formated webservice would be ideal.

So far I found a few websites by googling such as finam.ru. But it is in Russian and I was not able to find any clearly documented source on exact URL format how to obtain it.

Could someone guide me how to do it?

Solution

  • On the url: http://www.finam.ru/analysis/profile041CA00007/ you can choose the data to export in csv.

    You should choose "Мировые валюты" (International Currencies) on the left combobox and you can choose the currency on the right.

    Then you click on the button and download a csv file. Here is an example with minute data:

    http://195.128.78.52/AUDCAD_140501_140526.txt?market=5&em=181410&code=AUDCAD&df=1&mf=4&yf=2014&dt=26&mt=4&yt=2014&p=2&f=AUDCAD_140501_140526&e=.txt&cn=AUDCAD&dtf=1&tmf=1&MSOR=0&mstime=on&mstimever=1&sep=1&sep2=1&datf=2&at=1

    I have never seen a documentation for it. Just experimented with the date and got the needed period.

    You can add the header:Referer:http://www.finam.ru/analysis/profile2C4A000007/ to get the tick data. The code in C# that allows to bypass the restrictions is at the end of the answer.

    In the form the periods are the following:

    <select id="issuer-profile-export-period" name="p" style="width: 135px; display: none;">
<option value="1">тики</option> ticks
<option value="2">1 мин.</option> 1 minute
<option value="3">5 мин.</option> 5 minutes
<option value="4">10 мин.</option> 10 minutes
<option value="5">15 мин.</option> 15 minutes
<option value="6">30 мин.</option> 30 minutes
<option value="7" selected="selected">1 час</option> 1 hour
<option value="8">1 день</option> 1 day
<option value="9">1 неделя</option> 1 week
<option value="10">1 месяц</option> 1 month
</select>

    mstimer = 1 - moscow time mstimer = 0 - not moscow time f - mane of file

    It is most likely that to change the currency you need to change it in three places: em=(these are the codes from the code below), code=, cn=

    Currencies:

    <ul>
<li><a href="#" index="1" value="181410" class="">Aud/Cad</a></li>
<li><a href="#" index="2" value="181411" class="">Aud/Chf</a></li>
<li><a href="#" index="3" value="181418" class="">Aud/Dkk</a></li>
<li><a href="#" index="4" value="181408" class="">Aud/Jpy</a></li>
<li><a href="#" index="5" value="181417" class="hover">Aud/Nok</a></li>
<li><a href="#" index="6" value="181409" class="">Aud/Nzd</a></li>
<li><a href="#" index="7" value="181419">Aud/Sek</a></li>
<li><a href="#" index="8" value="181416">Aud/Sgd</a></li>
<li><a href="#" index="9" value="66699">Aud/Usd</a></li>
<li><a href="#" index="10" value="181389">Cad/Chf</a></li>
<li><a href="#" index="11" value="181390">Cad/Jpy</a></li>
<li><a href="#" index="12" value="181455">Cad/Usd</a></li>
<li><a href="#" index="13" value="181403">Chf/Dkk</a></li>
<li><a href="#" index="14" value="21084">Chf/Jpy</a></li>
<li><a href="#" index="15" value="181396">Chf/Sgd</a></li>
<li><a href="#" index="16" value="181454">Chf/Usd</a></li>
<li><a href="#" index="17" value="181402">Dkk/Usd</a></li>
<li><a href="#" index="18" value="181414">Eur/Aud</a></li>
<li><a href="#" index="19" value="176166">Eur/Byr</a></li>
<li><a href="#" index="20" value="181413">Eur/Cad</a></li>
<li><a href="#" index="21" value="106">Eur/Chf</a></li>
<li><a href="#" index="22" value="83226">Eur/Cny</a></li>
<li><a href="#" index="23" value="88">Eur/Gbp</a></li>
<li><a href="#" index="24" value="181407">Eur/Hkd</a></li>
<li><a href="#" index="25" value="181422">Eur/Huf</a></li>
<li><a href="#" index="26" value="84">Eur/Jpy</a></li>
<li><a href="#" index="27" value="176172">Eur/Kzt</a></li>
<li><a href="#" index="28" value="176177">Eur/Lvl</a></li>
<li><a href="#" index="29" value="176179">Eur/Mdl</a></li>
<li><a href="#" index="30" value="181401">Eur/Nok</a></li>
<li><a href="#" index="31" value="181415">Eur/Nzd</a></li>
<li><a href="#" index="32" value="66860">Eur/Rub</a></li>
<li><a href="#" index="33" value="181406">Eur/Sek</a></li>
<li><a href="#" index="34" value="181395">Eur/Sgd</a></li>
<li><a href="#" index="35" value="176174">Eur/Tjs</a></li>
<li><a href="#" index="36" value="176168">Eur/Uah</a></li>
<li><a href="#" index="37" value="83">Eur/Usd</a></li>
<li><a href="#" index="38" value="176170">Eur/Uzs</a></li>
<li><a href="#" index="39" value="181412">Gbp/Aud</a></li>
<li><a href="#" index="40" value="181388">Gbp/Cad</a></li>
<li><a href="#" index="41" value="181387">Gbp/Chf</a></li>
<li><a href="#" index="42" value="181386">Gbp/Jpy</a></li>
<li><a href="#" index="43" value="181400">Gbp/Nok</a></li>
<li><a href="#" index="44" value="181405">Gbp/Sek</a></li>
<li><a href="#" index="45" value="181394">Gbp/Sgd</a></li>
<li><a href="#" index="46" value="86">Gbp/Usd</a></li>
<li><a href="#" index="47" value="181420">Hkd/Usd</a></li>
<li><a href="#" index="48" value="181421">Huf/Usd</a></li>
<li><a href="#" index="49" value="181450">Jpy/Usd</a></li>
<li><a href="#" index="50" value="181385">Mxn/Usd</a></li>
<li><a href="#" index="51" value="181399">Nok/Usd</a></li>
<li><a href="#" index="52" value="181392">Nzd/Cad</a></li>
<li><a href="#" index="53" value="181391">Nzd/Jpy</a></li>
<li><a href="#" index="54" value="181398">Nzd/Sgd</a></li>
<li><a href="#" index="55" value="181425">Nzd/Usd</a></li>
<li><a href="#" index="56" value="181423">Pln/Usd</a></li>
<li><a href="#" index="57" value="176165">Rub/Eur</a></li>
<li><a href="#" index="58" value="176176">Rub/Lvl</a></li>
<li><a href="#" index="59" value="176164">Rub/Usd</a></li>
<li><a href="#" index="60" value="181404">Sek/Usd</a></li>
<li><a href="#" index="61" value="181397">Sgd/Jpy</a></li>
<li><a href="#" index="62" value="181393">Sgd/Usd</a></li>
<li><a href="#" index="63" value="176167">Usd/Byr</a></li>
<li><a href="#" index="64" value="66700">Usd/Cad</a></li>
<li><a href="#" index="65" value="85">Usd/Chf</a></li>
<li><a href="#" index="66" value="83225">Usd/Cny</a></li>
<li><a href="#" index="67" value="82">Usd/Dem</a></li>
<li><a href="#" index="68" value="182106">Usd/Idr</a></li>
<li><a href="#" index="69" value="181873">Usd/Inr</a></li>
<li><a href="#" index="70" value="87">Usd/Jpy</a></li>
<li><a href="#" index="71" value="176173">Usd/Kzt</a></li>
<li><a href="#" index="72" value="176178">Usd/Lvl</a></li>
<li><a href="#" index="73" value="176180">Usd/Mdl</a></li>
<li><a href="#" index="74" value="901">Usd/Rub</a></li>
<li><a href="#" index="75" value="176175">Usd/Tjs</a></li>
<li><a href="#" index="76" value="176169">Usd/Uah</a></li>
<li><a href="#" index="77" value="176171">Usd/Uzs</a></li>
<li><a href="#" index="78" value="181426">Xag</a></li>
<li><a href="#" index="79" value="181427">Xau</a></li>
<li><a href="#" index="80" value="181424">Zar/Usd</a></li></ul>

    The only difference in requests that bypass the restrictions and that don't is the header: "Referer:http://www.finam.ru/analysis/profile2C4A000007/"

    Here is the link to tick data

    http://195.128.78.52/AUDJPY_140526_140526.txt?market=5&em=181408&code=AUDJPY&df=26&mf=4&yf=2014&dt=26&mt=4&yt=2014&p=1&f=AUDJPY_140526_140526&e=.txt&cn=AUDJPY&dtf=1&tmf=1&MSOR=0&mstime=on&mstimever=1&sep=1&sep2=1&datf=6&at=1

    Here are the headers of the original request that worked:

    Remote Address:195.128.78.52:80
Request URL:http://195.128.78.52/AUDJPY_140526_140526.txt?market=5&em=181408&code=AUDJPY&df=26&mf=4&yf=2014&dt=26&mt=4&yt=2014&p=1&f=AUDJPY_140526_140526&e=.txt&cn=AUDJPY&dtf=1&tmf=1&MSOR=0&mstime=on&mstimever=1&sep=1&sep2=1&datf=6&at=1
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4,it;q=0.2
Connection:keep-alive
Host:195.128.78.52
Referer:http://www.finam.ru/analysis/profile2C4A000007/
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36

    Here are the headers of the request that did not work:

    Remote Address:195.128.78.52:80
Request URL:http://195.128.78.52/AUDJPY_140526_140526.txt?market=5&em=181408&code=AUDJPY&df=26&mf=4&yf=2014&dt=26&mt=4&yt=2014&p=1&f=AUDJPY_140526_140526&e=.txt&cn=AUDJPY&dtf=1&tmf=1&MSOR=0&mstime=on&mstimever=1&sep=1&sep2=1&datf=6&at=1
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4,it;q=0.2
Connection:keep-alive
Host:195.128.78.52
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36

    The code to bypass restrictions and get the tick data:

    using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApplication6
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = new WebClient();
            client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36");
            client.Headers.Add("Accept-Language", "ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4,it;q=0.2");
            client.Headers.Add("Referer", "http://www.finam.ru/analysis/profile2C4A000007/");


            string url = "http://195.128.78.52/AUDJPY_140526_140526.txt?market=5&em=181408&code=AUDJPY&df=26&mf=4&yf=2014&dt=26&mt=4&yt=2014&p=1&f=AUDJPY_140526_140526&e=.txt&cn=AUDJPY&dtf=1&tmf=1&MSOR=0&mstime=on&mstimever=1&sep=1&sep2=1&datf=6&at=1";
            string htmlString = client.DownloadString(url);

            Console.WriteLine(htmlString);
            Console.ReadLine();
        }
    }
}