[NaddmrParticipant April 25, 2015 at 10:28 pm Post count: 3]

in reply to: Historic Rate Data from Gain Capital #20509

Hi folks,

I use their data since 2012 to build up my own tick based data ware house.
Their data is pretty accurate albeit they changed their format a few times so the import script had to deal with the changes accordingly.

Here’s the part of my perl script with the different regexps to break up their formats to get started:

if (/((^d*),([A-Z]*)/([A-Z]*),(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}.000),(d*.d*,)(d*.d*,)D)/) {

my $fract=(split(/,/))[0];

if (length($fract) > 3 ) {

$fract=substr($fract, length($fract)-3,3);

}

$fract=sprintf("%03d", $fract);

$tc_symbol=(split(/,/))[1];

$tc_time=(split(/,/))[2];

$tc_time=(split(/./, $tc_time))[0] . "." . $fract;

$tc_bid=(split(/,/))[3];

$tc_ask=(split(/,/))[4];

$struct_ok=1;

}

# Format 2: 

elsif (/(^(d*),"([A-Z]*)/([A-Z]*)",(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}),("(d*).(d*)s*"),("(d*).(d*)s*"))/) {

$tc_symbol=(split(/,/))[1];

$tc_symbol=~s/"//g;

my $fract=(split(/,/))[0];

if (length($fract) > 3 ) {

$fract=substr($fract, length($fract)-3,3);

}

$fract=sprintf("%03d", $fract);

$tc_time=(split(/,/))[2] . "." . $fract;

$tc_bid=(split(/,/))[3];

$tc_bid=~s/"|s//g;

$tc_ask=(split(/,/))[4];

$tc_ask=~s/"|s//g;

$struct_ok=1;

}

# Format 3: 

elsif (/(^(d*),"([A-Z]*)/([A-Z]*)",(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}),(d*.d*),(d*.d*),"D")/) {

my $fract=(split(/,/))[0];

if (length($fract) > 3 ) {

$fract=substr($fract, length($fract)-3,3);

}

$fract=sprintf("%03d", $fract);

$tc_symbol=(split(/,/))[1];

$tc_symbol=~s/"//g;

$tc_time=(split(/,/))[2] . "." . $fract;

$tc_bid=(split(/,/))[3];

$tc_ask=(split(/,/))[4];

$struct_ok=1;

}

# Format 4: 

elsif (/(^(d*),([A-Z]*)/([A-Z]*),(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}),(d*.d*),(d*.d*),D)/) {

my $fract=(split(/,/))[0];

if (length($fract) > 3 ) {

$fract=substr($fract, length($fract)-3,3);

}

$fract=sprintf("%03d", $fract);

$tc_symbol=(split(/,/))[1];

$tc_time=(split(/,/))[2] . "." . $fract;

$tc_bid=(split(/,/))[3];

$tc_ask=(split(/,/))[4];

$struct_ok=1;

}

# Format 5: 

elsif (/(^(d*),D,([A-Z]*)/([A-Z]*),(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}),(d*.d*),(d*.d*))/) {

my $fract=(split(/,/))[0];

if (length($fract) > 3 ) {

$fract=substr($fract, length($fract)-3,3);

}

$fract=sprintf("%03d", $fract);

$tc_symbol=(split(/,/))[2];

$tc_time=(split(/,/))[3] . "." . $fract;

$tc_bid=(split(/,/))[4];

$tc_ask=(split(/,/))[5];

$struct_ok=1;



}

# Format 6: 

elsif (/(^(d*),D,([A-Z]*)/([A-Z]*),(d{4}-d{2}-d{2}sd{2}:d{2}:d{2}.d{9}),(d*.d*),(d*.d*))/) {

$tc_symbol=(split(/,/))[2];

$tc_time=(split(/,/))[3];

$tc_bid=(split(/,/))[4];

$tc_ask=(split(/,/))[5];

$struct_ok=1;

}

else {

print("$name mismatch: n");

}

The most outstanding feature of their tick data is the time range covered. Some pairs have tick data as early as March 2003.

[NaddmrParticipant April 20, 2015 at 5:44 am Post count: 3]

in reply to: HST files overwritten despite "Generate HST-file" is clear #20664

Hi Mr. Tickstory,

this was quick. Thanks a lot for the fast reply.

Like said before: The “Remember-last-settings-thing” is more of a comfort thing than a real issue.

And for the creation of unneeded HST files I’m currently going the detour of pasting an empty writeable directory to keep my old ones untouched.

I already tried to write protect the HST folder, but then the export fails completely – which is to be expected from a clean error handling method. 😉

Thanks again and have a nice and successful week ahead.

Naddmr

[Author]

Posts