Data gaps and bad ticks - error source?

[Haeggor]

Hey guys,

I am reposting this since I did not see the thread after submitting it and there was no "awaiting moderation" message or similar, sorry if this is a duplicate.

I was only surfing through the crosses that contain eur and/or usd only so far. I have found several crossrates that have bad ticks that can not be true market movement, as well as massive data holes that seem to be years long in many cases.

I have downloaded data from 30 march 07 to 03 jan 15. Then I have exported them to csv with 1min timeframe.
data format is:
{BarBeginTime:yyyy.MM.dd} {BarBeginTime:HH:mm:ss},{Open},{High},{Low},{Close},{Volume}
Tickstory is v1.5.2.0. I'm on win7 64bit, 32gig ram.

Below the list of problems I found on a first visual and simple statistics check:
EURNOK has no data for 2008
EURSEK has no data in at least 2009
EURDKK has a massive spike at ca. 234
eurjpy has a massive downspike to 0.677
eurrub has several missing dataspans from 2013-2014 and 2014-2015 ca

usdnzd has a spike at 101
usdbrl has two spikes, the higher one at 1022
usdhkd has a massive gap in the middle
usdsgd has a gap between 2009-2011

When looking at the 1min data in the tickstory viewer, the chart looks intact, however, in the csv export the data is corrupted. See eurrub screenshot - left is tickstory viewer, right is pandas plotting function (which is consistent with the .csv data, I checked).

So I wonder what the problem is - has dukascopy missing data and it can't get any better then that, or is there a problem with download or csv export? Any ideas?

thanks

[tickstory]

Hi Haeggor,

Tickstory utilises the Dukascopy as-is without making any changes to the data (although it does construct bars from the tick data). We did a check with some of the data points you mentioned on the EURRUB. In this case, there do appear to be gaps in the originating data from Dukascopy. These gaps are shown in the Tickstory charts if you zoom in on the period. We also cross-checked that these gaps were actually in the raw data from Dukascopy. In the case of a gap between July-August 2014, you can browse to Dukascopy store location to view the "raw" data:

(Dukascopy Store Location)\EURRUB\2014\07\06

Note there are zero-sized data files indicating there was no data for this period. We have downloaded the data outside of Tickstory and confirmed that Dukascopy is delivering these zero-sized files.

I can only assume that these gaps are due to times where the exchange was illiquid or not trading for some reason.

If you want to cross-check any other data you can do so by downloading the Dukascopy JForex platform. Suffice to say that we are unaware of any Tickstory bugs that would be causing these gap & spike issues.
Do note that in future versions of Tickstory we will be offering an analysis feature to highlight these gaps and also smooth any anomalous spikes.

Hope this helps.

[mainw]

agree that a analysis feature for highlighting gaps will be very useful.

I've been using the "export to mt4" in tickstory and what what i missed (comparing it to csv2fxt) is that there's no logs telling me if there are any gaps in the generated fxt/hst.

Also, may i know how to use the "include weekends" option when exporting to mt4? My broker is aligned with "5pm New York market close", so for me that means:

1. inlcude weekends - unchecked
2. timezone - leave it at UTC
3. daylight savings - unchecked

Is my understanding correct?

Also, is it possible to append to fxt/hst? Am asking because it seems each time (say every month) i need to backtest with the latest data, I have to build the fxt/hst from the start all over again. There must be a better way?

[tickstory]

Hi Mainw,

There is a Timezone called "(EST +07:00) New York Trading Hours" at the very top of the selection combo which is likely what you're looking for.

Regarding the feature to extend an existing FXT file, this is one of the features we are currently working on for the next major upgrade of Tickstory.

Kind regards.

[mainw]

thanks for the reply.

How about the "include weekends" option? I still don't quite understand what it does even after reading the faq

appending to an existing FXT file will be superb, can't wait

[tickstory]

Hi Mainw,

The 'filter weekends' option filters out all data between Friday 22:00hrs to Sunday 21:00hrs UTC (in most cases, Dukascopy has no data between these periods however there may be occasions where there is).

Kind regards.