Checking for missing data

[jfroth82]

First of all - thanks for the great software. I'm really enjoying using it.

Second - I wanted to ask if you support any kind of checking for missing data. Let's say I downloaded 1 year of historical data and I'd like to know if there are any gaps in that data before I start doing my testing. Does Tickstory do any kind of checking for that?

Thanks again and keep up the great work.

[admin]

Hi jfroth - welcome to the forum and thank you for the positive feedback.

The download logic is explained in the user manual (http://www.tickstory.com/help/tickstorylite). Basically it will fill in any gaps you have in your data. If, for example, you only downloaded the odd week or two and then decided to download the entire 1 year, the system will only retrieve the missing portions from the web and refer to the "store location" for the portions that have already been downloaded.

If you are referring to the quality of the data, at the moment it assumes that there are no gaps in the Dukascopy data between the first date that is available to the current day. If there was any gap, you would receive an error message in the alert box (regardless of whether you're downloading or exporting).

Please let me know if this clarifies.

Thanks.

[jfroth82]

Thanks a lot for your helpful reply. I was referring to data gaps from the source (Dukascopy) and if there are any checks in Tickstory to make sure the data that's imported doesn't have any gaps.

I'm a little confused, though, because you said: "at the moment it assumes that there are no gaps in the Dukascopy data between the first date that is available to the current day" and then "If there was any gap, you would receive an error message in the alert box" which seem like contradictory statements.

So what exactly would happen if you downloaded data from Dukascopy with gaps and then tried to export. Did you mean that it would download the data without any warnings but then when you would export it, it would give you a warning? So that would mean there IS a gap checking mechanism during the data export phase? If that's the case, then is it intelligent enough to know about days that the market is closed (e.g. Christmas, New Year) and not treat those as gap days?

Thanks again for a great product.

[admin]

Hi jfroth,

Let's see if I can clarify both parts of the question of "gaps" in the data.

Firstly, as far as the quality of Dukascopy data goes, there is an assumption that the data is complete - that is, it always has tick data for all days that the market was open and trading. For those days that the market was not trading (holidays, weekends, etc), Dukascopy still provides an empty data file for that day/hour.

Secondly, from a technical standpoint, the question is whether all this data is downloaded by Tickstory completely. The answer to this:

• Tickstory will attempt to download all information for every day, regardless of whether the market was open. For those days that the market was not open, Tickstory will still download the zero length file that Dukascopy provides. All information is downloaded and stored as-is by Tickstory.
• In the event any particular piece of tick data is missing, corrupted or in any way inaccessible (out of disk space, etc) Tickstory will alert that there was an error. If you see any errors while downloading or exporting, you need to trouble-shoot this by reading the FAQ. In the event of an error, your export may or may not be missing important data depending on the error. For example:
  
  • Out of disk space. In this case, you will miss information in your final export and you should rectify the situation and redo your export.
  • Internet disconnection/proxy issue. In this case, you will miss information in your final export and you should rectify the situation and redo your export.
  • Data corruption. In this case, data will be re-downloaded by Tickstory and usually recovers correctly. It is unlikely that your export was effected, however to make sure you can retry the export to ensure the error is not recurring.
  • Requesting data outside the range of data that Dukascopy currently provides for that symbol (for eg. trying to download XAGUSD from 2007 which Dukascopy does not have data for or downloading for today's date which Dukascopy doesn't make available until a day later). In this case you will obviously miss this information but only because it's not available.
• In the event you download data for say the days between 10th Oct 2012 and 17th Oct 2012 and then subsequently do an export for 1st Oct 2012 and 31st Oct 2012, Tickstory will automatically determine which data has been downloaded by firstly checking the local disk and, if it does not exist, go to the Dukascopy site for retrieval. In our example, it will see that 1-9th have not been downloaded, automatically download and store the data. 10th-17th already exist locally on disk, so it will use that. 18th to the 31st will be automatically downloaded from Dukascopy and stored and your export will be complete. If you subsequently do an export for the same date range, the system will retrieve all the data from the local disk because that's where it now resides.
  In other words, it is not necessary to explicitly download before exporting. The download option is merely there for convenience so you can download in say an overnight batch, ready for export the next day.

Hope this helps answer your questions!

[JohnTaylorHK]

Hi All, I have a script which will find data holes in MT4. Not sure how to attach a file here so please PM me at john.taylor.hk@gmail.com for a copy,

Best

John