Tickstory

Hello Admin,

I'm new here.
I also have a plan for creating my own tick data downloading and maintaining solution, but if you could implement my suggestions below, then I would willingly drop my project and could focus on trading related programming tasks again.

First, eliminating duplicate ticks would be a great improvement. But as I've read, you'll introduce this feature in next version.

Then, my solution would have an "auto-update" feature. This could happen, for example, by calling Tickstory as a scheduled task at end of every day or week (user selectable), and it would download the new data from Dukascopy for all (or the selected) pairs, then it would update the .fxt files automatically (if MT4 is not running, of course). This way we always could have a fresh, up to date set of .fxt files without the need of manual procedure. Of course, we must make an updating setup first (what pairs, when, which TFs, etc.), but then it would be automatic thereafter.

Next, during download, I would decompress the .bi5 files from Dukascopy and would make a new file with 4 byte integers instead of the 8 byte doubles (or longs) of the original. The timestamp could be in yyyy.mm.dd hh:mm:ss MT4 format without the millisecond part, as it would not be an issue at creating .fxt files. Also the prices could be multiplied by 100000 and stored as 4 byte integers, just like the volumes. Later, at creating .fxt or .csv files these price/volume data could be retrieved by dividing the integer by 100000 and normalizing the result to 2/3/4/5 digits (determined by fxt header info). This way the uncompressed data would be 50% of the original size and after compression it would take around 30-40% less space on the storage media. You could even unify the hourly data and store them as days instead, since backtester's smallest time interval is 1 day anyways. This would make even smaller data files after compression and also would reduce number of file accesses later, at generating .fxt's.

Also, I'd implement an .fxt maintainer, which could manage existing .fxt files. I mean we could re-adjust .fxt files' parameters, like spread, leverage, commission, stop level, swap etc. without generating a new file every time.

-----------------------------

There are also some issues that needs to be investigated in my opinion:

- The date fields should be in MT4's format (yyyy.mm.dd) instead of the dd/mm/yyyy, as it is confusing for non-english speaking people. MT4 date would be a generally understood format, anyways.

- Every set of tick data of every pair on Dukascopy has a starting date. For example EURUSD tick data begins on 2007.03.30. At downloading part, if no custom interval was selected yet, these default dates could appear in "from" field to make things easier for those who want to download the whole available interval. These data could be found in Birt's csv2fxt script or I can also provide them, if needed.

- Just noticed, that if I want to convert Dukas data from the very beginning, and this starting date/time is not a "round" number (like 00:00 at beginning of a day), then .fxt ticks will be shifted if timeframe is bigger than M1. For example, on Dukascopy EURUSD tick data is beginning on 2007.03.30 16:01. Tickstory is taking this date as a base and builds bars data shifted by 1 minute. So, if someone wants to make full size tick data in an M5 .fxt, then there will be 16:01, 16:06, 16:11, 16:16, 16:21, 16:26 etc. bars instead of 16:05, 16:10, 16:15, 16:20, 16:25 etc. ones. Of course, if you start at 2007.03.31., then fxt conversion will begin at 00:00 and the 1 minute offset will not present.

- I wanted to compare Tickstory generated fxt files with Birt's csv2fxt output. For this, I have used Tickstory to export downloaded data to csv. Then I have generated the fxt files using the csv2fxt script, but the backtester could not accept them (backtest ended immediately after start, no error message was produced). When I replaced the fxt to the one that was generated by Tickstory, the backtest has been completed correctly. Since I have already used csv2fxt before and it produced correct fxt files, I guess the Tickstory produced csv file can be the reason of the wrong fxt's. What do you think? (those floating point volumes are looking quite interesting, for example)

- I cant make .fxt and .hst files in other than preset MT4 folders. I can select the folder where I want to put the generated files, but Tickstory will yet to put them into the MT4 folder that was set before.

That's all for now.
I hope we can help each other.

Hi Tom,

Welcome to the forum - thanks for your comprehensive post. Let me address each one of your points:

- Auto-updating of data: This feature is on the list of enhancements and will be part of the "new generation" Tickstory.

- Data storage: Dukascopy data is already stored in a significantly optimised format as part of their data format changes last year. This includes storing in the manner you described along with utilising the fo;e directory structure to provide the baseline timestamp.
In order to offer optimised data retrieval, we are also working on our own tick database which should sport a competitive compression scheme and allow much faster data management.

- MT4 FXT management tool: This is currently on our list of things to do (TICKLITE-15).

- Dukascopy data date-ranges: If you have the dates handy for the instrument date-ranges that would be great. These can be incorporated into the system (TICKLITE-78).

- Candle time starting at 1 minute: Thanks for noting this. A ticket has been raised to investigate this further. I would recommend the workaround you suggested (i.e. selecting one day later) if you want to export data from the beginning of Dukascopy's dataset. (TICKLITE-79)

- Date format on export screens: Thanks for highlight this. This will be addressed in the upcoming version (TICKLITE-77).

- CSV2FXT conversion using Tickstory export: It could well be some validation that is being done on the volume in MT4. Have you tried to export the data with a volume of 1 and 0 just to rule this out?

- Custom HST/FXT locations: This has been raised as an issue and will be addressed in the upcoming version (TICKLITE-76).

Thanks for your feedback!

Thanks for answers. I'm awaiting new versions.

Meanwhile I've read posts about "gaps" in tick data. I think these gaps (missing data) are due to missing .bi5 files. It's just happened to me today while I was downloading GPBUSD tick data. Some .bi5 files were missing here and there but then the whole September of 2008 was simply "skipped" and not downloaded at all. I had to run download procedure again for GBPUSD and that time September data were downloaded completely. Now I'm checking all the downloaded data of other pairs and already found missing files and whole parts in them, too.
I don't know for sure what is causing this, it could be network error or a bug. I think (hope) you're doing several retries in case of download errors before the program is giving up and continues. I'd suggest to make an internal logfile for these errors. When downloading has finished, the program could try downloading the erroneous files in that log again, with extensive retrying attempts, because it seems Dukascopy's server is sometimes can't withstand the numerous requests and simply refuses serving the file in question. Further help could be that a year (365 days) will consist of 8760 files (hours). A leap year has 8784 files (hours). If someone is downloading all the data, then you simply can check number of files in a year, for example. Or you can calculate the number of files to be downloaded based on starting and ending date interval. If the number of downloaded file is not matches calculated value, then there must be a download error, it's worth retrying download procedure. But I think the "logfile of errors" is a better solution.

Also, I have tried to copy some data from the log window, but I was not able to do that. Will you please allow copying of log entries from that window? Or is there a way to save the logfile? I have tried to find it in various temp folders, but had no success. I wanted to keep some entries regarding missing files.

I am now working on actual list of starting dates of tick data. I'll post it here if I've finished.

Hi Tom,

Thanks again for the feedback. Regarding retries, yes, Tickstory does attempt to re-download the data up to 3 times at the moment. Do you have the error that you got from the server (screenshot will do)? There have been reports of some network errors however it wasn't conclusive that this happened on the Dukascopy end. If it's possible reproduce what you're seeing then that would be handy. The plan is to have a check for data gaps in the future so this can be done as a post-check (as opposed to logs which can get unwieldy).
At the moment the log viewer does not have the ability to copy/paste entries - this has been put on the list and will be released in the upcoming version (TICKLITE-82).

Kind regards.

I have already closed Tickstory since that, and I was not able to copy the error messages out from the onscreen log, so I don't have it. It's something like "unable to resolve URL for xxxxxx.bi5 file" or something, so it's not a numbered error (like 404 or 500 etc.). It's a network error however, and it seems Tickstory is giving up too early. Of course, it can't retry forever, but it could tell it about somehow. Like it were write with red letters next to the symbol "Download finished, but not completed. XX files are missing, please retry". If everything was downloaded correctly, then the usual "...completed" text would be written in black.
I'm trying to download another pair's data and now I'll take a snapshot of the error.

Hi Tom - That's right, unfortunately it can't retry indefinitely and logs aren't the most ideal way to communicate an issue. This will be revisited in the new version. For the moment however, as per the trouble-shooting manual, the recommendation is to resolve any recurring errors before using your export.

Hope this helps.

I've just finished discovering starting dates for each Tickstory pair's tick data. These could be used as default starting date instead of today's date. See below:

Format:
PAIR Time (in mql format of yyyy.MM.dd hh:mm)
There are a few dates missing, because Tickstory was not able to download anything in these cases. I guess Dukascopy just has plans for making these pairs available, but there are no tickdata for download at the moment.
It is advisable to re-check these starting dates periodically in the future (especially the ones after 2010-2011), because they might be changed (extended). For example XAUUSD's starting date has just extended from 2011.05.10 07:00 to 2010.11.11 16:00. I think there will be further extensions regarding new pairs (for example the ones with East-European currencies involved).

Hi Tom,

Many thanks for getting back with these details. Given that the date ranges could potentially vary, it sounds like it's better to prompt with a warning if users try to go beyond but not disallow it altogether.

We'll figure out how best to implement this in the revamped version.

Kind regards.

Hi Tom,

Thanks for all your suggestions and feedback. A number of your items have been addressed in the latest v0.8 Release Candidate which is currently undergoing testing. If you want to get a hands-on look prior to the official release, please refer to the following post.

http://www.tickstory.com/forum/viewtopic.php?f=2&t=122

If you come across any issues, please let us know.

Regards.

Thanks for the heads up. I'll peek into it.

Edit: done...

- It's not saving main window's size and position. It's needed to be pulled and tossed around every time when I start the program.

- Download date format is still english (dd/mm/yyyy). It's not just the "look", but for example when I want to give "30" for the day and the month is still on "02" then it won't allow it and changes day number back to original. It's ok if I can notice it, but if not, then it could ruin several hours of downloading session because the wrong date... Regular MT4 format would be good (YYYY.mm.dd)

- Export to MT4, Data export tab:
- From date should be the date of the earliest downloaded data (if any). If no data exist yet, then you could use the date from the table I've posted yesterday. Also, To date could be the date of latest downloaded data. If no data exist yet, then it could be today's date.
- Adjust Timezone: put "UTC (Coordinated Universal Time)" by default. While the current default (UTC) Monrovia, Reykjavík is basically the same (GMT+0, no DST), yet it could confuse some users. By default, Dukascopy has GMT+0, no DST on their data.
- Filter duplicate ticks: should be enabled by default as it is more common and is producing less than half sized files (4 GB barrier issue will disappear even for pairs with 2007.03.30 beginning date and today's ending date). And of course, backtests will be about twice as fast as with duplicate ticks.

- Export to MT4, MetaTrader info tab:
- "Load" button forces loading files with .mt4config extension only. However, the TickstoryInfoExpert EA can save files with any name. I have saved the info with a .txt extension and Tickstory was not able to find it, of course. I'd suggest either to put the ".mt4config" extension after the user given filename in the EA or allow loading files with other extensions in TickStory. The first one is the quicker way, the second one needs some checks if the file to be downloaded is a valid Tickstory mt4config file or not.

- Export to file:
- It should have the same defaults for dates like in MT4 export.
- There is no field for custom path for the file to be saved. Tickstory is putting the .csv in its own directory. I'd suggest to put the .csv into the downloaded pair's main directory. Like if it was EURUSD, then .csv would go to in EURUSD directory by default. Of course, user could change that.

- Log window: copying from menu is ok, copying by Ctrl+C is ok, copying by Ctrl+Insert (which I'm using the most) is doing nothing.

That's all for now. I'm currently trying to figure out what are the differences between your .csv and Birt's .csv