MT4/tickstory/TDS

[imported_tickstoryParticipant October 10, 2013 at 10:40 am Post count: 1479]

#19385

Hi Gabriele,

Given that you are just getting familiar with back-testing, we would recommend that you consult the Help guide to find out more about how MT4 databases work (and also Tickstory). You can go to the Help guide by going to this link: http://www.tickstory.com/help/tickstorylite/ or simply pressing F1 when you are using Tickstory. The section on “MT4 – A Quick Primer” will be useful for you to learn more about FXT & HST files. There is also a section in “The Basics”->”Downloading data” which also explains how data is retrieved from Dukascopy and stored on your local computer so it does not need to continually request this information over the Internet.

To address some of your other questions, note that MT4 databases (and therefore HST and FXT files) do not get generated until you perform an ‘Export’ from Tickstory (this is also explained in the Help guide). This may be why you are not seeing FXT files (and the existing HST files may already exist because they were downloaded from your broker).

As a starting point, have a look at our 5-step video guide for using Tickstory to back-test:

http://www.youtube.com/watch?v=R-rMOen4UHg

Hope this helps.

[gabrieleParticipant October 10, 2013 at 12:51 pm Post count: 10]

#19386

tickstory,

thanks for your reply.
However the step by step guide and help pages do no provide any of the answers I am looking for.
I have followed every single step , and I have found myself in the position described on my previous post.
So no luck for me on the guide, my only hopes are for some generous soul to reply with step by step instructions for my previous questions and failing that, I will have to try to do a new download and start from scratch again.

Gabriele

[imported_tickstoryParticipant October 11, 2013 at 9:39 pm Post count: 1479]

#19387

Hi Gabriele,

Here is a response to all your questions above:

So my first question is should I have a matching number of HST and FXT files?

A HST file is generated for every time-frame. If you have generated FXTs for every time-frame then yes, you will also have the same number of FXT files.

If so then, is there a known issue on why the HST files are downloaded but not the FXTs?

If you have exported from Tickstory you will automatically generate the FXT files.

Do I need to download the data again or can I extract them from my Ctickstory Lite folders?

You don’t need to touch the Dukascopy data files (*.bi5 files) in any way. These files are automatically downloaded and contain the tick data used by Tickstory to export to MT4. Once they are downloaded they do not get downloaded again.

Also, how can I make sure that once downloaded, the data stays on my pc? I have the feeling, but not the proof, that some of the files get deleted as I close my MT4 and shut down my pc.

– Data downloaded from Dukascopy is never deleted unless you explicitly choose to delete it.
– Data that you export for MT4 from Tickstory will reside in the Metatrader directory. Ensure you have selected ‘Mark files read-only’ when you do your export and always launch MT4 via the Tickstory ‘Launch’ feature (press F8 in the application). This will ensure MT4 cannot delete your FXT files. If you do not follow the ‘Launch’ procedure then yes, it is possible that MT4 will delete the FXT files when you try to back-test (and if these files are deleted then, yes, you will need to generate them again).

Hope this answers your questions.

Regards.

[gabrieleParticipant October 14, 2013 at 9:44 am Post count: 10]

#19388

thanks for you exhaustive and precise reply.

Something is not working on my side.
I have followed the guide rigorously and yes the “make FXT read-only box” is ticked.
However my metatrader/tester/history folder only keeps files until I close my machine.
If I just log out, and never mind if I restart my pc, and I re-open tickstory and then metatreder from there with F8, the FXT files are no longer there.

Therefore so far in order to perform a back test I need to download the data, over and over again, before being able to run a test.

regards,

Gabriele

[imported_tickstoryParticipant October 14, 2013 at 10:23 am Post count: 1479]

#19389

Hi Gabriele,

I’m not sure if your issue has something to do with Windows 7 file virtualisation? The first thing to do is identify exactly when your files are deleted so you can hone-in on the culprit.
Check the files just before shutdown/logout, then when you start up again, then when you start MT4 and then when you hit the ‘back-test’ button. Once you know the exact point at which your files are deleted you can most likely find out why this is occurring. As far as we’re aware, if the files are marked as read-only, MT4 cannot delete them.

Hope this helps.

[gabrieleParticipant October 14, 2013 at 12:36 pm Post count: 10]

#19390

Thank you.
I am using windows vista business.
I will try to establish when the files are getting deleted.

kind regards,

Gabriele

[experienced_testerParticipant October 20, 2013 at 5:59 am Post count: 17]

#19391

Gabriele

First you have to check if the generated FXT files are read-only. This check should be done immediately after downloading/exporting the data towards the mt4.

If they are not read-only then we face the same issue, you can check up my thread here:
viewtopic.php?f=2&t=246

although I am using a different operating system than yours: XP

Hope this helps, good luck.

[gabrieleParticipant October 23, 2013 at 10:03 am Post count: 10]

#19392

Dear expert tester,
thanks for your post. While it does not solve my issue, it does make feel better as at least I am not the only “mad guy” in town 🙂

Something is seriously wrong.
117 files for over 100 GB of data have disappeared from my database over the week end.
C:Program FilesMetaTrader – Alpari UKtesterhistory

I am following all the instructions.

I believe the issue lies either with ticksory, with MT4 or possibly both working together.
Last week in order to protect my work, following previous disappearances, I made a backup of all my downloaded data to an external HD.
The data on the external HD is still there, while the files saved on the above directory have gone!

Following the post from expert tester, I have checked both the few files left on my C drive and all the files that I still have on the external HD.
The are all saved as “read only”.

However I have noticed something quite interesting.
Yesterday by mistake I had started to download a new pair, only to stop the process when I realized that the download was not intended.
This morning those “mistaken files” were on my C drive, but they were not “read-only”

I have re opened tickstory to see if there was a box to be ticked for the files to be downloaded as “read only” and I couldn’t find it.
Am I missing something?

Whether or not my missing files were saved in C as read-only or not, why would they get deleted anyway?

And if indeed the missing files were not saved as read only, how comes that the copy I made from my C drive to the external HD were transferred as “read only” ?

And how is possible that following always the same “Exprt to MT4” process some files are most definitely being saved as “read only”, and possibly some others (the missing ones) are not?

Could anyone help?

Kind regards,

Gabriele

[experienced_testerParticipant October 23, 2013 at 4:10 pm Post count: 17]

#19393

Hi Gabriele

1. Even if you studied and applied the rules let me remind once more one of them, just in case:
– you choose an mt4 terminal for backtest and then you NEVER start that terminal by itself, without Tickstory. The terminal for backtest purposes will only be started through Tickstory. It is better for you to delete the terminal shortcut from the desktop.

2. No offense, you did not answer my question:
– immediately after downloading/exporting the data, the FXT files are read-only? In order to answer this question you could download/export a short interval, like a week of data. This will be fast enough.
I remind you the option “Make FXT read-only” in the export menu is checked by default. In fact you can leave untouched that group of 5 default settings.

Try again, good luck.

Kind regards

[imported_tickstoryParticipant October 24, 2013 at 1:21 am Post count: 1479]

#19394

Hi Gabriele,

Just to add to Experienced_tester’s helpful response, please note that if you prematurely stop an export then Tickstory does not have the opportunity to fully complete the process – this means it will not mark the data as read-only, nor will the exported data be complete/usable.

Regarding your question about why data is deleted, just a bit of background for you: MT4 does not natively support the use of tick data. Whenever you do a back-test, the first thing it does is delete the existing FXT files and freshly generates it from the historical data center. Unfortunately the most granular data stored in the historical data center is 1-minute data. This is where the Tickstory export and launcher comes in: When you generate the tick data in Tickstory, it creates FXT files with accurate tick data. The launcher then prevents MT4 from deleting these files. If you don’t start MT4 via the launcher and the files are not marked as ‘read-only’, then the FXT files will be deleted.

Hope this helps.

[gabrieleParticipant October 24, 2013 at 12:00 pm Post count: 10]

#19395

Dear Exp_tester and Tickstory,
thank you both for your response and willingness to help, this is much appreciated.

Following your responses I have done the following:

1) Downloaded a month worth of data of a pair I never used before.
On my tickstory the option make FXT read only was ticked.
The only box that is un-ticked is filter duplicate ticks. I did tick the later as well, before the export to MT4

2) checked that FXT were downloaded as “read only”. They were not!

3) changed the attribute of 1 of the 8 downloaded files to “read only” and left 7 untouched.

4) started MT4 without launching it from Ticksory .

5) checked my history folder and all the 8 (7+1) files were there

6) removed the attribute of read only from the 1 file, so to have 8 normal files, and rebooted my pc

7) started again MT4 from the alpari shortcut, checked my history folder to find the the files are still there.

So here I have an issue on exporting the files, as they are not saved, or at least they were not in this last attempt, as “read only”.
The above can be easily solved by manually changing the attributes following a new download.

However I was not able to replicate this:
“The launcher then prevents MT4 from deleting these files. If you don’t start MT4 via the launcher and the files are not marked as ‘read-only’, then the FXT files will be deleted.”
Again for the sake of safety, I can remove the alpari shortcut and always open MT4 from tickstory, whether or not I am using MT4 for back testing.

Still any of the above “safety measures” can’t explain why 100 GB of data was lost and what caused it.
Please bear in mind that when I copied the files from my C directory to my external HD the files were automatically saved as “read only”, which point (although not in an exclusive way) to the fact that the source (the files on my C drive) were also originally saved as read only and this attribute was transferred across .

kind regards,

Gabriele

[experienced_testerParticipant October 25, 2013 at 4:26 pm Post count: 17]

#19396

Hi Gabriele

1. So I take it the FXT files still remain Read-only after you manually switched the properties. Then they are suitable for use.

2. It is easier for you to use this particular mt4 terminal for backtesting only. Remember there’s no need to be a good broker cause you will feed perfect tick data anyway. So use a broker you won’t ever need to run forward. It is preferable that the broker’s time to be GMT time.

3. The backtesting terminal will always be started ONLY through Tickstory launcher.

4. You can copy 1-2 files to that HDD again. A movie or a song, it doesn’t matter. Make sure they are not read-only. Then check their copies on that HDD: it is possible for them to be read-only.

Good luck, kind regards

[imported_tickstoryParticipant November 6, 2013 at 12:01 pm Post count: 1479]

#19397

Hi Gabriele,

After Step 7) I believe if you had attempted to press the back-test button on a particular symbol that did not have its FXT marked as ‘read-only’, this file would have been deleted by MT4 as part of its normal back-testing process (which is to clear the FXT files and regenerate them).

We unfortunately have not been able to reproduce the scenario where the FXT are not marked as read-only when the corresponding option has been selected. If you must, ensure the files are always marked as read-only before commencing any back-testing.

Hope this helps.

[Author]

Posts