Beginner questions - Setting up MT4 for back-testing

[billworld]

Hi BIllworld,

Could you please explain why you need to edit the FXT files that are being created by your FXCM MT4? If you are testing with FXCM data, there should be no need to edit these files as they will already contain the specific settings for the account you are using. This is normal behaviour for MT4 and you do not require Tickstory for this.

Thanks.

Are you kidding? I've explained and re-explained a number of times. I'm trying to backtest strategies with commissions applied. This requires editing the FXT using Tickstory. But, when the backtest is run the FXT is overwritten and the customized commissions data is wiped out. I've clearly explained this a number of times and at this point and starting to feel this is a hack piece of software which doesn't actually work.

[tickstory]

Hi Bill,

You have explained two scenarios for your back-testing:

1) Back-testing with FXCM data (and editing the FXT files for some reason). This scenario is not supported by our software and in fact should not require you to use Tickstory at all.
2) Back-testing with Tickstory data with specific commission settings. This scenario is supported and requires you to set the commission information (etc) when you first export the data from Tickstory. Once you have done this, there is no need to edit any FXT files as they will contain the information you specified when you exported. From there, you simply launch and run your back-test.

Apologies if we are seeking clarification, however you mention:

Regarding 2), at this point for various reasons I desire to use the data I've gotten from FXCM via their data downloader vs Tickstory data. I just need Tickstory to edit the FXT files so I can backtest with commissions. Will this work?

This is would be effectively Scenario 1) and we do not support typically support this, however we are willing to see if we can assist you in this endeavour. If we are talking about Scenario 2), this is the basic use-case of Tickstory.

Appreciate your patience while we try to assist you.

Thanks.

[billworld]

Support for Scenario 1) is what I was led to believe was possible from multiple sources include the following:

The following article "How To: Include Commission Cost During Back Test" http://www.jtatoday.info/forum/showthread.php?tid=20511 which led me to Tickstory says:

-------
Download your data using whatever you like, Birts, Tick Story, etc.
Export the Data to MT4 in whatever manner you like, I have been using SQ Tick Data Downloader and I like it. Whatever you choose should download and export it into the FXT format that MT4 needs
Once the FXT has been created, use Tick Story, and go to "Tools > MT4 Data Editor"
Click the "Open FXT" button and browse to your MT4 Data folder, locating the "Tester\History" folder. NOTE: This will be stored in a path such as the following by default "C:\Users\YourName\AppData\Roaming\MetaQuotes\Terminal\BB190E062770E27C3E79391AB0D1A117\tester\history"
Select the FXT file for the pair you wish to enter commission for.
Enter the correct commission for a round trip trade. In the image you see my round trip cost for 1 standard lot is listed as "Base Comm = 8". This would equate to an $8.00 trade cost
Enter the correct "Comm Type" with Money = 0, Pips =1, or Percent =2 depending on your brokers commission type.
Enter the correct "Base Comm". If your commission is charged in Money, then use a value of 0, If it is charged in Pips, then use a value of 1 here.
Enter the correct "Comm Lots" value. Commission per lot = 0 or Commission per deal = 1
Click the "Save" button once these changes have been made
-----

Regarding "Download your data using whatever you like" I did this. I used FXCM's Historical Data Downloader.
Then I edited the FXT files using TS to no avail as they get overwritten by the MT4 backtester.

You say "This scenario is not supported by our software and in fact should not require you to use Tickstory at all." which if confusing. How does one backtest in MT4 with commissions set and applied without some add-on tool? This feature is NOT present in MT4 installations from FXCM, Tallinex, Oanda and others I've tried. I've read countless posts searching various forums on the topic of "how to get commissions to apply in an MT4 backtest" and thought TS would provide a solution for my intended use and this issue just continues to get more and more confusing.

[tickstory]

Hi Billworld,

Thanks for a bit of context. I hope you can appreciate that without this sort of background information about what you're trying to achieve, we can get confused as well.
There is a glaring omission from step 3) in the list of instructions - that is, how exactly is the FXT file being produced? Tickstory is the only tool that produces an FXT file in a single step. Other methods require fiddling with scripts and these steps appear to have been omitted with perhaps the assumption by the author that you already know how to do this.

Here is some brief background of the standard MT4 behaviour (which is also offered in the Tickstory manual): MT4 uses a historical chart (HST file) to produce a data (FXT) file)for the purposes of back-testing. Depending on back-testing model you have chosen (Tick, Open prices or Control points), this FXT file will contain specific data points (or "ticks") which simulate the desired model (for example, the Control Point method will generate 4 ticks - one tick for open, high, low and close). When you press "Test" on the Strategy Tester, MT4 automatically creates the FXT file with all your broker's commission (etc) settings and the desired simulated tick data. In the case of Tickstory, it by-passes this FXT creation step and produces its own FXT file with highly accurate tick-by-tick data. During Tickstory's export process, you need to manually specify all your commission (etc) settings in Tickstory's Export to MT4->MT4 Settings tab.

With this knowledge in mind, you now have the following testing options:

1) Export tick data with Tickstory. In a single step, Tickstory downloads data from Dukascopy and produces an FXT file. There is no need for exporting to a CSV file first. The FXT file contains tick-by-tick data. This is the recommended and supported method. (The limitation at the moment is that we only support Dukascopy as a data source, however we are in the process of supporting other sources, including from your own CSV file).
2) Export 1-minute data from another source such as FXCM (etc) as a CSV file and then import it into Metatrader 4. This is done through MT4's Historical Data Center. MT4 supports the importing of CSV file and converts it to HST data (and the most granular HST data you can import is 1-minute). This method does not require Tickstory. When you press 'Test' on the Strategy Tester, the normal FXT generation process will occur.
3) Export tick data from another source as a CSV file and then produce an FXT tick data file via a script such as Birt's CSV2FXT. This is not a recommended method since it does require expert knowledge of what you are doing.

If you are opting for Step 3), then you will require further understanding of how to convert CSV to FXT (at least until we support this functionality). You can augment this chosen method with the use of Tickstory's FXT editor to "tweak" various settings such as Commissions. This should work but you will require some know-how around the process. You may appreciate that we do not support issues surrounding the use of any third-party software.

Hope this helps.

[billworld]

After a whole lot more testing and research I've discovered that non of my FXCM 1010 installs are showing commissions factored into the running trade-by-trade balance but my Tallinex MT4 install does (for the same EA and same testing period). I've tested connected both online and offline to my live FXCM commission based account as well as a newly created FXCM demo account (which I don't know if it is commission based or not). I always just tried the nifty SpreadOnChart EA from Birt (which shows commissions and spreads) for additional confirmation that indeed the FXCM installs aren't factoring in commission.

So, a long trail of discovery sifting through tons of information, much of it contradicting what I was actually experiencing.

Now I have to figure out why the FXCM 1010 installs, even a freshly installed one, are not showing the commissions.

Sorry for the trouble and I understand how complicated this all is with all the variables. However, my initial guidance (the Web article) as well as reading the TS docs made no distinction that I could see regarding how the FXT files are generated, only that the only way to edit them to get commissions to apply is to use TS, which, of course has not worked for me when using FXCM MT4 1010.