History issue

[imported_tickstoryParticipant January 2, 2014 at 3:53 am Post count: 1479]

#19542

Hi Render86,

It is more than likely you are coming across the FXT size limitation that MT4 has. See this post for details:

viewtopic.php?f=4&t=202

Regards.

[render86Participant January 2, 2014 at 3:57 am Post count: 51]

#19541

@tickstory wrote:

Hi Render86,

It is more than likely you are coming across the FXT size limitation that MT4 has. See this post for details:

viewtopic.php?f=4&t=202

Regards.

How was it then possible for me first to be able to run the backtests since 01.01.2008 until 07.06.2013 just fine yet be unable to run backtests since 01.02.2009 until 07.06.2013?

[imported_tickstoryParticipant January 2, 2014 at 4:45 am Post count: 1479]

#19543

Hi Render86,

Is your FXT file larger than the maximum size? If so, please try reducing the size of it and see if it eliminates the issue for you. Given that we are not privy to the internals of MT4, we can only suggest workarounds to any issues you are experiencing.

Thanks.

[render86Participant January 2, 2014 at 11:04 am Post count: 51]

#19544

@tickstory wrote:

Hi Render86,

Is your FXT file larger than the maximum size? If so, please try reducing the size of it and see if it eliminates the issue for you. Given that we are not privy to the internals of MT4, we can only suggest workarounds to any issues you are experiencing.

Thanks.

Could the FXT file size somehow change after I set a later (not earlier) starting date for the backtest without downloading the historical data again?

[imported_tickstoryParticipant January 2, 2014 at 11:03 pm Post count: 1479]

#19545

The Metatrader FXT file size remains the same once you have finished exporting the data from Tickstory. It could be that the behaviour you’re seeing is a result of MT4 trying to find the date you specify by locating an offset within the file and this number is sometimes too large to handle.

Hope this helps.

[render86Participant January 2, 2014 at 11:12 pm Post count: 51]

#19546

@tickstory wrote:

The Metatrader FXT file size remains the same once you have finished exporting the data from Tickstory. It could be that the behaviour you’re seeing is a result of MT4 trying to find the date you specify by locating an offset within the file and this number is sometimes too large to handle.

Hope this helps.

Should re-downloading the history with GMT offset 0 help?

[imported_tickstoryParticipant January 2, 2014 at 11:18 pm Post count: 1479]

#19547

I doubt it as this makes no difference to the size of the FXT file. You can reduce the size of the FXT file somewhat by checking the “Filter duplicate ticks” option in the MT4 Export window. Of course, you should only use this option if you know your EA wont be affected by the filtering of this data (eg. some scalper EAs could have their back-test results affected if they are dependent on price action/activity).

Regards.

[render86Participant January 2, 2014 at 11:27 pm Post count: 51]

#19548

@tickstory wrote:

I doubt it as this makes no difference to the size of the FXT file. You can reduce the size of the FXT file somewhat by checking the “Filter duplicate ticks” option in the MT4 Export window. Of course, you should only use this option if you know your EA wont be affected by the filtering of this data (eg. some scalper EAs could have their back-test results affected if they are dependent on price action/activity).

Regards.

I always have the “Filter duplicate ticks” checked. Have you tried to reproduce the issue I am experiencing?

[imported_tickstoryParticipant January 2, 2014 at 11:55 pm Post count: 1479]

#19549

As mentioned, we are not privy to the inner workings of MT4 nor can we help if there is an issue/limitation in MT4 itself. We could obviously assist further if there was a bug in the Tickstory software but this does not appear to be the case. Our best recommendation is to ensure you work within the MT4 limitations. The FXT file size limit is a well-documented problem which is discussed regularly on MT4 forums and if you need further information, other MT4 users may be able to shed more light on the problem you are having.

Hope this helps.

[render86Participant January 3, 2014 at 12:03 am Post count: 51]

#19540

@tickstory wrote:

As mentioned, we are not privy to the inner workings of MT4 nor can we help if there is an issue/limitation in MT4 itself. We could obviously assist further if there was a bug in the Tickstory software but this does not appear to be the case. Our best recommendation is to ensure you work within the MT4 limitations. The FXT file size limit is a well-documented problem which is discussed regularly on MT4 forums and if you need further information, other MT4 users may be able to shed more light on the problem you are having.

Hope this helps.

What is this specific MT4 limitation? I can contact MetaQuotes like I did on multiple occasions in the past and ask them to fix it. But I need a detailed description/justification that the problem is indeed on their end. Else, their underpaid customer service representative will take a similar convenience of putting all blame on Tickstory.

[imported_tickstoryParticipant January 3, 2014 at 12:49 am Post count: 1479]

#19550

I can imagine that you may well get this response. Metatrader 4 is not designed for the sort of high-end tick-level back-testing that Tickstory tries to facilitate and as a result, some compromises are necessary if you want to obtain tick-accurate back-tests. If you were to source your back-test data from the “official” data sources you will note that the most granular you will get is 1-minute data points – a comparitively small amount of data to the volume of tick data that Tickstory can export.

Given that MT4 is built on handling at-best 1-minute data, MT4 developers have assumed a maximum size for how large the historical data file can be – initially this was 2GB but may have been extended recently to 4GB. Of course, when you’re dealing with 1-minute quotes, it is a very reasonable assumption that your FXT file will be well within the limit – 7 years of 1-minute data, for example, would produce approximately a 200MB FXT file. This is obviously not the case for tick data which is much more voluminous.

Here are some further discussions on the topic that I believe have already been raised with Metaquotes:

http://www.mql5.com/en/forum/6687
http://forum.mql4.com/54343

Regards.

[render86Participant January 3, 2014 at 1:57 am Post count: 51]

#19551

@tickstory wrote:

I can imagine that you may well get this response. Metatrader 4 is not designed for the sort of high-end tick-level back-testing that Tickstory tries to facilitate and as a result, some compromises are necessary if you want to obtain tick-accurate back-tests. If you were to source your back-test data from the “official” data sources you will note that the most granular you will get is 1-minute data points – a comparitively small amount of data to the volume of tick data that Tickstory can export.

Given that MT4 is built on handling at-best 1-minute data, MT4 developers have assumed a maximum size for how large the historical data file can be – initially this was 2GB but may have been extended recently to 4GB. Of course, when you’re dealing with 1-minute quotes, it is a very reasonable assumption that your FXT file will be well within the limit – 7 years of 1-minute data, for example, would produce approximately a 200MB FXT file. This is obviously not the case for tick data which is much more voluminous.

Here are some further discussions on the topic that I believe have already been raised with Metaquotes:

http://www.mql5.com/en/forum/6687
http://forum.mql4.com/54343

Regards.

I’ll be honest with you: your standpoint about the FXT file size limit being exceeded just doesn’t make any sense to me. First, I can successfully run EURUSD backtests starting as early as 01.01.2008 until 07.06.2013. That would sure mean the FXT file size limit is not exceeded, wouldn’t it? But then all I do is simply change the backtest’s starting date from 01.01.2008 to 01.02.2009 in the MT4 strategy tester. I don’t download the history again or do anything else. Just change the starting date in the MT4 strategy tester. That’s it. Yet the backtest would not run since that date (01.02.2009). It instead runs since 23.07.2012. How could the FXT file size increase as a result of me merely changing the backtest’s starting date in the MT4 strategy tester?

[imported_tickstoryParticipant January 3, 2014 at 4:43 am Post count: 1479]

#19552

I’m not sure how we can make it any clearer that we cannot answer your questions as we have no scope to troubleshoot nor change MT4 behaviour – especially if you’re choosing to work beyond known limitations. This is not an issue of the FXT file size changing but the fact that MT4 cannot handle the large numbers it needs to. From a technical perspective, this can cause many different types of errors/anomalies in a program which requires the developer to look at the source-code to identify and fix the problem (and of course it isn’t really a “problem” from an MT4 perspective as it was never designed to handle this amount of information).

If you absolutely insist on working through the limitation rather than around it (such as breaking down your back-tests down into smaller periods), then the best bet is following up on the threads above to see if people have any further ideas of how to achieve what you need to do in MT4.

From a Tickstory perspective, the only way we can help is potentially allowing you to specify a custom filter that enables you to remove unneeded data, reducing the size of the FXT file. This feature is under development.

Regards.

[render86Participant January 10, 2014 at 7:20 am Post count: 51]

#19553

I’ve noticed when exactly the above issue happens. It occurs when I switch EAs in the MT4 strategy tester.

[Author]

Posts