Dear Tickstory Team,
I hope this message finds you well. We are currently developing a project that includes the storage of historical market data in a cloud-based storage. To successfully accomplish this aspect of our project, we require a clear understanding of the storage allocation we should account for.
We conducted a few tests using your complimentary application. We downloaded the full suite of your historical NASDAQ data, a month's worth of NASDAQ data, and a month's worth of EUR/USD data. We've observed some unexpected patterns in data size and would appreciate your insight into these matters.
The complete NASDAQ data amounted to approximately 35GB, while a single month accounted for 557MB. Calculating up to the complete data from the months worth of data it doesn't add up. The complete data seems too small or the months data too big. How does the entire historical data confine to just 35GB?
The month of EUR/USD data, meanwhile, amounted to a surprisingly compact 150MB. We speculate that this is due to the fewer symbols required for each row Could you confirm if our assumption is accurate?
As we plan to download the data of approximately 17 symbols, we are aware of the substantial storage commitment this will entail. However, before proceeding to purchase your tool and embark on downloading the requisite data, we seek your guidance in determining the volume of storage we should provide for. Additionally, any advice you could provide on the optimal methods for storing this data in a cloud solution would be greatly appreciated.
We understand that our request may require considerable effort on your part, and we appreciate your time and assistance in this matter. We are confident that this could mark the beginning of a long and mutually beneficial business relationship.
Thank you for your attention to this matter. We look forward to your response.
Best regards,
Chris
Guidance Requested on Data Storage Requirements and Best Practices for Cloud-Based Solutions
Re: Guidance Requested on Data Storage Requirements and Best Practices for Cloud-Based Solutions
Hi Chris,
Thanks for your query. The amount of data for each symbol is correlated with the amount of trading activity that is occurring on that symbol. In some cases, you will note that the data sizes are much smaller in the initial years, followed by greater sizes (activity) in the preceding years as more traders start to trade that symbol. More complexly, the compression of this data can be related to how the pricing unfolds - for example, if there are many duplicate prices, you would expect the data to have better compression. Of course, depending on which format you are using (CSV, MT4, SQL, etc), you can get vastly different sizes. It is therefore very much a matter of understanding these variables in order to plan the disk space you require.
The most simplest method would be to take a each symbol (eg. EURUSD) for a recent year (say 2021) and then import it into your data cloud. From there, you could roughly estimate the data space required across all of the years for that symbol.
Hope this answers your questions.
P.S. Please note that the support we provide for our licensed users revolves around the use of the application. If you are interested in a designing a specific solution, you are welcome to post any information in case other community members may be able to chime in. Please also feel free to contact us regarding any commercial licensing requirements you may have.
Thanks for your query. The amount of data for each symbol is correlated with the amount of trading activity that is occurring on that symbol. In some cases, you will note that the data sizes are much smaller in the initial years, followed by greater sizes (activity) in the preceding years as more traders start to trade that symbol. More complexly, the compression of this data can be related to how the pricing unfolds - for example, if there are many duplicate prices, you would expect the data to have better compression. Of course, depending on which format you are using (CSV, MT4, SQL, etc), you can get vastly different sizes. It is therefore very much a matter of understanding these variables in order to plan the disk space you require.
The most simplest method would be to take a each symbol (eg. EURUSD) for a recent year (say 2021) and then import it into your data cloud. From there, you could roughly estimate the data space required across all of the years for that symbol.
Hope this answers your questions.
P.S. Please note that the support we provide for our licensed users revolves around the use of the application. If you are interested in a designing a specific solution, you are welcome to post any information in case other community members may be able to chime in. Please also feel free to contact us regarding any commercial licensing requirements you may have.
Re: Guidance Requested on Data Storage Requirements and Best Practices for Cloud-Based Solutions
Hello Tickstory Team,
I appreciate your prompt and thorough response!
My apologies for the redundancy - I wasn't aware that my initial topic had already been posted here. It seems I missed the notification, even though I thought I'd set it up correctly. I've inadvertently posted another topic with the same question, so please disregard that.
Your explanation regarding the correlation between data size and trading activity for each symbol was enlightening. It's one of those things that seems obvious in hindsight - so thank you for pointing it out!
Considering your valuable insights, is there a possibility that you could possibly share more details about the data sizes on your servers? It would be immensely helpful in further understanding the storage requirements for our project.
Thank you again for your time and help.
Best regards,
Chris
I appreciate your prompt and thorough response!
My apologies for the redundancy - I wasn't aware that my initial topic had already been posted here. It seems I missed the notification, even though I thought I'd set it up correctly. I've inadvertently posted another topic with the same question, so please disregard that.
Your explanation regarding the correlation between data size and trading activity for each symbol was enlightening. It's one of those things that seems obvious in hindsight - so thank you for pointing it out!
Considering your valuable insights, is there a possibility that you could possibly share more details about the data sizes on your servers? It would be immensely helpful in further understanding the storage requirements for our project.
Thank you again for your time and help.
Best regards,
Chris
Re: Guidance Requested on Data Storage Requirements and Best Practices for Cloud-Based Solutions
Hi Chris,
As mentioned, the support you are requesting is outside the scope of what we provide for individual use of our software. If you have a suggestion for a commercial arrangement and/or if you are overseeing a project requiring 10+ licenses, you are most welcome to contact us directly to discuss further possibilities of getting the coverage you require.
Thanks.
As mentioned, the support you are requesting is outside the scope of what we provide for individual use of our software. If you have a suggestion for a commercial arrangement and/or if you are overseeing a project requiring 10+ licenses, you are most welcome to contact us directly to discuss further possibilities of getting the coverage you require.
Thanks.