How can we implement a filter to ENOURMOUS duka data spikes?

[Threshold]

Apparently the dukascopy data has massive erroneous data spikes they did not occur in actual reality, atleast not on any of the other exchanges.
This apparently is in the BI5 files downloaded from their servers.

If this is fact, then everyone has the same anomalies in their data.

Can we not target these rows to be edited during the export?
If this is indeed from the server, and everyone has the same data, then can an overwrite be implemented during or after the export to target and edit these exact dates?
Many of them occurred on April 1st (ironically), 2007.

We would have to write down the exact dates and times and edit them to appropriate price ranges and add that as an overwrite in the export process.

TLDR;
If everyone has the same data, then everyone has the same error on (for example) April 1, 2007. Filter this day. And do that for the other days that have errors. The export process would have to have some sort of reference to the file title (AUDUSD.csv for example) to open the proper filter.

(not a programmer)

My thoughts and suggestion.

[tickstory]

Thanks, Threshold. At this stage we're thinking the best way to add this feature is to allow a filter of "x % from current price" or something similar. That is, if the price deviates more than this percentage from the previous tick or bar, that data would be eliminated.

Any further thoughts are welcome.

[4EverMaAT]

Thanks, Threshold. At this stage we're thinking the best way to add this feature is to allow a filter of "x % from current price" or something similar. That is, if the price deviates more than this percentage from the previous tick or bar, that data would be eliminated.

Any further thoughts are welcome.

Birt's CSV2FXT script includes some of these filters. He released the source code for the script so that you can see exactly how the filters work.