Загрузка и анализ значительных объемов данных

Before you make a decision its probably worth understanding what is going on with your JVM as well as your physical system resources.

There are several factors that could be at play here:

• jvm heap size
• garbage collection algorithms
• how much physical memory you have
• how you load the data - is it from a file that is fragmented all over the disk?
• do you even need to load all of the data at once - can it be done it batches
• if you are doing it in batches you can vary the batch size and see what happens
• if your system has multiple cores perhaps you could look at using more than one thread at a time to process/load data
• if using multiple cores already and disk I/O is the bottleneck, perhaps you could try loading from different disks at the same time

You should also look at http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp if you aren't familiar with the settings for the VM.

Если ваши данные имеют реляционные свойства, нет ничего более естественного, чем хранить их в некоторой базе данных SQL. Там вы можете решить свою самую большую проблему - производительность, затратив «всего лишь» на написание соответствующего кода SQL.

Мне это кажется очень простым.

Я бы изучил анализ с помощью R. Это статистический язык с возможностями построения графиков. Это может помочь вам продвинуться вперед, особенно если вы собираетесь проводить именно такой анализ. Зачем писать весь этот код?

Я бы порекомендовал запустить профилировщик, чтобы увидеть, какая часть процесса загрузки занимает больше всего времени, и есть ли возможность быстрой оптимизации. Вы можете загрузить ознакомительную лицензию JProfiler или YourKit .

Ah, yes: large data structures in Java. Good luck with that, surviving "death by garbage collection" and all. What java seems to do best is wrapping a UI around some other processing engine, although it does free developers from most memory management tasks -- for a price. If it were me, I would most likely do the heavy crunching in Perl (having had to recode several chunks of a batch system in perl instead of java in a past job for performance reasons), then spit the results back to your existing graphing code.

However, given your suggested choices, you probably want to go with the SQL DB route. Just make sure that it really is faster for a few sample queries, watch the query-plan data and all that (assuming your system will log or interactively show such details)

Edit,(to Jim Ferrans) re: java big-N faster than perl (comment below): the benchmarks you referenced are primarily little "arithmetic" loops, rather than something that does a few hundred MB of IO and stores it in a Map / %hash / Dictionary / associative-array for later revisiting. Java I/O might have gotten better, but I suspect all the abstractness still makes it comparitively slow, and I know the GC is a killer. I haven't checked this lately, I don't process multi-GB data files on a daily basis at my current job, like I used to.

Feeding the trolls (12/21): I measured Perl to be faster than Java for doing a bunch of sequential string processing. In fact, depending on which machine I used, Perl was between 3 and 25 times faster than Java for this kind of work (batch + string). Of course, the particular thrash-test I put together did not involve any numeric work, which I suspect Java would have done a bit better, nor did it involve caching a lot of data in a Map/hash, which I suspect Perl would have done a bit better. Note that Java did much better at using large numbers of threads, though.