Can Python handle a common TAQ dataset? « Economics Job Market Rumors

Can Python handle a common TAQ dataset?

Economist
32df

A TAQ dataset can be 50GB. Can Python do data manipulation, analysis, and modeling without making the computer crash? My laptop is Intel 4th, Core i5, RAM 8GB, 500GB HD

7 years ago # QUOTE 0 Volod 0 Vlad !
Economist
4a84

The real question is: can you handle it? You just need to construct your code around your memory limitations. Or get access to a cluster.

7 years ago # QUOTE 0 Volod 0 Vlad !
Economist
8757

Get thee to a cluster.

Analyzing TAQ on a desktop/laptop is a nightmare.

7 years ago # QUOTE 0 Volod 0 Vlad !
Economist
6e06

----> Software and Programming subforum

7 years ago # QUOTE 0 Volod 0 Vlad !
Economist
86f6

You are going to need an on-disk store with a front-end that provides an expressive DSL into that store.

For example, in R, dplyr is a unified interface into multiple database backends, data.frame and data.table.

pandas for Python also has multiple backends, but blaze is superior since it plugs into PyTables, MongoDB and a bunch of others.

All of this is naturally going to be slower than accessing the data store directly with its native query language.

7 years ago # QUOTE 0 Volod 0 Vlad !