Having a problem parsing a huge csv file,
Other than using the csv module are there any other tips or tutorials, example .toe files with code examples for parsing .csv?
Is there any advantage in trying to convert to .json?
How big is big?
Are you working with string data or is it integers and floats?
With large CSVs I’m mostly found that the hard part is viewing the data in a DAT. If you can work with viewers turned off on the largest views of your CSV it can be easier to work with them.
If you’re working with integers and floats you might instead consider opening those with a file in CHOP. If you’re going to convert to CHOP data anyway it’s sometimes helpful to skip a step there - especially as reordering samples and deleting samples is faster / easier in CHOP land (IMO).
JSON you can put into storage - can be easier to parse, but you’ve got to do all of that with some python instead of with nodes. In that respect I think it comes partially down to your preferred working style.
Thanks Matthew. Mostly integers and floats but with strings for first 3 columns and some column heads.
When I view it here https://www.convertcsv.com/csv-viewer-editor.htm
I can see 50+ columns of data but I only want the 2,3rd and 4th columns as well as the last column whose heading is today’s date, ie 4/7/20.
Not sure how to strip out all the unnecessary data. JSON data is easier to deal with.