I have a csv file (19MB) with 500 000 + rows. One example row looks like this:
6AH8,F,B,0, 60541765, 60541765,90.52, 1
I have already fixed so that the index 0 - 6AH8 and index 1 and 2 - F,B
gets to be minimized by exporting them to 2 different list(index0List and index12List) with index number from list as reference. i.e.
['6NH8', '6AH8', 'QMH8', 'QKH8']
['F,A', 'F,C','F,B', 'F,T']
--> 2, 3, 0, 60541765, 60541765, 90.52, 1
My problem is
index 4 - 60541765 #(milliseconds-after-midnight format)
index 5 - 60541765 #(milliseconds-after-midnight format)
index 6 - 90.52 #(price)
In this example the index 4 and 5 are the same, but not always.
What ever I do, it either takes to much time to iterate and still not compressed.
My question:: How do I minimize these indexes so that compress the file size? at this moment I tried to remove index 4,5,6 from the list and write it out to a csv file, and the output goes from 19MB to 5.3 MB, but i also write those indexes in a separate file which is 12.1MB.
NOTE: I'm not allowed to use any 3rd party or framework provided compression libraries for your solution. And only use libraries provided with the base compiler (e.g. no Boost)