I have a series of csv files I'm expecting to receive on a regular basis, and there are two problems with these files.

  1. I am not guaranteed to have headers.
  2. I cannot rely on how many data entries there will be per row (sometimes there may be 50 or so, sometimes more)

What I am guaranteed is that, let's say, the 1st, 10th, and 30th column have the data I want. Up until now I've been using a perl script to collapse these csv files to something where the rows have only the elements I'm interested in.

Is there a way in datatool to tell it to create headers for certain columns and ignore the rest?

For example, it would be nice to be able to do something like the following:


which would mean that modelname is in the first column, serialnumber is in the 10th column and weight is in column 30.

Here is an extremely simplified example of a csv file I might work with.

  "toaster",    "",      "",       "",       "234-02015-23-948",  "",  "",      "",      "",      "",  "1255",  "",  "" 
  "blender",    "junk",  "",       "",       "753-20842-46-824",  "",  "",      "junk",  "",      "",  "1567",  "" 
  "microwave",  "",      "",       "stuff",  "376-73178-68-487",  "",  "",      "",      "junk",  "",  "2434",  "",  "",  ""
  "radio",      "",      "stuff",  "junk",   "724-86499-61-446",  "",  "",      "",      "junk",  "",  "2343" 
  "tv",         "",      "",       "",       "423-90219-60-149",  "",  "junk",  "",      "",      "",  "3023",  "",  "",  ""

A solution that creates a header for every column is completely infeasible because I have upwards of 50 columns only 3 or 4 of which I'm interested in.

What I'd like to be able to do is write:

  • 1
    When you say "not guaranteed to have headers" does that mean sometimes you do and sometimes you don't? If so, I assume you can tell by inspection if the first row is or isn't a header? – Alan Munn Jan 21 '13 at 16:33
  • If datatool doesn't turn out to be feasible pgfplotstable can be an alternative which can handle this. – percusse Jan 21 '13 at 16:35
  • @alanmunn. Sometimes there are are headers, but only certain columns have them and many columns have duplicate names suchs comment, but most columns lack a header even when others have them. – A.Ellett Jan 21 '13 at 16:39
  • So that means there is always a header row, even if every column doesn't make use of it. This is an important distinction. :) – Alan Munn Jan 21 '13 at 16:49
  • @alanmunn. But datatool in such examples, if I tell datatool there is a header, it complains about duplicate header names. – A.Ellett Jan 21 '13 at 16:51

Unless I'm misunderstanding something, this isn't too difficult to do, although perhaps not in exactly the way you envision. When datatool reads in a file, and you don't specify the headers, it will automatically assign headers of Column1 ... Column-n to your database. If you specify noheader then you don't need to worry about duplicate column names. You can use this to pick columns selectively from the database when you iterate through. In the example below, I'm assuming the file has a header row but it contains duplicates or blanks in some columns. If the file sometimes truly has no header, then the example would need to be modified to deal with that case as well.

\one & \six & \nine\\

output of code

  • Thank you very much. I can't find this approach in the manual. How did you know to do this? – A.Ellett Jan 21 '13 at 18:00
  • Section 5.2 (p. 39 or thereabouts, depending on your version) of the manual discusses this in the description of the "keys" option – Alan Munn Jan 21 '13 at 18:03
  • Ah ha! I'd seen that but I hadn't understood how to use it. Thanks all over again! – A.Ellett Jan 21 '13 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.