# I want to take a CSV file, process the data, and generate a report file, where can I go to learn how to do this?

I have access to MATLAB for data processing, and TeXstudio for the report. I am familiar with MATLAB about 100x more than LaTeX. The data flow I am trying to create:

1. CSV file is generated with 50 columns of data, 1 timestamp column for everything, 1 column where timestamps for events are noted by the user since data is collected continuously (they push a button while data is being collected to create the event timestamp)

2. MATLAB looks in the event timestamps column and finds the data at those times by comparing it against the timestamp column to locate the segments of data for processing.

3. MATLAB finds the parameters of the data segments that the user requests (min, max, median, rise times, fall times, etc)

4. MATLAB stores all of this information in a file somewhere or builds a large table with the information. There may also be graphs created from those data segments in (2)

Here's where LaTeX comes in:

1. LaTeX will create the report for this data. With pre-entered statements like during Task X the maximum value reached was Y, where Y is taken from the MATLAB table that was created for that task in (3).

2. This is repeated for each task. Some data columns may be grouped, and there may be some plots, table and figures that would also need to be dispersed throughout the final report.

I can figure out most of the MATLAB portions myself, it is the combining of the two together and the LaTeX portion i want to learn. Where are the best references or examples to do what I am proposing?

The data should look something like this (sorry I cant seem to figure out how to format a table on this thing but with a zillion more rows and data lines:

Time        Data Line 1  Data Line 2  Data Line 3  Data Line 4  Data Line 5

7:49:23 AM  0.43493256   851.4415556  0.032704144  -1.24928     -0.016921

7:49:24 AM  0.52979029   851.4415556  0.032704144  -1.24928     -0.016921

• The documentation for datatool and csvsimple are good places to start. Also there are a lot of related questions on this site... – DG' Dec 9 '14 at 18:46
• Generating the report from the CSV file could require advanced LaTeX techniques. Consider an alternative approach: generate a LaTeX file using some programming language that you already know. This might be not as satisfying but could be a lot quicker. – Christian Lindig Dec 9 '14 at 18:54
• There's also the Report Generator to automatically create data reports. – andy mcevoy Dec 9 '14 at 19:03
• i dont have the report generator toolbox, so i was thinking latex might be a good free option. – LatexGuy Dec 9 '14 at 19:04
• Two general approaches come to mind. (1) Create a template of the .tex file, and plug in the values for each run, using Matlab or perhaps another language to connect/manage all the pieces. This approach is more oriented toward the Matlab/programming side. A makefile might be useful. (2) Use an approach that allows executable code to be embedded in the LaTeX document. My pythontex package allows this for Python, Octave, Ruby, and Julia. I've been planning to add Matlab support. Related paper. – G. Poore Dec 9 '14 at 20:08

Here's an example document using Python, with both code and output.

To use this example, you will need Python. Anaconda is a good option for this sort of thing. You also need the pythontex package. It's in TeX Live. It can also be installed manually; download the latest version from GitHub, extract, then run the Python installer in the pythontex directory.

To compile the document, you will need to use a three-step compile when Python code needs to be executed. Run LaTeX, then pythontex, then LaTeX again. For example, for a file analysis.tex, you might use

• pdflatex -interaction=nonstopmode analysis.tex
• pythontex analysis.tex
• pdflatex -interaction=nonstopmode analysis.tex

When pythontex is properly installed, you should have a wrapper/symlink that allows it to be run from the command line.

Data file data.csv:

Time,A,B,C,D,E
7:49:23 AM,0.43493256,851.4415556,0.032704144,-1.24928,-0.016921
7:49:24 AM,0.52979029,851.4415556,0.132704144,-3.07928,-0.016921
7:49:25 AM,0.34579029,851.4415556,0.173704144,-2.24258,-0.012351


TeX source:

\documentclass{article}

\usepackage{graphicx}
\usepackage{pythontex}

\begin{pycode}
# Import functions etc. that may be needed
from numpy import median, average, mean, std
import collections
from matplotlib import pyplot as plt

# Read in data and parse each column into a list of values
# Put the data in a dictionary, with keys corresponding to column labels
# This assumes simple, well-behaved CSV, with no quoting
with open('data.csv') as f:
# Add the data file as a dependency to be tracked
# This causes Python code to be re-executed when changes are detected
# Optional, but can be useful

# Need a way to convert times into a format that allows easy comparison
# Could also find a library to do this
def time_to_ISO8601_int(t):
t = t.strip()
hms, meridian = t.split(' ', 1)
h, m, s = map(int, hms.split(':'))
if 'AM' in meridian and h == 12:
h = 0
elif 'PM' in meridian and h != 12:
h += 12
return h*10000 + m*100 + s

# Store processed data in an ordered dictionary
# Ordered dictionaries store data in order of insertion
# So columns maintain their ordering
data = collections.OrderedDict()
# Create an entry in the data dictionary for each column
# Create an empty list for each column, which will be filled in later
# Python is zero-indexed
data[item.strip()] = []
# Process data into dictionary
# For very large data sets, a more efficient approach might be beneficial
for line in raw_data[1:]:
if line:
vals = list(line.split(','))
vals[0] = time_to_ISO8601_int(vals[0])
vals[1:] = map(float, vals[1:])
for n, data_list in enumerate(data.values()):
data_list.append(vals[n])
\end{pycode}

\begin{document}

Make a table of max values.

\begin{table}[h!]
\begin{center}
\begin{pycode}
print('\\begin{tabular}{|c|c|}')
print('\\hline')
print('Field & Max \\\\')
print('\\hline')
for key in data:
if key != 'Time':
print(key + '&' + str(max(data[key])) + '\\\\')
print('\\hline')
print('\\end{tabular}')
\end{pycode}
\end{center}
\caption{Maximum values}
\end{table}

Plot some data.

\begin{figure}[h!]
\begin{pycode}
plt.figure(figsize=(4,3))
plt.plot(data['Time'], data['A'], label='A')
plt.plot(data['Time'], data['C'], label='C')
plt.plot(data['Time'], data['E'], label='E')
# Prevent plotting from reformatting tick labels
plt.ticklabel_format(useOffset=False, axis='x')
plt.xticks(data['Time'])
plt.xlabel('Time (ISO 8601)')
plt.legend()
plt.savefig('fig.pdf', bbox_inches='tight')
\end{pycode}
\begin{center}
\includegraphics{fig}
\end{center}
\caption{Figure caption.}
\end{figure}

\end{document}


Output:

• So i run the code through anaconda and compile it, will it spit out a tex file for me to then run through texlive? I'm confused on the steps – LatexGuy Dec 11 '14 at 13:21
• yea i just isntalled anaconda and don't even know what it is i installed. some launcher and something about a cloud? I'm going to be frank with you and say I'm at the pictures of what to do and handholding stage of this still. I have the basic python 3.4 gui that I know how to use, can I use that instead? – LatexGuy Dec 11 '14 at 13:27
• @LatexGuy You can copy the code above into a data.csv file and a analysis.tex file. Put these in a directory together. Then, on the command line, when in the directory with these files, you can run the three commands I listed. In this approach, the .tex file itself contains the executable code. So a .tex file isn't generated. Rather, the pythontex package extracts executable code from the .tex file and the pythontex program takes care of the execution of that code. – G. Poore Dec 11 '14 at 14:53