Large datasets on CTAN

Question

I've acquired a recent interest in GIS and do not know of much support in drawing maps/boundaries/locations within LaTeX. pst-geo provides some mapping features, specifically at the PostScript level. I'm interested in creating something more open/available in the form of boundary files that have easily accessible latitude/longitude coordinates for the boundaries/shapes, similar to what KML files provide. However, I can see how this could easily blow out of proportion when considering the entire globe (jurisdictions within jurisdictions within jurisdictions, ...)

As time goes by, more and higher detail data would probably become available, increasing the size of the data sets.

I'm looking for answers to the following:

• What is the best way to tackle this, specifically in terms of its location on CTAN?
• I think it would be unreasonable to require an installation of the entire data set on a user's computer. How would one require users to install large datasets in a piece-meal fashion?
• Would all of it be hosted on CTAN, or do I need to host the large "external" data sets on a server of my own?

Here are my thoughts on this:

• Create some base-level package, say gis-maps;

• Allow users to load modules, perhaps specific to a country using

  \usepackage[italy,south-africa,canada]{gis-maps}
  

  or

  \usepackage{gis-maps}
  \gissetup{maps={italy,south-africa,canada},...}
  

  that would load a load a list of helper-macros specific to those jurisdictions. For example, based on country codes, the above might create something like \drawITA, \drawZAF and \drawCAN (amongst a host of other macros, perhaps based on some geographical hierarchy).

• The above modules would also load the coordinates of the boundaries.
• The base package and modules might be big, but still manageable. However, the data sets themselves would be very large. So one would include instructions on how to add these to your distribution as a manual addition, perhaps to a location like texmf-local. I don't know how this would work...

_{Stay calm, I won't be including any treasure maps...}

Answer 1

I'm writing this answer as requested in the comments to the question, incorporating some of Bruno Le Floch's ideas. Since I know nothing about KML syntax, suggestions in this regard are very welcome!

Maintaining TeX-specific versions of the datasets is likely to produce a less than ideal solution. First, it is inefficient since it will need to duplicate maintenance work already done elsewhere. Second, I don't think the TeX community is likely to do a good job maintaining huge datasets over time, especially ones which are not inherently tied up with the system.

So it would be better to think of the problem as requiring two things:

1. identification or creation of suitable datasets designed for general use;
2. design and maintenance of a way for TeX to interact with these datasets, perhaps using scripts.

So the idea, as elaborated by Bruno Le Floch would be for users to download multi-purpose KML files as required. A script could be provided to download these files and to extract a useful subset of the information in them into a format which TeX could then use directly in typesetting. This need not itself be written in TeX code.

One option would be to use something like perl which should make the script usable on the platforms supported by TeX Live, for example, since TL itself depends on scripts written in perl. (In the case of Windows, TL provides perl itself; OS X, GNU/Linux etc. already have perl available.) perl is used for the getnonfreefonts script mentioned by egreg.

A package would then be provided to interact with the extracted subset of information, offering user-friendly macros to utilise this information in documents. Since the extracted subset would be smaller than the original KML dataset, this would be faster to parse, speeding up typesetting. Since the extraction would be scripted, it would be easy to update by re-downloading and re-extracting information from the original source of the datasets. In cases where currency is really critical, the updating could be automatically managed by having TeX run the download and extraction script during typesetting. But I assume this would not be very useful in the majority of cases.

This would solve several problems:

1. The problem of storing huge datasets would evaporate since the KML files would be stored wherever they are stored anyway.
2. It would avoid the issue of duplicating work within the TeX community which is better done by (probably larger and better equipped) communities elsewhere.
3. Moreover, Bruno Le Floch also pointed out that it would allow users to convert and use their own private KML files.
4. Indeed, it would allow the use of KML files from any source, and would easily generalise should files with similar syntax be used in other contexts. (I don't know anything about KML so this is a purely theoretical/hypothetical point!)