Best practice of (technically) preparing files (on Windows systems) for CTAN

Question

Similar questions were asked before but I did not found one for the recently discovered problem.

What I want to do is uploading a package to CTAN which is in good technical shape. I did this before using the instructions given by http://tug.org/texlive/pkgcontrib.html:

1. I created a flat directory containing all files.

2. I created a TDS directory structure holding the files at the intended position.

3. I zipped the TDS structures.

4. I zipped the flat directory and the .tds.zip to a package ZIP file.

5. I uploaded the package ZIP file to CTAN with accompanying information.

So far, so good. My intention was and is to cause as few as possible trouble to the CTAN team for installing my package.

Recently, I discovered that my naive idea that they just unzip my package was not true. They process my uploaded files and sometimes fix things by hand. That is not what I want.

Now to the question:

What are the (untold) technical details authors should respect to make the job of the CTAN team easier?

I know of a Perl script ctanify which probably makes all automatically on Linux systems (but not for Windows as the documentation states (?)). Aside from that, I would like to know (and remove) the technical reasons for problems.

Some things to correct seem to arise from preparing files on Windows systems - I put that in the title of the question.

What I know, is the following:

(A) Text files on Windows systems end lines with CRLF (hex: 0D0A) while Unix systems use LF (hex: 0A) only. I saw that CTAN filters uploaded text files such that CRLF is replaced by LF. This causes work and moreover problems, if it is not clear which file actually is a text file. This part of the answer I can give myself: replace CRLF by LF before zipping.

(B) Zipping is zipping (?) - or not? Are there good and bad ZIP files for CTAN? How can I produce a good ZIP file (on a Windows) system. I know that the answer will be tool-dependent but are there some general hints what to consider for CTAN?

Edit: for example, the Java based zipper of Apache Ant 1.9.2 produces bad ZIP files. From CTAN, I get the following report back:

,-----[ tds_fault_finder ]------------------------------------------------
|  WARNING: tcolorbox.tds.zip: Bad zipper?  No file is labelled as text by zipinfo
`-------------------------------------------------------------------------

Presumably, this is because some "text vs. binary" flags are not set by this zipper. According to CTAN, they do receive good ZIP files from Windows systems. So, this part of the question could be formulated as: What are good zipper tools (for Windows)?

Edit 2: According to the given answers and some further research, the brief answer seems to be: everything based on Info-ZIP is a good zipper for CTAN uploads. Windows binaries for Info-Zip are found here; see the answers for other good zippers.

(Anything else) What else does the CTAN team do to process uploaded files which can be avoided by the authors?

Answer 1

I use Info-ZIP as part of the Gnu on Windows bundle for my zip creation on Windows. (The version direct from Info-ZIP is also fine, but you need to install to the path manually.)

To do the building without mess-ups, I use a batch file (make.bat, so I just make ctan on Windows and on Unix). For example, the script I have for siunitx on Windows is:

@echo off

  if not "%1" == "" goto :init

:help

  echo.
  echo  make clean        - delete all generated files
  echo  make ctan         - create an archive ready for CTAN
  echo  make doc          - typesets documentation
  echo  make localinstall - extract packages
  echo  make tds          - create a TDS-ready archive
  echo  make unpack       - extract packages

  goto :EOF

:init

  setlocal

  rem The name of the package to create should be set here: here, the
  rem example package "testpkg" is in use

  set PACKAGE=siunitx

  rem It is possible to unpack dtx files without needing any extra files, but
  rem some people prefer a separate ins file (or there may be no unpacking
  rem to do). This should be set up here: for a self-extracting dtx the
  rem standard setting is fine.

  set UNPACK=%PACKAGE%.dtx

  rem A list of pdf files to be typeset and included in the archive files
  rem created. The files named here will be typeset (looking for source files
  rem in the order .dtx, .tex, .ltx).

  set INCLUDEPDF=%PACKAGE%

  rem Plain text files to be included in the archives: the .txt extension is
  rem automatically stripped when creating the archive.

  set INCLUDETXT=README

  rem Files to typeset

  rem The settings for cleaning up after compilation are divided into two
  rem parts. AUXFILES are deleted after each (La)TeX run, CLEAN only
  rem when the user calls "make clean"

  set AUXFILES=aux bbl blg cmds dvi glo gls hd idx ilg ind ist log los out tmp toc
  set CLEAN=bib bst cfg cls eps gz ins pdf sty tex txt zip

  rem Sets the order for searching for source files for pdfs

  set PDFSOURCES=dtx

  rem The file types for inclusion in the archive files: note that a CTAN
  rem archive should not contain "unpacked" files. Typeset files and their
  rem sources are not inlcuded here: they are dealt with separately

  set CTANFILES=dtx ins pdf
  set TDSFILES=%CTANFILES% cfg sty 

  rem Locations for building archives

  set CTANROOT=ctan
  set CTANDIR=%CTANROOT%\%PACKAGE%
  set TDSROOT=tds

  cd /d "%~dp0"

:main

  if /i "%1" == "clean"        goto :clean
  if /i "%1" == "ctan"         goto :ctan
  if /i "%1" == "doc"          goto :doc
  if /i "%1" == "help"         goto :help
  if /i "%1" == "localinstall" goto :localinstall
  if /i "%1" == "tds"          goto :tds
  if /i "%1" == "unpack"       goto :unpack

  goto :help

:clean

  echo.
  echo Deleting files

  for %%I in (%CLEAN%) do (
    if exist *.%%I del /q *.%%I
  )

  for %%I in (%TXT%) do (
    if exist %%I del /q %%I
  )

:clean-aux

  for %%I in (%AUXFILES%) do (
    if exist *.%%I del /q *.%%I
  )

  goto :end

:ctan

  call :zip
  if errorlevel 1 goto :EOF

  call :tds
  if errorlevel 1 goto :EOF

  for %%I in (%SOURCES%) do (
    xcopy /q /y %%I "%CTANDIR%\" > nul
  )  
  for %%I in (%CTANFILES%) do (
    xcopy /q /y *.%%I "%CTANDIR%\" > nul
  )
  for %%I in (%INCLUDETXT%) do (
    xcopy /q /y %%I.txt "%CTANDIR%\" > nul
    ren "%CTANDIR%\%%I.txt" %%I
  )

  xcopy /q /y %PACKAGE%.tds.zip "%CTANROOT%\" > nul

  pushd "%CTANROOT%"
  %ZIPEXE% %ZIPFLAG% %PACKAGE%.zip .
  popd
  copy /y "%CTANROOT%\%PACKAGE%.zip" > nul

  rmdir /s /q %CTANROOT%

  goto :end

:doc 

  call :unpack

  set SOURCES=

  for %%I in (%INCLUDEPDF%) do (
    for %%J in (%PDFSOURCES%) do (
      echo.
      if exist %%I.%%J call :typeset-%%J %%I.%%J
    )
  )

  goto :clean-aux

:file2tdsdir

  set TDSDIR=

  if /i "%~x1" == ".cfg" set TDSDIR=tex\latex\%PACKAGE%\config
  if /i "%~x1" == ".dtx" set TDSDIR=source\latex\%PACKAGE%
  if /i "%~x1" == ".ins" set TDSDIR=source\latex\%PACKAGE%
  if /i "%~x1" == ".pdf" set TDSDIR=doc\latex\%PACKAGE%
  if /i "%~x1" == ".sty" set TDSDIR=tex\latex\%PACKAGE%
  if /i "%~x1" == ".tex" set TDSDIR=doc\latex\%PACKAGE%  
  if /i "%~x1" == ".txt" set TDSDIR=doc\latex\%PACKAGE%

  goto :EOF

:localinstall

  call :unpack

  echo.
  echo Installing

  if not defined TEXMFHOME set TEXMFHOME=%USERPROFILE%\texmf

  for %%I in (%TDSFILES%) do (
    call :localinstall-int *.%%I
  )

  goto :end

:localinstall-int

  call :file2tdsdir %1

  if defined TDSDIR (
    xcopy /q /y %1 "%TEXMFHOME%\%TDSDIR%\" > nul
  ) else (
    echo Unknown file type "%~x1"
  )

  goto :EOF

:tds

  call :zip
  if errorlevel 1 goto :EOF

  call :doc
  if errorlevel 1 goto :EOF

  echo.
  echo Creating archive

  for %%I in (%SOURCES%) do (
    call :tds-int %%I
  )
  for %%I in (%TDSFILES%) do (
    call :tds-int *.%%I
  )
  for %%I in (%INCLUDETXT%) do (
    xcopy /q /y %%I.txt "%TDSROOT%\doc\latex\%PACKAGE%\" > nul
    ren "%TDSROOT%\doc\latex\%PACKAGE%\%%I.txt" %%I
  )

  pushd "%TDSROOT%"
  %ZIPEXE% %ZIPFLAG% %PACKAGE%.tds.zip .
  popd
  copy /y "%TDSROOT%\%PACKAGE%.tds.zip" > nul

  rmdir /s /q "%TDSROOT%"

  goto :end

:tds-int

  call :file2tdsdir %1

  if defined TDSDIR (
    xcopy /q /y %1 "%TDSROOT%\%TDSDIR%\" > nul
  ) else (
    echo Unknown file type "%~x1"
  )

  goto :EOF

:typeset-dtx

  echo Typesetting %1

  pdflatex -draftmode -interaction=nonstopmode "\AtBeginDocument{\DisableImplementation} \input %1" > nul
  if ERRORLEVEL 1 (
    echo ! Compilation failed
  ) else (
    makeindex -q -s gglo.ist -o %~n1.gls %~n1.glo > nul
    makeindex -q -s gind.ist -o %~n1.ind %~n1.idx > nul
    pdflatex -interaction=nonstopmode "\AtBeginDocument{\DisableImplementation} \input %1" > nul
    makeindex -q -s gind.ist -o %~n1.ind %~n1.idx > nul
    pdflatex -interaction=nonstopmode "\AtBeginDocument{\DisableImplementation} \input %1" > nul 
  )

  goto :EOF

:typeset-tex

  echo Typesetting %1

  set SOURCES=%SOURCES% %1

  pdflatex -interaction=nonstopmode -draftmode %1 > nul
  if ERRORLEVEL 1 (
    echo ! Compilation failed
  )
  pdflatex -interaction=nonstopmode %1 > nul
  pdflatex -interaction=nonstopmode %1 > nul

  goto :EOF

:unpack

  echo.
  echo Unpacking files

  for %%I in (%UNPACK%) do (
    tex %%I > nul
  )

  goto :end

:zip 

  if not defined ZIPFLAG set ZIPFLAG=-r -q -X -ll

  if defined ZIPEXE goto :EOF

  for %%I in (zip.exe "%~dp0zip.exe") do (
    if not defined ZIPEXE if exist %%I set ZIPEXE=%%I
  )

  for %%I in (zip.exe) do (
    if not defined ZIPEXE set ZIPEXE="%%~$PATH:I"
  )

  if not defined ZIPEXE (
    echo.
    echo This procedure requires a zip program,
    echo but one could not be found.
    echo
    echo If you do have a command-line zip program installed,
    echo set ZIPEXE to the full executable path and ZIPFLAG to the
    echo appropriate flag to create an archive.
    echo.
  )

  goto :EOF

:end

  shift
  if not "%1" == "" goto :main

This is set up to be 'portable' as far as possible, but the zip operation comes down to

zip -r -q -X -ll

on a suitably laid out folder. You'll notice that I'm using Info-ZIP to deal with the line endings, so I never alter things in the working files. To date, I've not hand issues with the CTAN team using the above set up (it's been refined over several years).

Answer 2

I once had the same problem with 7zip. I then decided to write my own bash script in the cygwin environment and use the provided zip! No problem since.

For Windows, my guess is to use one of the very old simple zip.exe, like described here. Maybe Gzip for Windows is a good test candidate.

For your other problems, I'm using something like

find ./$CTANDIR -name \* -exec dos2unix -k -q {} \;
find ./$CTANDIR -type d -exec chmod 755 '{}' \;
find ./$CTANDIR -type f -exec chmod 644 '{}' \;

to get files with LF line endings and sane permissions. Otherwise cgywin's zip uses 700 by default for Windows files.

EDIT:

Info-ZIP's zip version seems to be a good choice. My working cygwin work flow uses:

$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Cygwin. Original by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.