4

I recently started using the monokai theme with minted which I use because it works well with XeLaTeX. I found that when using unicode characters in code they get highlighted is red. This behaviour is not to be seen in the default theme of minted for Python so I'm guessing that it has something to do with the monokai theme. Any ideas on how to remove the ugly highlighting?

Here's what the problem looks like (the unicode parenthesis are highlighted red):

Illustration of the highlighting problem

Edit Here's a minimal working example :

\documentclass[12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[frenchb]{babel}
\usepackage{amsmath}
\usepackage{multicol}
\usepackage{color}
\usepackage{tkz-tab}


\usepackage{minted}
\usepackage{fontspec}
\setmonofont{DejaVu Sans Mono}

\usemintedstyle{monokai}
\definecolor{codeBg}{HTML}{282822}
\definecolor{textBg}{HTML}{1c1c17}

\setminted[python]{
    breaklines=true,
    encoding=utf8,
    fontsize=\footnotesize,
    bgcolor=codeBg
}

\begin{document}

\pagecolor{textBg}
\color{white}

\begin{multicols*}{2}

Commençons par un exemple : vérifions que la matrice
\[A = \begin{pmatrix}3&9&-9\\2&0&0\\3&3&3\end{pmatrix}\] 
est nilpotente d'indice 3.
\begin{minted}{python}
>>> from matrix import *
>>> A = Matrix([
... [3, 9,-9],
... [2, 0, 0],
... [3, 3,-3]
... ])
>>> A**2

⎛0 0   0 ⎞
⎜6 18 -18⎟
⎝6 18 -18⎠

>>> A**3

⎛0 0 0⎞
⎜0 0 0⎟
⎝0 0 0⎠

\end{minted}

\end{multicols*}
\end{document}
  • have you checked your editor's setting on how parentheses are highlighted? – naphaneal Mar 24 '18 at 13:54
  • This is a screenshot of the pdf file. It has nothing to do with the way my editor is set up. But I look into the possibility that my pdf viewer does that. – Pharoah Jardin Mar 24 '18 at 16:40
  • Could you provide a minimal example that shows the behaviour? Maybe it is because of your PDF viewer, maybe there is a bug. It would be easier to track down, if we had a minimal working example. – Stephan Lukasczyk Mar 24 '18 at 16:41
  • I don't see a matrix module at pypi.python.org/pypi. Hence it is very time costly to try to hep you because you did not paste simply your interactive session as characters but an image. – user4686 Mar 24 '18 at 17:47
  • @jfbu I didn't paste the Python session because I didn't think you would want to copy and paste it. The matrix module is just a custom built module, it can't be found on the internet. – Pharoah Jardin Mar 24 '18 at 18:27
5

As was pointed out in comments the Pygmentize lexers may be (depending on programming language involved) very picky about Unicode characters and may flag them as error tokens. This is inherited by the monokai style.

The real fix is to fix the lexer. Oops, I now understand correct fix to your problem. For sentimental reason, I keep my initial one. But skip to bottom for real fix.

edit

  • I had not observed OP's code loaded all of inputenc, fontenc, and fontspec with \setmainfont. I have removed inputenc and fontenc loading in an edit.

  • In my sentimentally kept approach the replacement text of \PYGmonokai@tok@err did not use correctly prefixed macro. I have fixed it too. But I don't recall now when is the @bc activated. (one needs to examine Pygmentize latex templates for that). Could be that bc is for background color` but right now I don't recall.


I did not immediately fing in minted suitable hooks from the documentation, so I decided to patch the fancyvrb Verbatim environment which is used by minted. I also have a bit lost out of memory the fancyvrb hooks, so I did it brutally via etoolbox.

\documentclass[12pt]{article}

%%%% \usepackage[utf8]{inputenc}% NO! fontspec loaded below
%%%% \usepackage[T1]{fontenc}
\usepackage[frenchb]{babel}
\usepackage{amsmath}
\usepackage{multicol}
\usepackage{color}
\usepackage{tkz-tab}


\usepackage{minted}
\usepackage{fontspec}
\setmonofont{DejaVu Sans Mono}

\usemintedstyle{monokai}
\definecolor{codeBg}{HTML}{282822}
\definecolor{textBg}{HTML}{1c1c17}

\setminted[python]{
    breaklines=true,
    encoding=utf8,
    fontsize=\footnotesize,
    bgcolor=codeBg
}


\makeatletter
\usepackage{etoolbox}
\AtBeginEnvironment{Verbatim}{%
%\show\PYGmonokai@tok@err
   \def\PYGmonokai@tok@err {\def\PYGmonokai@bc##1{\strut ##1}}%
%\show\PYGmonokai@tok@err
}%
\makeatother
\begin{document}

\pagecolor{textBg}
\color{white}

\begin{multicols*}{2}

Commençons par un exemple : vérifions que la matrice
\[A = \begin{pmatrix}3&9&-9\\2&0&0\\3&3&3\end{pmatrix}\] 
est nilpotente d'indice 3.

\begin{minted}{python}
>>> from matrix import *
>>> A = Matrix([
... [3, 9,-9],
... [2, 0, 0],
... [3, 3,-3]
... ])
>>> A**2

⎛0 0   0 ⎞
⎜6 18 -18⎟
⎝6 18 -18⎠

>>> A**3

⎛0 0 0⎞
⎜0 0 0⎟
⎝0 0 0⎠

\end{minted}

\end{multicols*}
\end{document}

enter image description here

Notice in this context that the Pygmentize latex stylesheets have some bugs, which are inherited by minted. Here are some links to the Pygments issue tracker. (it does not look like those issues will get fixed any time soon)


Coming back to real problem: you are using the python lexer for an interactive session. You should have used the pycon lexer.

$ pygmentize -L lexers | grep -C 1 py
    ANTLR With Perl Target (filenames *.G, *.g)
* antlr-python:
    ANTLR With Python Target (filenames *.G, *.g)
--
    Bro (filenames *.bro)
* bst, bst-pybtex:
    BST (filenames *.bst)
--
* cobol:
    COBOL (filenames *.cob, *.COB, *.cpy, *.CPY)
* cobolfree:
--
    Cypher (filenames *.cyp, *.cypher)
* cython, pyx, pyrex:
    Cython (filenames *.pyx, *.pxd, *.pxi)
* d-objdump:
--
    Ioke (filenames *.ik)
* ipython2, ipython:
    IPython 
* ipython3:
    IPython3 
* ipythonconsole:
    IPython console session 
--
    NSIS (filenames *.nsi, *.nsh)
* numpy:
    NumPy 
--
    Puppet (filenames *.pp)
* py3tb:
    Python 3.0 Traceback (filenames *.py3tb)
* pycon:
    Python console session 
* pypylog, pypy:
    PyPy Log (filenames *.pypylog)
* pytb:
    Python Traceback (filenames *.pytb)
* python, py, sage:
    Python (filenames *.py, *.pyw, *.sc, SConstruct, SConscript, *.tac, *.sage)
* python3, py3:
    Python 3 

Thus here is with :

\begin{minted}{pycon}
>>> from matrix import *
>>> A = Matrix([
... [3, 9,-9],
... [2, 0, 0],
... [3, 3,-3]
... ])
>>> A**2

⎛0 0   0 ⎞
⎜6 18 -18⎟
⎝6 18 -18⎠

>>> A**3

⎛0 0 0⎞
⎜0 0 0⎟
⎝0 0 0⎠

\end{minted}

enter image description here

  • IMO it may be better to put the “correct” solution (use pycon, not python, because those characters are indeed errors in Python code) at the top of the answer, and the rest of it later (if still relevant). – ShreevatsaR Mar 25 '18 at 8:06
  • @ShreevatsaR no, leaving as is. That minted provides no easily found in its doc means for patches such as those is a problem, because it makes it hard to workaround bugs of Pygments LaTeX styles. – user4686 Mar 25 '18 at 8:13
  • OK sure, but in this case there is no bug (just user error), I think? Or is there? – ShreevatsaR Mar 25 '18 at 8:17
  • @ShreevatsaR no you are right, here it is only user error. But the Pygments highlighting of error tokens is buggy, and the fact that some Pygments lexers (the one for Coq language for example) emit error tokens when encountering Unicode chars despite the language syntax allowing them is another limitation. In my brief investigation I saw no quick minted way to workaround those things, this is why I am leaving my answer as is in case it can be recycled. – user4686 Mar 25 '18 at 9:24
  • Waw. Very helpful. pycon seems to be the right way to solve the problem (even if the colored matrices are nice as well). Thanks for your time ! – Pharoah Jardin Mar 25 '18 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.