Some preliminary remarks about how (La)TeX works:
The command \the⟨counter⟩
may not only deliver digit-tokens in range 0..9 which denote the value of the counter. It may also deliver (non-expandable) tokens for pretty-printing the value which disturb calculation and \setcounter
-assignments.
Instead use \number\value{⟨counter⟩}
.
Desite expl3's misleading terminology where the word "function" is used a lot, the programming-paradigma of LaTeX is not a procedural/functional one as is the case with high level programming languages like C++ or Java, but is rather a macro-based declarative and symbolic one, where the symbols are formed by so-called tokens and where symbols/tokens get replaced by other symbols/tokens during the stage of expansion.
In Knuth's analogy to a digestive process TeX has
- eyes
- a digestive tract
- the ability to produce tokens and put them into its mouth where the digestion process begins.
TeX's eyes read the .tex-input-file. Hereby TeX takes the input for a set of instructions for producing tokens and putting these tokens into its mouth one by one. So the tokens form a "token-stream" whose elements one by one are lead through TeX's digestive tract.
Expansion of a(n expandable) token—i.e., replacement of that token (and probably those tokens that form its arguments) by other tokens—takes place while tokens are transported through TeX's gullet.
Assignments (defining macros, assigning values to \count
-registers and the like) in that analogy take place in TeX's stomach.
The final result of TeX's digestion-process will be the resulting output-files (.pdf-file/.dvi-file, .log-file, auxiliary files like the .aux-files and .toc-file and .lot/.lof-file etc) and things that are written to the console.
A crucial point hereby is:
TeX-⟨number⟩-quantities must be formed in TeX's gullet, during the stage of expansion, so that the token-sequence which forms the TeX-⟨number⟩-quantity is available in TeX's stomach where assignments etc are performed.
\setcounter
implies assigning (in LaTeX's stomach) to the \count
-register that underlies the LaTeX-counter in question a value which is delivered through LaTeX's gullet (where expansion takes place) as a token-sequence which forms a TeX-⟨number⟩-quantity.
So the result of expanding a macro is not just the final value. The result is a set of tokens/symbols that get inserted into the token-stream.
E.g., with
\newcommand{\reste}[2]{%
\setcounter{a}{#1}%
\setcounter{b}{#2}%
\whiledo{\(\thea>\theb\)\OR\(\thea=\theb\)}{%
\setcounter{a}{\thea-\theb}%
}%
\thea
}
the result of expanding \reste{29}{6}
will not just be the final thingie that comes from \thea
.
The result will be removing the token-sequence
\restecontrol-word-token
,
{explicit catcode-1(begin-group)-character-token
,
2explicit catcode-12(other)-character-token
,
9explicit catcode-12(other)-character-token
,
}explicit catcode-2(end-group)-character-token
,
{explicit catcode-1(begin-group)-character-token
,
6explicit catcode-12(other)-character-token
,
}explicit catcode-2(end-group)-character-token
from the token-stream and instead of that token-sequence inserting the token-sequence
\setcountercontrol-word-token
,
{explicit catcode-1(begin-group)-character-token
,
aexplicit catcode-11(letter)-character-token
,
}explicit catcode-2(end-group)-character-token
,
{explicit catcode-1(begin-group)-character-token
,
2explicit catcode-12(other)-character-token
,
9explicit catcode-12(other)-character-token
,
}explicit catcode-2(end-group)-character-token
,
\setcountercontrol-word-token
,
{explicit catcode-1(begin-group)-character-token
,
bexplicit catcode-11(letter)-character-token
,
}explicit catcode-2(end-group)-character-token
,
{explicit catcode-1(begin-group)-character-token
,
6explicit catcode-12(other)-character-token
,
}explicit catcode-2(end-group)-character-token
,
\whiledocontrol-word-token
,
{explicit catcode-1(begin-group)-character-token
,
\(control-symbol-token
,
\theacontrol-word-token
,
>explicit catcode-12(other)-character-token
,
\thebcontrol-word-token
,
\)control-symbol-token
,
\ORcontrol-word-token
,
\(control-symbol-token
,
\theacontrol-word-token
,
=explicit catcode-12(other)-character-token
,
\thebcontrol-word-token
,
\)control-symbol-token
,
}explicit catcode-2(end-group)-character-token
,
{explicit catcode-1(begin-group)-character-token
,
\setcountercontrol-word-token
,
{explicit catcode-1(begin-group)-character-token
,
aexplicit catcode-11(letter)-character-token
,
}explicit catcode-2(end-group)-character-token
,
{explicit catcode-1(begin-group)-character-token
,
\theacontrol-word-token
,
-explicit catcode-12(other)-character-token
,
\thebcontrol-word-token
,
}explicit catcode-2(end-group)-character-token
,
}explicit catcode-2(end-group)-character-token
,
\theacontrol-word-token
into the token-stream.
That removal and insertion takes place in LaTeX's gullet.
Armed with this knowledge, it becomes obvious that
\setcounter{t}{\reste{\thep}{\theq}}
cannot work:
The second argument of a \setcounter
directive is expected to be a set of tokens whose elements after expansion only yield components of a TeX-⟨number⟩-quantity.
But the result of expanding \reste{\thep}{\theq}
, which is inside the second argument of the \setcounter{t}{...}
-directive, does not only yield components of a TeX-⟨number⟩-quantity.
That result also contains tokens unrelated to denoting components of TeX-⟨number⟩-quantities.
E.g., the (nested) directives \setcounter{a}{\thep}
and \setcounter{b}{\theq}
.
E.g., the \whiledo
-directive.
The (nested) \setcounter
-directives (beneath other things) at the stage of expansion in LaTeX's gullet yield tokens that don't denote digits/components of a TeX-⟨number⟩-quantity but denote the action of assigning a value to a LaTeX-counter/of assigning a value to the TeX-\count
-register that underlies the LaTeX-counter in question.
These assignment-denoting-tokens that come from the nested \setcounter
-directives disturb the surrounding \setcounter{t}{...}
-directive when that directive's underlying \count
-register-assignment gets carried out in LaTeX's stomach.
With the \whiledo
-directive the result is not delivered by doing expansion (in LaTeX's gullet) only. \whiledo
also (in LaTeX's stomach) performs temporary assignments. The tokens underlying these temporary assignments are not tokens that yield components of a TeX-⟨number⟩-quantity. Therefore they also "disturb" the \setcounter{t}{...}
-assignment.
(Within the "nested" \setcounter
-directives \setcounter{a}{\thep}
and \setcounter{b}{\theq}
themselves there might be basically the same problem:
Unlike with \number\value{⟨counter⟩}
it is not guaranteed that \the⟨counter⟩
delivers a set of tokens whose elements only yield components of a TeX-⟨number⟩-quantity. With \the⟨counter⟩
there might be things/tokens for "pretty-printing" which cannot be components of a TeX-⟨number⟩-quantity and which therefore "disturb" the \setcounter
-assignment.)
How to resolve the issue?
With \reste
you can, e.g., have an optional argument which denotes the tokens that are to be inserted behind these tokens that yield assigning the desired value to the LaTeX-counter a
.
Modifying as few things of your code as possible, you can probably get the desired results as follows:
\documentclass[a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage{calc}
\usepackage{ifthen}
\usepackage[french]{babel}
\newcounter{a}
\newcounter{b}
\newcommand{\reste}[3][\thea]{%
\setcounter{a}{#2}%
\setcounter{b}{#3}%
\whiledo{\(\number\value{a}>\number\value{b}\)\OR\(\number\value{a}=\number\value{b}\)}%
{\setcounter{a}{\number\value{a}-\number\value{b}}}%
#1%
}
\newcounter{p}
\newcounter{q}
\newcounter{t}
\newcommand{\pgcd}[3][\thep]{%
\setcounter{p}{#2}%
\setcounter{q}{#3}%
\whiledo{\number\value{q}>0}{%
\reste[{\setcounter{t}{\number\value{a}}}]{\number\value{p}}{\number\value{q}}%
\setcounter{p}{\number\value{q}}%
\setcounter{q}{\number\value{t}}%
}%
#1%
}
\begin{document}
Le reste dans la division entière de $29$ par $6$ est $\reste{29}{6}$.
Le reste dans la division entière de $30$ par $6$ est $\reste{30}{6}$.
Le plus grand commun diviseur de $29$ et $6$ est $\pgcd{29}{6}$.
\end{document}

If \numexpr
of the ε-TeX-extensions is available, you can get the result without using LaTeX-counters at all.
Instead of LaTeX-counters use macro-arguments for holding values during the expansion-cascade:
\documentclass[a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[french]{babel}
\usepackage{amsmath}
\makeatletter
\newcommand\PassFirstToSecond[2]{#2{#1}}%
\newcommand{\reste}[3]{%
% #1 = marker what the remainder shall look like;
% Assume: (integer divisor) = (integer dividend)*(integer number K) + (integer remainder)
% p/P = remainder positive
% s/S = remainder with same sign as dividend
% v/V or anything else = remainder with smallest absolute value
% #2 = divisor
% #3 = dividend
\romannumeral0%
\ifnum#3=0 %
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
{ \text{non-défini}}%
{%
\PassFirstToSecond{#1}{%
\ifnum#3<0 \expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{\PassFirstToSecond{-}}%
{\PassFirstToSecond{}}%
{%
\ifnum#2<0 \expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{\PassFirstToSecond{-}}%
{\PassFirstToSecond{}}%
{%
\expandafter\PassFirstToSecond\expandafter{\number\ifnum#3<0 -\fi\number#3}{%
\expandafter\PassFirstToSecond\expandafter{\number\ifnum#2<0 -\fi\number#2}{\resteloop}%
}%
}%
}%
}%
}%
}%
\@ifdefinable\gobbletoexclam{\long\def\gobbletoexclam#1!{}}%
\@ifdefinable\forkpsv{\long\def\forkpsv#1!p!s!v!P!S!V!#2#3!!!!{#2}}%
\newcommand\psvfork[4]{%
% #1 = argument to check
% #2 = tokens in case argument to check is p or P
% #3 = tokens in case argument to check is s or S
% #4 = tokens in case argument to check is something else.
\if\relax\detokenize\expandafter{\gobbletoexclam#1!}\relax
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
{%
\forkpsv!#1!s!v!P!S!V!{#2}% <- case p
!p!#1!v!P!S!V!{#3}% <- case s
!p!s!#1!P!S!V!{#4}% <- case v
!p!s!v!#1!S!V!{#2}% <- case P
!p!s!v!P!#1!V!{#3}% <- case S
!p!s!v!P!S!#1!{#4}% <- case V
!p!s!v!P!S!V!{#4}% <- case something else without !
!!!!%
}%
{#4}% <- case something else with !
% The fork internally has eight branches:
% 1. p
% 2. s
% 3. v
% 4. P
% 5. S
% 6. V
% 7. something else without "!"
% 8. something else with "!"
% Branches 1 and 4 collapse into one branch.
% Branches 2 and 5 collapse into one branch.
% Branches 3, 6, 7 and 8 collapse into one branch.
% Thus you could instead implement a fork with six branches:
% 1. p
% 2. s
% 3. P
% 4. S
% 5. something else without "!"
% 6. something else with "!"
% and collapse branches 1 and 3 into one branch
% and branches 2 and 4 into one branch
% and branches 5 and 6 into one branch.
% But this way introducing different behavior for case v or V
% than for case something else with or without "!" is more
% easy if needed. (You may wish the triggering of error-
% messages in the latter case rather than the same behavior as in
% case v/V.)
}%
\newcommand{\resteloop}[5]{%
% #1 = absolute value of divisor (digit sequence)
% #2 = absolute value of dividend (digit sequence)
% #3 = sign of divisor (empty = not negative; - = negative)
% #4 = sign of dividend (empty = not negative; - = negative)
% #5 = marker what the remainder shall look like;
% Assume: (integer divisor) = (integer dividend)*(integer number K) + (integer remainder)
% p/P = remainder positive
% s/S = remainder with same sign as dividend
% v/V or anything else = remainder with smallest absolute value
\ifnum#1<#2 \expandafter\@secondoftwo\else\expandafter\@firstoftwo\fi
{\expandafter\resteloop\expandafter{\number\numexpr#1-#2\relax}{#2}{#3}{#4}{#5}}%
{%
\psvfork{#5}{%
%Case p/P:
\ifx\relax#3\relax\expandafter\@secondoftwo\else\expandafter\@firstoftwo\fi
{%
\@firstofone{\expandafter} \number\numexpr\number#3#1+#2\relax
}%
{%
\@firstofone{\expandafter} \number#3#1%
}%
}{%
%Case s/S:
\ifx\relax#3\relax\expandafter\@secondoftwo\else\expandafter\@firstoftwo\fi
{%
\ifx\relax#4\relax\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{%remainder negative module positive
\@firstofone{\expandafter} \number\numexpr\ifnum#1=0 \else#2+\fi\number#3#1\relax
}{% remainder negative module negative
\@firstofone{\expandafter} \number#3#1%
}%
}{%
\ifx\relax#4\relax\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{% remainder positive module positive
\@firstofone{\expandafter} \number#3#1%
}{% remainder positive module negative
\@firstofone{\expandafter} \number\numexpr\ifnum#1=0 \else-#2+\fi\number#3#1\relax
}%
}%
}{%
%Case someting else, e.g. v/V:
\ifnum\number\numexpr#2-#1\relax<#1 %
\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{%
\@firstofone{\expandafter} \number\numexpr\ifnum\number#3#1<0 \else-\fi#2+\number#3#1\relax
}{%
\@firstofone{\expandafter} \number#3#1%
}%
}%
}%
}%
\newcommand{\pgcd}[2]{%
% #1 = first integer number
% #2 = second integer number
\romannumeral0%
\ifnum#1=0 \expandafter\@firstofone\else\expandafter\@secondoftwo\fi
{%
\ifnum#2=0 \expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{ \text{non-défini}}%
}%
{%
\expandafter\PassFirstToSecond\expandafter{%
\number\ifnum#2<0 -\fi\number#2%
}{%
\expandafter\PassFirstToSecond\expandafter{%
\number\ifnum#1<0 -\fi\number#1%
}{\pgcdloop}%
}%
}%
}%
\newcommand{\pgcdloop}[2]{%
\ifnum#2>0 \expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
{%
\expandafter\PassFirstToSecond\expandafter{%
\romannumeral0\resteloop{#1}{#2}{}{}{p}%
}{\pgcdloop{#2}}%
}%
{ #1}%
}
\makeatother
\begin{document}
\footnotesize\parindent=0ex
Le reste avec la valeur absolue la plus petite dans la division entière de $0$ par $6$ est $\reste{v}{0}{6}$.\\
Le reste avec le meme signe que le module dans la division entière de $0$ par $6$ est $\reste{s}{0}{6}$.\\
Le reste avec valeur positive dans la division entière de $0$ par $6$ est $\reste{p}{0}{6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $6$ par $0$ est $\reste{v}{6}{0}$.\\
Le reste avec le meme signe que le module dans la division entière de $6$ par $0$ est $\reste{s}{6}{0}$.\\
Le reste avec valeur positive dans la division entière de $6$ par $0$ est $\reste{p}{6}{0}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $29$ par $6$ est $\reste{v}{29}{6}$.\\
Le reste avec le meme signe que le module dans la division entière de $29$ par $6$ est $\reste{s}{29}{6}$.\\
Le reste avec valeur positive dans la division entière de $29$ par $6$ est $\reste{p}{29}{6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $29$ par $-6$ est $\reste{v}{29}{-6}$.\\
Le reste avec le meme signe que le module dans la division entière de $29$ par $-6$ est $\reste{s}{29}{-6}$.\\
Le reste avec valeur positive dans la division entière de $29$ par $-6$ est $\reste{p}{29}{-6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $-29$ par $6$ est $\reste{v}{-29}{6}$.\\
Le reste avec le meme signe que le module dans la division entière de $-29$ par $6$ est $\reste{s}{-29}{6}$.\\
Le reste avec valeur positive dans la division entière de $-29$ par $6$ est $\reste{p}{-29}{6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $-29$ par $-6$ est $\reste{v}{-29}{-6}$.\\
Le reste avec le meme signe que le module dans la division entière de $-29$ par $-6$ est $\reste{s}{-29}{-6}$.\\
Le reste avec valeur positive dans la division entière de $-29$ par $-6$ est $\reste{p}{-29}{-6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $30$ par $6$ est $\reste{v}{30}{6}$.\\
Le reste avec le meme signe que le module dans la division entière de $30$ par $6$ est $\reste{s}{30}{6}$.\\
Le reste avec valeur positive dans la division entière de $30$ par $6$ est $\reste{p}{30}{6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $30$ par $-6$ est $\reste{v}{30}{-6}$.\\
Le reste avec le meme signe que le module dans la division entière de $30$ par $-6$ est $\reste{s}{30}{-6}$.\\
Le reste avec valeur positive dans la division entière de $30$ par $-6$ est $\reste{p}{30}{-6}$.\\
Le reste avec la valeur absolue la plus petite dans la division entière de $0$ par $0$ est $\reste{v}{0}{0}$.\\
Le reste avec le meme signe que le module dans la division entière de $0$ par $0$ est $\reste{s}{0}{0}$.\\
Le reste avec valeur positive dans la division entière de $0$ par $0$ est $\reste{p}{0}{0}$.\\
Le plus grand commun diviseur de $29$ et $6$ est $\pgcd{29}{6}$.
Le plus grand commun diviseur de $0$ et $6290$ est $\pgcd{0}{6290}$.
Le plus grand commun diviseur de $6324$ et $0$ est $\pgcd{6324}{0}$.
Le plus grand commun diviseur de $0$ et $-6290$ est $\pgcd{0}{-6290}$.
Le plus grand commun diviseur de $-6324$ et $0$ est $\pgcd{-6324}{0}$.
Le plus grand commun diviseur de $6324$ et $6290$ est $\pgcd{6324}{6290}$.
Le plus grand commun diviseur de $-6324$ et $6290$ est $\pgcd{-6324}{6290}$.
Le plus grand commun diviseur de $6324$ et $-6290$ est $\pgcd{6324}{-6290}$.
Le plus grand commun diviseur de $-6324$ et $-6290$ est $\pgcd{-6324}{-6290}$.
Le plus grand commun diviseur de $0$ et $0$ est $\pgcd{0}{0}$.
\end{document}

\reste{\thep}{\theq}
is a set of instructions to print the rest of the division, it doesn't just deliver the rest as a number. – egreg Feb 25 '20 at 10:10