Edit:
There was linked code in the comments, which generated wrong page order. That was caused by another bug in tex4ebook in epub3 mode. The problem is that pages which contain math should be marked in the OPF file. Every chapter creates it's own html page. Math was configured to register current in page sequence list and to register mathml property for the curent page. The problem was that pages without math were registered in the page sequence later than pages with math, so they came earlier in the page order. I've fixed that on Github, I will wait some time for other bug before I will update CTAN.
The other issue is with \uv command, which is provided by Czech babel and it is used to print quotes. It is translated as
<sub>”</sub>=lsložitý“
with OP's code. The issue can be solved with fontenc package with T1 encoding. Correct file would be then:
\documentclass[11pt]{report}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[czech]{babel}
\usepackage{hyperref}
% \usepackage{tex4ebook}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{latexsym}
\usepackage{esvect}
\usepackage{color}
\usepackage{makeidx}
\title{Geometrické algoritmy}
\author{Mgr. Stanislav Kopejtko}
\date{25. 7. 2014}
\begin{document}
\maketitle
\tableofcontents
\chapter*{Předmluva}
%
\uv{složitý}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\addcontentsline{toc}{chapter}{Předmluva}
\markboth{Předmluva}{Předmluva}
\part{test 2}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\markboth{}{}
\clearpage
\chapter*{test 3}
\(x=\dfrac{x^2}{x}\)
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\section{test 4}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\subsection{test 5}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\chapter{test 6}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\(x=\dfrac{x^2}{x}\)
\section{test 7}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\subsection{test 8}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
\(x=\dfrac{x^3}{x^2}\)
\end{document}
note that you don't need to use tex4ebook directly in your document unless you want to include the cover image in the main document.
I've found also another problem, which is actually tex4ht bug:
\chapter*{Předmluva}
%
\uv{složitý}
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
is converted as
<h2 class="likechapterHead"><a
id="x3-2000"></a>Předmluva</h2> „složitý“ Lorem ipsum dolor sit amet, consectetur adipisicing
elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim
ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat. <a
id="Q1-3-3"></a>
with missing <p> tags. The problem is in configuration for \chapter* command, it doesn't request new paragraph. Until this bug is resolved in tex4ht, you can use following config file, mycfg.cfg:
\Preamble{xhtml}
\Configure{likechapter}{}{}
{\IgnorePar\EndP\HCode{<h2 class="likechapterHead">}}
{\HCode{</h2>}\IgnoreIndent\par}
\begin{document}
\EndPreamble
\chapter* is configured with \Configure{likechapter}, the only change from default configuration is \par command after \IgnoreIndent.
Compile the document with:
tex4ebook -f epub3 -c mycfg filename mathml
the result:
<h2 class="likechapterHead"><a
id="x3-2000"></a>Předmluva</h2>
<p class="noindent">„složitý“ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. <a
id="Q1-3-3"></a>
</p>
This is a regression, Latin1 output with XML entities for accented characters is generated. It probably doesn't work in epub3 because Unicode is required in this format, in epub2 it is optional.
This bug was caused by unification between make4ht and tex4ebook, in order to support the same handling of options. But I forgot that I need to handle Unicode, which is used by default in tex4ebook, but with --unicode option in make4ht.
I've updated tex4ebook on Github and will update it on CTAN soon.
