Fonts issues in xelatex

Question

I would like to know what is the right way to define all the text font as a specific size. I use \documentclass[20pt]{article} but it won't change the font size in xelatex. And these symbols (" and ') are not Times fonts and now matter how I changed the fonts , they look the same. Here is the min working code. I would like to know what is the right way to address font selection in xelatex. Can I get like this ?
Instead of this ?

 %!TEX TS-program = xelatex

 \documentclass[20pt]{article}
 \usepackage{geometry}
 \usepackage{mathspec}
 \usepackage{lipsum}
 \setmainfont{Times}
 \setmathsfont(Digits,Latin,Greek){Times}
 \usepackage{titling}
 \usepackage{titlesec}
 \usepackage{setspace}
 \usepackage[colorlinks,urlcolor=black]{hyperref}
 \doublespacing
 \titleformat{\section}{\singlespacing\fontsize{20}{24}\bfseries\rmfamily\uppercase} {\thesection.}{0.5em}{}
  \begin{document}
 \section{"check"}
"just checking" this is a "check" and 'check'. 
 \end{document}

Answer 1

In case if you are tied to article document class, here are the two options:

Option-1

You can use scrextend package, that extends the facilities (like font sizes) of `KOMA-script classes to other classes:

%!TEX TS-program = xelatex

\documentclass{article}
\usepackage{scrextend}         %%% this line
\changefontsizes[24pt]{20pt}   %%% and this line
\usepackage{geometry}
\usepackage{mathspec}
\usepackage{lipsum}
\setmainfont[Ligatures=TeX]{Times New Roman}
\setmathsfont(Digits,Latin,Greek){Times New Roman}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{setspace}
\usepackage[colorlinks,urlcolor=black]{hyperref}
\doublespacing
\titleformat{\section}{\singlespacing\fontsize{20}{24}\bfseries\rmfamily\uppercase} {\thesection.}{0.5em}{}
\begin{document}
\section{``check''}
``just checking'' this is a ``check'' and `check'.
\end{document}

Option-2

You can use extsizes package, which is a package version of extarticle mentioned by Karl Koeller in his answer.

%!TEX TS-program = xelatex

\documentclass{article}
%\usepackage{scrextend}
%\changefontsizes[24pt]{20pt}
\usepackage[20pt]{extsizes} %%% this line
\usepackage{geometry}
\usepackage{mathspec}
\usepackage{lipsum}
\setmainfont[Ligatures=TeX]{Times New Roman}
\setmathsfont(Digits,Latin,Greek){Times New Roman}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{setspace}
\usepackage[colorlinks,urlcolor=black]{hyperref}
\doublespacing
\titleformat{\section}{\singlespacing\fontsize{20}{24}\bfseries\rmfamily\uppercase} {\thesection.}{0.5em}{}
\begin{document}
\section{``check''}
``just checking'' this is a ``check'' and `check'.
\end{document} 

Answer 2

The article class only supports 10pt, 11pt, and 12pt.

You can use the extarticle (extended article) class which supports 20pt.

In the following MWE I've changed "Times" to "Times New Roman" and I get the correct quotes using Ligatures=TeX option, but you have to use the standard LaTeX way for double quotes.

%!TEX TS-program = xelatex

\documentclass[20pt]{extarticle}
\usepackage{geometry}
\usepackage{mathspec}
\usepackage{lipsum}
\setmainfont[Ligatures=TeX]{Times New Roman}
\setmathsfont(Digits,Latin,Greek){Times New Roman}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{setspace}
\usepackage[colorlinks,urlcolor=black]{hyperref}
\doublespacing
\titleformat{\section}{\singlespacing\fontsize{20}{24}\bfseries\rmfamily\uppercase} {\thesection.}{0.5em}{}
\begin{document}
\section{``check''}
``just checking'' this is a ``check'' and `check'.
\end{document} 

Output

If you want "..." to behave this way, you can add these two lines in your preamble:

\usepackage{csquotes}
\MakeOuterQuote{"}

Answer 3

use

\documentclass[fontsize=20pt]{scrartcl}