You can use tikz
to create nodes in the tree with whatever options you prefer. For example, you could create a node with a label with a dotted border with rounded corners like this:
\begin{tikzpicture}
[
scale=1.2,
every node/.style={align=center},
level distance=50pt,
sibling distance=5pt,
]
\Tree
[.{$\alpha(\beta)$}:$\tau^a$
[.{$\alpha$}:$\langle\sigma^a,\tau^a\rangle$ \node[label={[draw, densely dotted, rounded corners]below:\strut$\gamma:\rho^c$}]{$\bullet$};
]
[.$\beta$:$\sigma^a$ \node[label={[draw, densely dotted, rounded corners]below:\strut$\delta:\vartheta^c$}]{$\bullet$};
]
]
\end{tikzpicture}

However, I would recommend forest
which offers all the power of TikZ and more. If you wish, you could then put the final lines into a node and have the circles added automatically by Forest with whatever options you wish, adding the dotted border etc. at the same time. For example, you might use the following code for the tree's preamble.
for tree={
math content,
By default, nodes have maths content.
parent anchor=children,
child anchor=parent,
Neaten things a bit.
},
before typesetting nodes={
Delay this until everything is ready to go.
where n children=0{
For the terminal nodes only.
no edge,
Don't draw an edge to them.
draw,
Do draw a border for them.
densely dotted,
Make that border densely dotted.
content/.wrap value={\strut#1},
Make them all the same height.
rounded corners,
Round the corners.
replace by={[, circle, fill, no edge, inner sep=0pt, minimum size=5pt, append]}
Insert another node before the terminal node which is just a small, filled circle. Don't draw an edge to that node either. the node is asserted by replace by
ing the original with the substitute and then append
ing the original. We do this later because otherwise the original nodes haven't all been saved and we may lose stuff if we try moving them around.
}{}
Don't do anything to non-terminal nodes.
}
End of delayed stuff.
This allows us to then specify the tree itself quite concisely and without switching to maths mode all the time.
[\alpha(\beta):\tau^a
[{\alpha:\langle\sigma^a,\tau^a\rangle}
[\gamma:\rho^c
]
]
[\beta:\sigma^a
[\delta:\vartheta^c
]
]
]
Assembling this,
\begin{forest}
for tree={
math content,
parent anchor=children,
child anchor=parent,
},
before typesetting nodes={
where n children=0{
no edge,
draw,
densely dotted,
content/.wrap value={\strut#1},
rounded corners,
replace by={[, circle, fill, no edge, inner sep=0pt, minimum size=5pt, append]}
}{}
}
[\alpha(\beta):\tau^a
[{\alpha:\langle\sigma^a,\tau^a\rangle}
[\gamma:\rho^c
]
]
[\beta:\sigma^a
[\delta:\vartheta^c
]
]
]
\end{forest}
produces

\documentclass[tikz,border=10pt,multi]{standalone}
\usepackage{tikz-qtree}
\usepackage{forest}
\begin{document}
\begin{tikzpicture}
[
scale=1.2,
every node/.style={align=center},
level distance=50pt,
sibling distance=5pt,
]
\Tree
[.{$\alpha(\beta)$}:$\tau^a$
[.{$\alpha$}:$\langle\sigma^a,\tau^a\rangle$ \node[label={[draw, densely dotted, rounded corners]below:\strut$\gamma:\rho^c$}]{$\bullet$};
]
[.$\beta$:$\sigma^a$ \node[label={[draw, densely dotted, rounded corners]below:\strut$\delta:\vartheta^c$}]{$\bullet$};
]
]
\end{tikzpicture}
\begin{forest}
for tree={
math content,
parent anchor=children,
child anchor=parent,
},
before typesetting nodes={
where n children=0{
no edge,
draw,
densely dotted,
content/.wrap value={\strut#1},
rounded corners,
replace by={[, circle, fill, no edge, inner sep=0pt, minimum size=5pt, append]}
}{}
}
[\alpha(\beta):\tau^a
[{\alpha:\langle\sigma^a,\tau^a\rangle}
[\gamma:\rho^c
]
]
[\beta:\sigma^a
[\delta:\vartheta^c
]
]
]
\end{forest}
\end{document}
forest
package? It's meant for all sorts of trees and very good in its Job...:)