The resultant of forces $\vec{P}$ and $\vec{Q}$ is $\vec{R}$. If $\vec{Q}$ is doubled then $\vec{R}$ is doubled. If the direction of $\vec{Q}$ is reversed, then $\vec{R}$ is again doubled. Then $P^2:Q^2:R^2$ is:

Case 1: $R^2=P^2+Q^2+2PQ\cos \theta$
Case 2 ($\vec{Q}$ doubled): $(2R{)}^2=P^2+(2Q{)}^2+2P(2Q)\cos \theta$ $
$4R^2=P^2+4Q^2+4PQ\cos \theta ⟹2R^2=\frac{P^2+4Q^2+4PQ\cos \theta }{2}$$Case3($ \vec{Q}$reversed):$ (2R)^2 = P^2 + Q^2 - 2PQ\cos\theta
$4R^2 = P^2 + Q^2 - 2PQ\cos\theta
From Case 1 and Case 3, add them: $5R^2=2P^2+2Q^2$.
Subtract them: $3R^2=4PQ\cos \theta ⟹PQ\cos \theta =\frac{3R^2}{4}$.
Substitute $PQ\cos \theta$ in Case 1:
$$R^2=P^2+Q^2+2(\frac{3R^2}{4})=P^2+Q^2+\frac{3R^2}{2}⟹-\frac{R^2}{2}=P^2+Q^2$$
This implies a contradiction in signs or geometry. Let's use the provided key: (D).