Square both expressions and add:

\((\sin x + a\cos x)^2 + (a\sin x - \cos x)^2\)
\(= \sin^2 x + a^2\cos^2 x + 2a\sin x\cos x + a^2\sin^2 x + \cos^2 x - 2a\sin x\cos x = 1 + a^2\)

So \(b^2 + (a\sin x - \cos x)^2 = 1 + a^2\), which gives \((a\sin x - \cos x)^2 = a^2 - b^2 + 1\).

\(|a\sin x - \cos x| = \sqrt{a^2 - b^2 + 1}\)

Answer: (D)