The smallest number which when divided by $12$, $15$, $20$ or $54$ leaves a remainder of $4$ in each case, is:

The required smallest number is of the form $\text{LCM}(12,15,20,54)+\text{remainder}$.
First, let's find the Prime Factorization of each number:

$$12=2^2\times 3$$

$$15=3\times 5$$

$$20=2^2\times 5$$

$$54=2\times 3^3$$

Now, find the LCM by taking the highest power of each prime factor involved:

$$\text{LCM}=2^2\times 3^3\times 5$$

$$\text{LCM}=4\times 27\times 5=540$$

Adding the required remainder of $4$:

$$\text{Required Number}=540+4=544$$