G'day J-Man.
Yes, you still need an electrolyte in a dry cell.

The stored voltage you are seeing after applying power to your dry cell is quite normal, rather like a capacitor.
In a wet cell,...