Yes its that cold. In fact thats pretty mild for Japan. You will find that during spring, so long as their isn't a cold wind, its still quite pleasant. And cold winds are not common until winter.
I would bring only one heavy jacket but I think you will find a light jumper will be all you need depending how long you stay.