You're probably having a tough time making a smooth transition from the A chord to the Bm.
For practice, you could try to cover all three strings of the A chord with the tip of your index finger, lifting the back of it enough to make sure that the open E is not muted, and then laying the fingers down for the Bm chord.
I tried to make a short video clip of what I mean:
I'm just thinking that it might make it easier for you, but of course, if you're just starting out, this will take some time. As mentioned, barre chords are a necessary evil. Might as well jump in now!
Just be patient, and don't get overly frustrated. It takes time.