The two-envelope switching trap

Two sealed envelopes are on the table. One holds exactly twice as much money as the other, but you do not know which is which. You pick one envelope and hold it, unopened. Then you reason:

"Let my envelope hold $A$ dollars. The other envelope is equally likely to hold $2A$ or $A/2$, so its expected value is $\tfrac12(2A) + \tfrac12\left(\tfrac{A}{2}\right) = \tfrac{5}{4}A$, which is more than $A$. I should switch."

But after switching you could relabel the new envelope's amount and run the same argument, telling you to switch back, forever.

Explain what is wrong with this reasoning.