Johnson has been under fire as sales plunged 25 percent during the first year of his plan to reinvent the department store chain. Last week, hedge fund manager Bill Ackman - who handpicked Johnson to lead the company - said the CEO had made "big mistakes" and that the impact of those mistakes had been "very close to a disaster" for the retailer.
Johnson had tried to do away with constant sales in favor of everyday low prices but shoppers appeared not to understand the plan. Shares rose 10.8% on news of Johnson's ousting.
"The lack of testing did him in," analyst Walter Loeb said of Johnson. "It is imperative that someone with a merchandising background take over."
Speculation is already mounting that Johnson could return to his vacant job position as SVP of Retail at Apple.