In North America you should be able to. Other regions have nuances that might impact a downgrade.
Never hurts to call and ask them, or your travel agent if booked through a TA.
I don't know about outside the U.S., but we just recently downgraded thru our travel agent and my purchases made on cruise planner stayed exactly the same.
The four year old doesnt need "the key"
Here is the fine print from the cruise planner.
guests ages 6 and older in the same stateroom are required to purchase the Package. Guests age 5 and younger accompanying a guest who purchased The Key can enjoy these benefits: priority check-in and boarding, exclusive welcome lunch at Chops Grille or Jamie’s Italian, priority departure at ports of call, seats in the exclusive VIP seating area at shows, and choice departure on debarkation day with the exclusive à la carte breakfast.
