How many fancy gold fish in a ..........?

I was very good and only bought 4 Fancy gold fish to live in my 50 gallon aquarium even though they're tiny at the moment. But I was in Walmart today getting popsicles and I always go look at the fish ( I shouldn't because I feel horrible for the poor things) and lo and behold they have 2 Panda Orada gold fish! Would it really kill my tank to have 2 more fish in it. I upgraded the filtration to a 1200 gph canister filter and I do water changes twice a week in most cases because I get bored. So what better way to pass the time and keep my fish healthy at the same time. Would it really cramp the tank to have to more fancy gold fish in it?