I recomend not doing a lot of exertion till you get over your actual cold. The toe thing, depends on how it's healing up. I went to work with a broken little toe for the entire time it was healing nearly, Just had to wait about three days before the swelling went down enough to get a shoe on comfortably.

As long as it's not rebreaking the bone or stressing the healing area, I personally think you're fine doing a fair amount of physical stuff, providing it doesn't hurt overly much while doing so. It's nice to be able to ignore pain, but it is the way your body signals damage too, so.

It does seem though, that if you have a cold and overdo it, the cold gets worse, based off a sampling of about 30-ish people who I have known to overdo it with colds.