Caulk

If this is an outside job, one that is to be exposed to lots of sunlight, and wild variations in temperature, then try to go with silicone. If the surfaces to be joined or filled are interior, and will not be in direct sunlight, chances are you will want to use a latex caulk.

Why?

Silicone caulks will have pretty much the same properties in harsh subzero conditions as it will on the hottest day of summer or next to say, a furnace or oven. Latex caulk breaks down rapidly in the UV rays of the sun unless it is painted. Breakdown means it will crack up and become crumbly. Extreme heat or cold will cause latex caulks to lose elasticity since they are water based ( heat dries them up and the cold makes them stiff). Freezing temperatures will cause latex caulks to become less elastic, perhaps even brittle if it was exposed to sunlight the previous summer. Silicone, however, will still be rubbery and flexible when the tree limbs are snapping like match sticks during an ice storm. Specially formulated silicone caulks are also available that are mildew resistant for the bathroom or kitchen. The bad part about silicone: it generally cannot be painted, is sold in few colors, and is tricky stuff to use and clean up.

Latex caulk is oh so easy to use. You can be a novice and still pull off a great caulking job with latex because it cleans up easily and remains workable for a little while. It is almost always white because it is paintable. Since latex caulk is water based, a damp rag or sponge will clean up the areas you didn't intend to caulk but somehow got smeared with the stuff. Again, it will shrink just a little as time passes.

A messy silicone job is perhaps the worst thing that can happen to a window, door frame, tub surround or shower door. Silicone is going to last a long time if it is applied to a clean and dry surface. It has excellent adhesion properties which means it will stick well to almost anything. It's best to take the necessary precautions to ensure a good silicone job rather than clean it up. In other words, use masking tape and keep some mineral spirits or whatever is suggested in the manufacturer's cleanup instructions.

Here are some more scientific but well explained comparisons by the makers of both:

General Electric
DAP