Two reasons.
1) Impending Doom has exponential growth. Every level of your main skill gem is very important. Elemental Weakness has the tags "Lightning", "Fire", "Cold", and "Curse" which gives you a bunch of options to rank the skill up and save on some currency when buying weapons. With putting it directly on Despair, you only have the options of "+all spell" and "+curse skills" to rank it up.
2) The build utilizes "Gain X damage as extra cold/fire/lightning damage". Assuming you are using a staff you should be hitting +100% or more total extra damage. This extra damage is as the type described in the text. Despair only gives chaos weakness, not elemental weakness. Elemental weakness scales this bonus (and this bonus over half your total damage).