Based upon what I took a look at, it's bitter almonds that have amygdalin. In terms of dates, they apparently have high levels of cadmium and lead (same as cacao, by the way) depending on where they're grown. As for cashews, they're edible since they do come from the same species as poison ivy.
For the cashew trade, I had no idea about the disgusting nature of it. Do you have proof for that I could look at?