They were made infertile because Abimlech unknowingly took Sarah as his wife. He did not know she was Abraham’s wife.

Source: Genesis 20:18