Muslim rulers broke temple