Mutual information quantifies the impact of observing one variable on the uncertainty of another.
(defn MI [X Y worlds]
(transduce (map (fn [[x y]]
(* (Pr (And x y) worlds)
(log2 (/ (Pr (And x y) worlds)
(* (Pr x worlds)
(Pr y worlds)))))))
+
(for [x (values-of X worlds)
y (values-of Y worlds)]
[x y])))
(assert (about= (ENT "Burglary" tb-3-1) 0.7219))
(assert (about= (MI "Burglary" "Alarm" tb-3-1) 0.3924))