entr

→Tensor

entr(x: Tensor)

source edit

Element-wise entropy kernel $-x \log x$ .

Returns the per-element contribution to Shannon entropy. This is the standard building block for evaluating $H(p) = \sum_i \mathrm{entr}(p_i)$ for a discrete distribution, with the limit convention $0 \log 0 = 0$ applied at the boundary.

Parameters

xTensor

Input tensor. Integer and bool inputs are taken as float32. Values below zero give -inf.

Returns

Tensor

Element-wise $-x \log x$ with the limit convention at x = 0; same shape as x.

Notes

Definition:

\mathrm{entr}(x) = \begin{cases} -x \log x, & x > 0 \\[2pt] 0, & x = 0 \\[2pt] -\infty, & x < 0. \end{cases}

The function is concave with maximum $1/e \approx 0.3679$ at $x = 1/e$ . It vanishes at both x = 0 (limit) and x = 1. Related quantities include rel_entr (point-wise relative entropy) and kl_div (KL kernel).

It is $-\operatorname{xlogy}(x, x)$ and takes the 0 \log 0 convention from lucid.xlogy, so the gradient at x = 0 is 0 rather than the one-sided $+\infty$ of $-(1 + \log x)$ that the reference framework returns. Below zero the gradient is NaN, as in the reference.

Examples

>>> import lucid
>>> from lucid.special import entr
>>> entr(lucid.tensor([0.0, 0.5, 1.0, 2.0]))
tensor([0., 0.3466, 0., -1.386])