class

ContinuousBernoulli

extendsDistribution

ContinuousBernoulli(probs: Tensor | float | None = None, logits: Tensor | float | None = None, validate_args: bool | None = None)

source edit

Continuous Bernoulli distribution on $[0, 1]$ .

A continuous relaxation of the lucid.distributions.Bernoulli distribution introduced by Loaiza-Ganem & Cunningham (2019) to fix the well-known bias incurred by VAE decoders that use a Bernoulli likelihood on real-valued image pixels in $[0, 1]$ . Unlike the naive "Bernoulli on [0,1]" likelihood, the Continuous Bernoulli has a proper normalising constant and yields unbiased ELBO gradients.

Parameters

probsTensor or float= None

Parameter

\lambda \in [0, 1]

. Note that

\mathbb{E}[X] \neq \lambda

in general — see Notes for the closed-form mean. Mutually exclusive with logits.

logitsTensor or float= None

Log-odds

\ell = \log(\lambda / (1 - \lambda))

. Mutually exclusive with probs.

validate_argsbool= None

If True, validate parameter constraints at construction time.

Notes

Probability density on $x \in [0, 1]$ :

p(x; \lambda) = C(\lambda)\, \lambda^x (1 - \lambda)^{1 - x}

where the normalising constant is

C(\lambda) = \begin{cases} \dfrac{2 \tanh^{-1}(1 - 2\lambda)}{1 - 2\lambda} & \lambda \neq \tfrac{1}{2} \\[4pt] 2 & \lambda = \tfrac{1}{2} \end{cases}

Mean (closed form):

\mathbb{E}[X] = \begin{cases} \dfrac{\lambda}{2\lambda - 1} + \dfrac{1}{2 \tanh^{-1}(1 - 2\lambda)} & \lambda \neq \tfrac{1}{2} \\[4pt] \tfrac{1}{2} & \lambda = \tfrac{1}{2} \end{cases}

Variance (closed form, $\ell = \operatorname{logit}\lambda$ ):

\mathrm{Var}[X] = \frac{\lambda(\lambda - 1)}{(1 - 2\lambda)^2} + \frac{1}{\ell^2} \quad (\lambda \neq \tfrac{1}{2}), \qquad \mathrm{Var}[X] = \tfrac{1}{12} \quad (\lambda = \tfrac{1}{2})

Both closed forms are differences of terms of order $1/(1 - 2\lambda)$ , which cancel catastrophically as $\lambda \to \tfrac{1}{2}$ ; within $|\lambda - \tfrac{1}{2}| < 0.1$ the mean and variance are evaluated from their Taylor series in $\lambda - \tfrac{1}{2}$ instead.

Important properties:

Support: $[0, 1]$ (continuous, not just $\{0, 1\}$ ).
Reduces to $\mathrm{Uniform}(0, 1)$ at $\lambda = 1/2$ .
At $\lambda \to 0$ or $\lambda \to 1$ the density concentrates near 0 or 1 respectively.
Reparameterisable via inverse CDF.

The Continuous Bernoulli is the correct likelihood for decoders that output a value in $[0, 1]$ (e.g. MNIST pixel intensities) when the model is to remain a valid probabilistic model. Using a plain Bernoulli on continuous data produces an improper density and a systematically biased ELBO.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> d = ContinuousBernoulli(probs=0.7)
>>> d.rsample((4,)).shape
(4,)
>>> d.log_prob(lucid.tensor(0.5))
tensor(-0.02974)

Used by 2

Constructors

dunder

init

→None

__init__(probs: Tensor | float | None = None, logits: Tensor | float | None = None, validate_args: bool | None = None)

source edit

Initialise a Continuous Bernoulli distribution.

Parameters

probsTensor | float | None= None

Parameter

p \in [0, 1]

. Mutually exclusive with logits.

logitsTensor | float | None= None

Log-odds

l = \log(p/(1-p)) \in \mathbb{R}

. Mutually exclusive with probs.

validate_argsbool | None= None

If True, validate parameter constraints at construction time.

Properties

prop

mean

→Tensor

mean: Tensor

source edit

Mean $\lambda/(2\lambda - 1) + 1/(2\tanh^{-1}(1 - 2\lambda))$ .

The second term is $-1/\operatorname{logit}\lambda$ . Near $\lambda = \tfrac12$ the two terms cancel, so there the Taylor series $\tfrac12 + \tfrac{t}{3} + \tfrac{16 t^3}{45} + \tfrac{704 t^5}{945} + \tfrac{27392 t^7}{14175}$ in $t = \lambda - \tfrac12$ is used.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=lucid.tensor([0.2, 0.5, 0.7])).mean
tensor([0.388, 0.5, 0.5698])

prop

variance

→Tensor

variance: Tensor

source edit

Variance $\lambda(\lambda-1)/(1-2\lambda)^2 + 1/\ell^2$ .

$\ell = \operatorname{logit}\lambda$ . The two terms are each of order $1/(1-2\lambda)^2$ and cancel near $\lambda = \tfrac12$ , so there the Taylor series $\tfrac1{12} - \tfrac{x}{15} - \tfrac{128 x^2}{945} - \tfrac{4864 x^3}{14175}$ in $x = (\lambda - \tfrac12)^2$ is used. Nothing is clamped: the variance is positive for every $\lambda$ , and a negative value would be a defect to see, not to hide.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=lucid.tensor([0.2, 0.5, 0.7])).variance
tensor([0.0759, 0.08333, 0.08043])

Instance methods

cdf

→Tensor

cdf(value: Tensor)

source edit

Cumulative distribution function of the Continuous Bernoulli.

F(x) = \frac{\lambda^x (1 - \lambda)^{1 - x} + \lambda - 1} {2\lambda - 1} = \frac{\operatorname{expm1}(\ell x)}{\operatorname{expm1}(\ell)}

with $\ell = \operatorname{logit}\lambda$ , and $x$ at $\lambda = \tfrac12$ . The second form has no cancellation as $\lambda \to \tfrac12$ , where the reference framework replaces the first with $x$ inside $|\lambda - \tfrac12| < 10^{-3}$ ; there the two differ by up to $|\ell|/8$ , and this is the exact value.

Returns probabilities in $[0, 1]$ ; 0 at and below 0, 1 at and above 1.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.7).cdf(lucid.tensor(0.5))
tensor(0.3956)

entropy

→Tensor

entropy()

source edit

Shannon entropy of the Continuous Bernoulli (in nats).

H = \operatorname{softplus}(\ell) - \ell\,\mathbb E[X] - \ln C(\lambda)

$-\mathbb E[\log p(X)]$ with $\log p(x) = x\ell - \operatorname{softplus}(\ell) + \ln C$ , so it reuses mean and the normaliser, each already evaluated by its series near $\lambda = \tfrac12$ .

Returns

Tensor

Entropy in nats, shape batch_shape; 0 at $\lambda = \tfrac12$ , where the distribution is uniform.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.5).entropy()
tensor(0.)

icdf

→Tensor

icdf(value: Tensor)

source edit

Quantile function of the Continuous Bernoulli.

Q(p) = \frac{\operatorname{log1p}\!\left(p \operatorname{expm1}(\ell)\right)}{\ell}

for $\ell < 0$ , reflected through $Q_\ell(p) = 1 - Q_{-\ell}(1 - p)$ for $\ell > 0$ , and $p$ at $\ell = 0$ .

Returns quantiles in $[0, 1]$ .

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.7).icdf(lucid.tensor(0.3956))
tensor(0.5)

log_prob

→Tensor

log_prob(value: Tensor)

source edit

x · l − softplus(l) + log C(p).

softplus rather than log(1 + exp(l)), which reached inf at l ≈ 89 in float32; and the logit of a probs of exactly 0 or 1 is taken one epsilon inside [0, 1] (logits), so it is never multiplied by a zero x while infinite.

logits

→Tensor

logits()

source edit

Log-odds of $\lambda$ — as given, or derived on access.

Derived from probs clamped one epsilon inside $[0, 1]$ , as the reference framework derives it.

probs

→Tensor

probs()

source edit

Parameter $\lambda$ — as given, or sigmoid(logits).

rsample

→Tensor

rsample(sample_shape: tuple[int, ...] = ())

source edit

Reparameterised sample via the closed-form icdf.

class

ContinuousBernoulli

extendsDistribution

ContinuousBernoulli(probs: Tensor | float | None = None, logits: Tensor | float | None = None, validate_args: bool | None = None)

source edit

Continuous Bernoulli distribution on $[0, 1]$ .

Parameters

probsTensor or float= None

Parameter

\lambda \in [0, 1]

. Note that

\mathbb{E}[X] \neq \lambda

in general — see Notes for the closed-form mean. Mutually exclusive with logits.

logitsTensor or float= None

Log-odds

\ell = \log(\lambda / (1 - \lambda))

. Mutually exclusive with probs.

validate_argsbool= None

If True, validate parameter constraints at construction time.

Notes

Probability density on $x \in [0, 1]$ :

p(x; \lambda) = C(\lambda)\, \lambda^x (1 - \lambda)^{1 - x}

where the normalising constant is

C(\lambda) = \begin{cases} \dfrac{2 \tanh^{-1}(1 - 2\lambda)}{1 - 2\lambda} & \lambda \neq \tfrac{1}{2} \\[4pt] 2 & \lambda = \tfrac{1}{2} \end{cases}

Mean (closed form):

\mathbb{E}[X] = \begin{cases} \dfrac{\lambda}{2\lambda - 1} + \dfrac{1}{2 \tanh^{-1}(1 - 2\lambda)} & \lambda \neq \tfrac{1}{2} \\[4pt] \tfrac{1}{2} & \lambda = \tfrac{1}{2} \end{cases}

Variance (closed form, $\ell = \operatorname{logit}\lambda$ ):

\mathrm{Var}[X] = \frac{\lambda(\lambda - 1)}{(1 - 2\lambda)^2} + \frac{1}{\ell^2} \quad (\lambda \neq \tfrac{1}{2}), \qquad \mathrm{Var}[X] = \tfrac{1}{12} \quad (\lambda = \tfrac{1}{2})

Important properties:

Support: $[0, 1]$ (continuous, not just $\{0, 1\}$ ).
Reduces to $\mathrm{Uniform}(0, 1)$ at $\lambda = 1/2$ .
At $\lambda \to 0$ or $\lambda \to 1$ the density concentrates near 0 or 1 respectively.
Reparameterisable via inverse CDF.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> d = ContinuousBernoulli(probs=0.7)
>>> d.rsample((4,)).shape
(4,)
>>> d.log_prob(lucid.tensor(0.5))
tensor(-0.02974)

Used by 2

Constructors

dunder

init

→None

__init__(probs: Tensor | float | None = None, logits: Tensor | float | None = None, validate_args: bool | None = None)

source edit

Initialise a Continuous Bernoulli distribution.

Parameters

probsTensor | float | None= None

Parameter

p \in [0, 1]

. Mutually exclusive with logits.

logitsTensor | float | None= None

Log-odds

l = \log(p/(1-p)) \in \mathbb{R}

. Mutually exclusive with probs.

validate_argsbool | None= None

If True, validate parameter constraints at construction time.

Properties

prop

mean

→Tensor

mean: Tensor

source edit

Mean $\lambda/(2\lambda - 1) + 1/(2\tanh^{-1}(1 - 2\lambda))$ .

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=lucid.tensor([0.2, 0.5, 0.7])).mean
tensor([0.388, 0.5, 0.5698])

prop

variance

→Tensor

variance: Tensor

source edit

Variance $\lambda(\lambda-1)/(1-2\lambda)^2 + 1/\ell^2$ .

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=lucid.tensor([0.2, 0.5, 0.7])).variance
tensor([0.0759, 0.08333, 0.08043])

Instance methods

cdf

→Tensor

cdf(value: Tensor)

source edit

Cumulative distribution function of the Continuous Bernoulli.

F(x) = \frac{\lambda^x (1 - \lambda)^{1 - x} + \lambda - 1} {2\lambda - 1} = \frac{\operatorname{expm1}(\ell x)}{\operatorname{expm1}(\ell)}

Returns probabilities in $[0, 1]$ ; 0 at and below 0, 1 at and above 1.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.7).cdf(lucid.tensor(0.5))
tensor(0.3956)

entropy

→Tensor

entropy()

source edit

Shannon entropy of the Continuous Bernoulli (in nats).

H = \operatorname{softplus}(\ell) - \ell\,\mathbb E[X] - \ln C(\lambda)

$-\mathbb E[\log p(X)]$ with $\log p(x) = x\ell - \operatorname{softplus}(\ell) + \ln C$ , so it reuses mean and the normaliser, each already evaluated by its series near $\lambda = \tfrac12$ .

Returns

Tensor

Entropy in nats, shape batch_shape; 0 at $\lambda = \tfrac12$ , where the distribution is uniform.

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.5).entropy()
tensor(0.)

icdf

→Tensor

icdf(value: Tensor)

source edit

Quantile function of the Continuous Bernoulli.

Q(p) = \frac{\operatorname{log1p}\!\left(p \operatorname{expm1}(\ell)\right)}{\ell}

for $\ell < 0$ , reflected through $Q_\ell(p) = 1 - Q_{-\ell}(1 - p)$ for $\ell > 0$ , and $p$ at $\ell = 0$ .

Returns quantiles in $[0, 1]$ .

Examples

>>> import lucid
>>> from lucid.distributions import ContinuousBernoulli
>>> ContinuousBernoulli(probs=0.7).icdf(lucid.tensor(0.3956))
tensor(0.5)

log_prob

→Tensor

log_prob(value: Tensor)

source edit

x · l − softplus(l) + log C(p).

logits

→Tensor

logits()

source edit

Log-odds of $\lambda$ — as given, or derived on access.

Derived from probs clamped one epsilon inside $[0, 1]$ , as the reference framework derives it.

probs

→Tensor

probs()

source edit

Parameter $\lambda$ — as given, or sigmoid(logits).

rsample

→Tensor

rsample(sample_shape: tuple[int, ...] = ())

source edit

Reparameterised sample via the closed-form icdf.