Skip to content

larq.quantizers

A Quantizer defines the way of transforming a full precision input to a quantized output and the pseudo-gradient method used for the backwards pass.

ste_sign

ste_sign(x)

Sign binarization function. \[ q(x) = \begin{cases} -1 & x < 0 \\ 1 & x \geq 0 \end{cases} \]

The gradient is estimated using the Straight-Through Estimator (essentially the binarization is replaced by a clipped identity on the backward pass). \[\frac{\partial q(x)}{\partial x} = \begin{cases} 1 & \left|x\right| \leq 1 \\ 0 & \left|x\right| > 1 \end{cases}\]

−2 −1 0 1 2 x −1.0 −0.5 0.0 0.5 1.0 y Forward pass −2 −1 0 1 2 x 0.0 0.2 0.4 0.6 0.8 1.0 dy / dx Backward pass

Arguments

  • x: Input tensor.

Returns

Binarized tensor.

References

approx_sign

approx_sign(x)

Sign binarization function. \[ q(x) = \begin{cases} -1 & x < 0 \\ 1 & x \geq 0 \end{cases} \]

The gradient is estimated using the ApproxSign method. \[\frac{\partial q(x)}{\partial x} = \begin{cases} (2 - 2 \left|x\right|) & \left|x\right| \leq 1 \\ 0 & \left|x\right| > 1 \end{cases} \]

−2 −1 0 1 2 x −1.0 −0.5 0.0 0.5 1.0 y Forward pass −2 −1 0 1 2 x 0.0 0.5 1.0 1.5 2.0 dy / dx Backward pass

Arguments

  • x: Input tensor.

Returns

Binarized tensor.

References

magnitude_aware_sign

magnitude_aware_sign(x)

Magnitude-aware sign for Bi-Real Net.

−2 −1 0 1 2 x −2 −1 0 1 2 y Forward pass −2 −1 0 1 2 x 0.0 0.2 0.4 0.6 0.8 1.0 dy / dx Backward pass

Arguments

  • x: Input tensor

Returns

Scaled binarized tensor (with values in \{-a, a\}, where a is a float).

References

SteTern

SteTern(threshold_value=0.1, ternary_weight_networks=False)

Ternarization function. \[ q(x) = \begin{cases} +1 & x > \Delta \\ 0 & |x| > \Delta \\ -1 & x < \Delta \end{cases} \]

where \Delta is defined as the threshold and can be passed as an argument, or can be calculated as per the Ternary Weight Networks original paper, such that

\[ \Delta = \frac{0.7}{n} \sum_{i=1}^{n} |W_i| \] where we assume that W_i is generated from a normal distribution.

The gradient is estimated using the Straight-Through Estimator (essentially the Ternarization is replaced by a clipped identity on the backward pass). \[\frac{\partial q(x)}{\partial x} = \begin{cases} 1 & \left|x\right| \leq 1 \\ 0 & \left|x\right| > 1 \end{cases}\]

−2 −1 0 1 2 x −1.0 −0.5 0.0 0.5 1.0 y Forward pass −2 −1 0 1 2 x 0.0 0.2 0.4 0.6 0.8 1.0 dy / dx Backward pass

Arguments

  • x: Input tensor.
  • threshold value: The value for the threshold, \Delta.
  • ternary_weight_networks: Boolean of whether to use the Ternary Weight Networks threshold calculation.

Returns

Ternarized tensor.

References