Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions src/posts/bf16-in-golang.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,16 @@ In the era of large language models, low-precision floating-point numbers are no

## Introduction to BF16

```
+------------+-----------------+-----------------+
| 1 sign bit | 8 exponent bits | 7 mantissa bits |
+------------+-----------------+-----------------+
```mermaid
---
config:
packet:
bitsPerRow: 16
---
packet
+1: "Sign"
+8: "Exponent"
+7: "Mantissa"
```

BF16 consists of 1 sign bit, 8 exponent bits, and 7 mantissa bits. Compared to FP32, it has the same number of exponent bits but significantly fewer mantissa bits. As a result, BF16 maintains the same dynamic range as FP32 (approximately $±3.4×10³⁸$), sacrificing some precision (around $2⁻⁷$) for a larger exponent range. The larger exponent range effectively avoids gradient overflow and underflow, making it suitable for backpropagation in deep learning. Additionally, it has strong compatibility with FP32, simplifying hardware design. BF16 is used to replace FP32 in model training and inference scenarios, reducing memory usage and bandwidth requirements while increasing the number of data processed by SIMD instructions in a single operation.
Expand Down
14 changes: 10 additions & 4 deletions src/zh/posts/bf16-in-golang.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,16 @@ tag:

## BF16介绍

```
+---------+--------+--------+
| 1符号位 | 8指数位 | 7尾数位 |
+---------+--------+--------+
```mermaid
---
config:
packet:
bitsPerRow: 16
---
packet
+1: "符号位"
+8: "指数位"
+7: "尾数位"
```

BF16由1位符号位、 8位指数位和7位尾数位构成。和FP32相比,指数位相同,但尾数位大幅减少。因此BF16动态范围与FP32完全一致(约$±3.4×10³⁸$),牺牲部分精度(约$2^{-7}$)换取更大的指数范围。大指数范围可有效避免梯度溢出,适合深度学习中的反向传播。同时与FP32兼容性强,硬件设计更简单。BF16用于在模型训练和推理场景替代FP32,降低内存占用和带宽需求,提升SIMD指令一次处理数据的数量。
Expand Down
Loading