計算機結構與作業系統

Regarding CPU-accelerator data orchestration for LLM inference, consider a system designed for LLM inference with the following specifications:

• Hardware Configuration:
  • CPU: 128 GB DDR5 memory, 50 GB/s memory bandwidth, 2 TFLOPS INT8 throughput
  • NPU (Neural Processing Unit): 8 GB on-chip SRAM, 1 TB/s internal bandwidth, 400 TFLOPS INT8 throughput
  • Interconnect: PCIe 5.0 x16, 50 GB/s effective transfer bandwidth
• Model Characteristics (70B parameter LLM, INT8 quantization):
  • Total model weights: 70 GB
  • Number of transformer layers: 80
  • Weight size per layer: 875 MB
  • Computation operations per layer: 140 billion INT8 operations

Since the NPU delivers 200x higher computation throughput than the CPU, running inference on the CPU alone would be impractically slow (35 seconds per token versus 175 ms on NPU). The system must use the NPU for all matrix computations. However, the NPU's 8 GB SRAM cannot hold the 70 GB model, requiring layer weights to be streamed from CPU memory during inference.

The following code generates a single output token:

void generate_one_token() {
    // Process all 80 layers sequentially
    for (int L = 0; L < 80; L++) {
        // Transfer: CPU memory -> NPU SRAM (875 MB per layer)
        pcie_transfer_to_npu(weight[L], 875 MB);
        // Compute: 140 billion INT8 operations at 400 TFLOPS
        npu_compute_layer(L);
    }
    sample_output_token();
}

• Calculated Latencies Per Layer:
  • PCIe transmission latency: 875 MB / 50 GB/s = 17.5 ms
  • NPU computation latency: $140 \times 10^9$ ops / $400 \times 10^{12}$ ops/s = 0.35 ms

Select correct description(s).