# VCS simulation speed-up in practice: 445s → 33s lever order and the -fgp verdict

## Bottom line

Profile first, then four levers in order — vendor-primitive models, debug options, time precision, process concurrency — took a 7.5-minute run to 1 minute; thread-level `-fgp` was a net loss.

| Lever | Single-run CPU | vs baseline | Cost |
|---|---|---|---|
| Baseline (`-debug_access+all`, vendor compiled models) | 444.8s | — | — |
| `-debug_access+pp` instead of `+all` | 341.6s | -23% | no interactive force/step |
| plus `+rad` | 308.0s | -31% | none |
| vendor primitives replaced by behavioural fast models | 33.4s | **-92% (13.3×)** | time base -0.04%, residual bit-boundary mismatch in SERDES |
| plus `-override_timescale=1ns/10ps` | 17~22s | -95% | time axis compressed by ~490ppm |
| 8 cases with `-P 8` | total wall 55s (serial ~440s) | throughput ~8× | one directory per case |

## 1. Measure before touching any option

Time compile and run separately: on this class of design compile is only 20~25s and everything else is simulation, so treating compile as the bottleneck wastes the effort.

Get the hot spots with `-simprofile` (after the run, read the Module / Construct View in `profileReport.txt`):

- hot spot in the vendor compiled primitive models (`xil_*_vl_mod*`, `B_OSERDESE2` and other secureip) → replace them with behavioural models, an order-of-magnitude gain;
- hot spot in a self-triggering `always`, a chained `assign #(real)` delay line or a clock tree → that part is inherently serial, no thread-level parallelism can help.

Compare `CPU Time` only (or `/usr/bin/time` user+sys), never wall clock: wall is a function of machine load.

## 2. Lever 1: behavioural fast models for vendor primitives (the biggest one)

76.9% of the time in the design under test sat in 10 OSERDESE2 secureip instances. Replacing unisims + secureip with behavioural RTL — same interfaces and parameters, no internal delays, no timing checks — took 444.8s to 33.4s.

Key points:

- Files to replace: `glbl`, `IBUFDS/OBUFDS/BUFG/BUFIO/BUFR`, `PLLE2_ADV`, `OSERDESE2`, `ISERDESE2`; also drop the four `secureip/*.vp` entries from the file list.
- PLL: `TVCO = CLKIN1_PERIOD * DIVCLK_DIVIDE / CLKFBOUT_MULT`; use the output with the smallest DIVIDE as the time base and derive every other CLKOUT from it by **integer division**. That keeps 70MHz:490MHz = 7:1 exact and drift-free, which is the precondition for OSERDES 7:1 to work. The divider must count **whole periods** (`always @(posedge base)`); counting both edges (half periods) puts the 70MHz edges on the falling edges of 490MHz, half a VCO period off, and the LVDS serial word boundary shifts with it.
- PLL `LOCKED` must be calibrated against the real model (example: 753 input periods after reset release). Designs that use `pll_locked` as their reset change behaviour with the lock time, so do not tweak it casually.
- OSERDESE2 / ISERDESE2: `dpar[0]=D1` is the bit sent first; **the word boundary must be synchronised to CLKDIV** — do not let a self-counting edge index run free, its initial relative phase is undefined and data that changes every cycle comes out shifted by a whole word.
- Do not pretend to support `DATA_WIDTH>8` (10/14 cascade); print an explicit `initial $display` warning.

Verification has three layers: primitive-level side-by-side (fixed data for phase, changing data for bit order) → whole-design `sim.log` identical line by line after stripping timestamps, with the same `$finish` time → align the reference model's output sequence and compare the match rate. `fsdbsanitycheck` to confirm the waveform file is complete.

## 3. Levers 2/3: compile options and time precision

| Option | Effect | Note |
|---|---|---|
| `-debug_access+pp` instead of `+all` | -23% | FSDB/KDB still work, only interactive debug is lost |
| `+rad` | -15% | no side effects |
| `-override_timescale=1ns/10ps` | a further -44% | 490MHz period 2.041→2.040ns, about -490ppm, ~9.8µs accumulated over 23ms |
| dropping `-debug_access` entirely | slower | `$fsdb*` become Undefined System Task; an explicit `-P novas.tab pli.a` compiles but is slower than `+pp` |
| `+nospecify +notimingcheck` | **6% slower** | removing path delays increases the number of toggling events |
| `-j N` / `-partcomp -j N` | compile gets slower | the design is too small, parallel overhead dominates |

## 4. Lever 4: process-level concurrency is the real parallelism

A single `simv` is single-threaded, so throughput only comes from running cases concurrently. Use a `-P N` regression driver: one directory per case with its own file list / logs / waveform, then a PASS/FAIL summary table plus the total wall time.

8 cases (22s compile + 32s simulation each) with `-P 8`: **total wall 55s** against ~440s serial, near-linear.

Two things easy to miss in the driver: a pipe-separated case list (`name|flist|lib|extra compile opts|extra run opts`) is the least painful format; and the summary must only cover the cases of the current list (scanning `run/*` silently includes every historical case).

## 5. Why `-fgp` did not accelerate anything (and when it does)

Syntax (measured on VCS 2023.12): at compile time `-fgp` is just a switch and `-fgp=4` is ignored with a warning; the thread count can only be given at run time as `simv -fgp=num_threads:N`; a bare `-fgp` at run time errors with `FGP_REQ_NUM_THREAD`; `-fgp=diag:exclreason` exists only in ENGINEER builds and the release build answers `UNSUP-FGPOPT`.

Measured (same case, same compile options, only the thread count changed):

| Config | Original-model CPU | Fast-model CPU | Threads observed |
|---|---|---|---|
| No FGP | 296.3s | 33.4s | 1 |
| `-fgp` (default threads) | 318.2s (+7%) | 36.1s (+8%) | 3 |
| `num_threads:3` | 361.0s (+22%) | 57.3s (+72%) | 13 |
| `num_threads:16` | 384.1s (+30%) | 72.8s (+118%) | 26 |
| `num_threads:32` | 408.5s (+38%) | 99.3s (+198%) | 42 |

More threads is always slower, and it is not a tuning mistake:

- process CPU utilisation stayed at 103~115% even with 32 threads, i.e. at most 1.15 cores doing work;
- `-fgp=hier` (a different partitioning strategy) moved it by single-digit percent;
- with the FSDB dump disabled (the only definitely single-threaded part, the PLI callbacks) the fast model ran 18.0s single-threaded and 30.2s with 8 threads (+68% CPU) — still a loss, so the root cause is available parallelism, not the PLI;
- the Construct View structure: 13.8% is the PLL's single self-triggering oscillator `always`, 12.4% the OSERDES serialiser block (10 instances × ~1.2%, tens of nanoseconds per event), 7.8% the 7-stage chained `assign #(dly)` delay line in the testbench, 21.3% the kernel scheduler.

In other words, the numerator (independent work available in the same time slot) is tiny while the denominator (per-sync overhead) is large.

Where `-fgp` does help: many mutually independent activities in the same time slot (multi-subsystem SoC where CPU cluster, DDR, ISP, DMA and multi-lane PHY all move traffic); heavy per-event computation (gate-level netlists, large random memory access, algorithmic models) so the sync cost amortises; few events crossing partition boundaries (multiple clock domains with little cross-domain traffic); and waveform/PLI not being the bottleneck. Published experience is roughly 1.3~2.5×, rarely above 3×.

## 6. Three traps

1. **A dirty control group**: I once credited the 21%+15% from `-debug_access+pp +rad` to `-fgp` and concluded "FGP gives 1.47×" — the opposite of the truth. For any parallel/option conclusion the two runs must differ in exactly one variable.
2. **The dump fraction moves**: on the original models the dump is only ~5%, on the fast models ~45% (33.4s vs 18.0s). Re-measure after accelerating, or you will undervalue trimming the dump scope.
3. **A macro switch in the testbench can flip results**: with the same testbench, merely changing which instance `$fsdbSuppress` covers changed the PLL lock time and the I2C read-back value (reproducible 2/2). When results change after swapping a model or an option, run this control before assigning a cause.

## 7. Reproduction checklist

- Timing: locate hot spots with `-simprofile`, compare `CPU Time`;
- Model swap: replace only the primitive files in the same file list, `diff` both `sim.log` (timestamps stripped) — must be line-identical;
- Precision change: hang `@(posedge {frame_pulse}) $display("TRACE %t", $time)` in the testbench, compare absolute times and convert to ppm;
- Regression: `-P N` driver + one directory per case + PASS/FAIL summary (pass = no compile Error and the run reaches `$finish`).