VCS simulation speed-up in practice: 445s → 33s lever order and the -fgp verdict

Bottom line

Profile first, then four levers in order — vendor-primitive models, debug options, time precision, process concurrency — took a 7.5-minute run to 1 minute; thread-level -fgp was a net loss.

1. Measure before touching any option

Time compile and run separately: on this class of design compile is only 20~25s and everything else is simulation, so treating compile as the bottleneck wastes the effort.

Get the hot spots with -simprofile (after the run, read the Module / Construct View in profileReport.txt):

• hot spot in the vendor compiled primitive models (xil_*_vl_mod*, B_OSERDESE2 and other secureip) → replace them with behavioural models, an order-of-magnitude gain;
• hot spot in a self-triggering always, a chained assign #(real) delay line or a clock tree → that part is inherently serial, no thread-level parallelism can help.

Compare CPU Time only (or /usr/bin/time user+sys), never wall clock: wall is a function of machine load.

2. Lever 1: behavioural fast models for vendor primitives (the biggest one)

76.9% of the time in the design under test sat in 10 OSERDESE2 secureip instances. Replacing unisims + secureip with behavioural RTL — same interfaces and parameters, no internal delays, no timing checks — took 444.8s to 33.4s.

Key points:

• Files to replace: glbl, IBUFDS/OBUFDS/BUFG/BUFIO/BUFR, PLLE2_ADV, OSERDESE2, ISERDESE2; also drop the four secureip/*.vp entries from the file list.
• PLL: TVCO = CLKIN1_PERIOD * DIVCLK_DIVIDE / CLKFBOUT_MULT; use the output with the smallest DIVIDE as the time base and derive every other CLKOUT from it by integer division. That keeps 70MHz:490MHz = 7:1 exact and drift-free, which is the precondition for OSERDES 7:1 to work. The divider must count whole periods (always @(posedge base)); counting both edges (half periods) puts the 70MHz edges on the falling edges of 490MHz, half a VCO period off, and the LVDS serial word boundary shifts with it.
• PLL LOCKED must be calibrated against the real model (example: 753 input periods after reset release). Designs that use pll_locked as their reset change behaviour with the lock time, so do not tweak it casually.
• OSERDESE2 / ISERDESE2: dpar[0]=D1 is the bit sent first; the word boundary must be synchronised to CLKDIV — do not let a self-counting edge index run free, its initial relative phase is undefined and data that changes every cycle comes out shifted by a whole word.
• Do not pretend to support DATA_WIDTH>8 (10/14 cascade); print an explicit initial $display warning.

Verification has three layers: primitive-level side-by-side (fixed data for phase, changing data for bit order) → whole-design sim.log identical line by line after stripping timestamps, with the same $finish time → align the reference model’s output sequence and compare the match rate. fsdbsanitycheck to confirm the waveform file is complete.

3. Levers 2/3: compile options and time precision

4. Lever 4: process-level concurrency is the real parallelism

A single simv is single-threaded, so throughput only comes from running cases concurrently. Use a -P N regression driver: one directory per case with its own file list / logs / waveform, then a PASS/FAIL summary table plus the total wall time.

8 cases (22s compile + 32s simulation each) with -P 8: total wall 55s against ~440s serial, near-linear.

Two things easy to miss in the driver: a pipe-separated case list (name|flist|lib|extra compile opts|extra run opts) is the least painful format; and the summary must only cover the cases of the current list (scanning run/* silently includes every historical case).

5. Why -fgp did not accelerate anything (and when it does)

Syntax (measured on VCS 2023.12): at compile time -fgp is just a switch and -fgp=4 is ignored with a warning; the thread count can only be given at run time as simv -fgp=num_threads:N; a bare -fgp at run time errors with FGP_REQ_NUM_THREAD; -fgp=diag:exclreason exists only in ENGINEER builds and the release build answers UNSUP-FGPOPT.

Measured (same case, same compile options, only the thread count changed):

More threads is always slower, and it is not a tuning mistake:

• process CPU utilisation stayed at 103~115% even with 32 threads, i.e. at most 1.15 cores doing work;
• -fgp=hier (a different partitioning strategy) moved it by single-digit percent;
• with the FSDB dump disabled (the only definitely single-threaded part, the PLI callbacks) the fast model ran 18.0s single-threaded and 30.2s with 8 threads (+68% CPU) — still a loss, so the root cause is available parallelism, not the PLI;
• the Construct View structure: 13.8% is the PLL’s single self-triggering oscillator always, 12.4% the OSERDES serialiser block (10 instances × ~1.2%, tens of nanoseconds per event), 7.8% the 7-stage chained assign #(dly) delay line in the testbench, 21.3% the kernel scheduler.

In other words, the numerator (independent work available in the same time slot) is tiny while the denominator (per-sync overhead) is large.

Where -fgp does help: many mutually independent activities in the same time slot (multi-subsystem SoC where CPU cluster, DDR, ISP, DMA and multi-lane PHY all move traffic); heavy per-event computation (gate-level netlists, large random memory access, algorithmic models) so the sync cost amortises; few events crossing partition boundaries (multiple clock domains with little cross-domain traffic); and waveform/PLI not being the bottleneck. Published experience is roughly 1.3~2.5×, rarely above 3×.

6. Three traps

1. A dirty control group: I once credited the 21%+15% from -debug_access+pp +rad to -fgp and concluded “FGP gives 1.47×” — the opposite of the truth. For any parallel/option conclusion the two runs must differ in exactly one variable.
2. The dump fraction moves: on the original models the dump is only ~5%, on the fast models ~45% (33.4s vs 18.0s). Re-measure after accelerating, or you will undervalue trimming the dump scope.
3. A macro switch in the testbench can flip results: with the same testbench, merely changing which instance $fsdbSuppress covers changed the PLL lock time and the I2C read-back value (reproducible 2/2). When results change after swapping a model or an option, run this control before assigning a cause.

7. Reproduction checklist

• Timing: locate hot spots with -simprofile, compare CPU Time;
• Model swap: replace only the primitive files in the same file list, diff both sim.log (timestamps stripped) — must be line-identical;
• Precision change: hang @(posedge {frame_pulse}) $display("TRACE %t", $time) in the testbench, compare absolute times and convert to ppm;
• Regression: -P N driver + one directory per case + PASS/FAIL summary (pass = no compile Error and the run reaches $finish).