Simulation Platform Based on SoC-FPGA Clusters for CXL-ethernet Heterogeneous Interconnection

doi:10.16182/j.issn1004731x.joss.25-0847

Abstract

Abstract:

To address the difficulty of balancing simulation speed and topology flexibility when simulating large-scale, CXL-Ethernet heterogeneous interconnect datacenter scenarios with existing CXL simulation platforms, this paper presents CNetSim, a semi-physical simulation platform based on an SoC-FPGA cluster. The platform employs SoC-FPGA hardware to emulate endpoint nodes, leveraging their real processors and memory devices as well as FPGA-implemented CXL. mem protocol to support high-speed execution of standard Linux systems and real distributed applications. Meanwhile, it utilizes the flexible topology configuration capability of software simulation to implement CXL switched networks and inter-rack Ethernet interconnects based on DPDK on simulation servers. Simulation experimental results demonstrate that, compared to traditional software simulators, the platform achieves at least three-orders-of-magnitude speedup when executing complete distributed applications, and accurately fits the performance characteristics of real systems by configuring key simulation parameters.

Key words: CXL, data center, memory semantic interconnect, SoC-FPGA, hardware-in-the-loop simulation

CLC Number:

TP338.4

Zhang Xu, Liu Ke, Chen Mingyu. Simulation Platform Based on SoC-FPGA Clusters for CXL-ethernet Heterogeneous Interconnection[J]. Journal of System Simulation, 2026, 38(7): 2037-2052.

Figures/Tables 13

Table 1

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 2

Fig. 8

Fig. 9

Table 3

Table 4

Execution time of real-world applications

应用程序	应用通信特征	输入	执行速度	单节点指令数	软件(Gem5)理论仿真速度
Linux启动			77 s		1 813 s
PageRank	带宽敏感型	soc-LiveJournal1	2.8 s	$1.9 × 1010$	$1.9 × 105$ s
ResNet18训练	带宽敏感型	cifar10	295 s/step	$1.4 × 1011$ /step	$1.4 × 106$ s/step
Redis	延迟敏感型	memtier_benchmark	80 kops	$4.3 × 108$ /s	$4.3 × 103$ s

Table 4

References 63

[1]	Malewicz G, Austern M H, Bik A J C, et al. Pregel: A System for Large-scale Graph Processing[C]//Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. New York: Association for Computing Machinery, 2010: 135-146.
[2]	Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters[J]. Communications of the ACM, 2008, 51(1): 107-113.
[3]	Liu Weibo, Wang Zidong, Liu Xiaohui, et al. A Survey of Deep Neural Network Architectures and Their Applications[J]. Neurocomputing, 2017, 234: 11-26.
[4]	Valiant L G. A Bridging Model for Parallel Computation[J]. Communications of the ACM, 1990, 33(8): 103-111.
[5]	Jia Zhihao, Kwon Y, Shipman Galen, et al. A Distributed Multi-GPU System for Fast Graph Processing[J]. Proceedings of the VLDB Endowment, 2017, 11(3): 297-310.
[6]	Jiang Yimin, Zhu Yibo, Lan Chang, et al. A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters[C]//Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. Berkeley: USENIX Association, 2020: 463-479.
[7]	Engelhardt Nina, Hayden Kwok-Hay So. GraVF: A Vertex-centric Distributed Graph Processing Framework on FPGAs[C]//2016 26th International Conference on Field Programmable Logic and Applications (FPL). Piscataway: IEEE, 2016: 1-4.
[8]	Li Huaicheng, Berger D S, Hsu L, et al. Pond: CXL-based Memory Pooling Systems for Cloud Platforms[C]//Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York: Association for Computing Machinery, 2023: 574-587.
[9]	Kwon Wonok, Park Chanho, Oh Myeonghoon. Gen-Z Memory Pool System Architecture[C]//2020 International Conference on Information and Communication Technology Convergence (ICTC). Piscataway: IEEE, 2020: 1356-1360.
[10]	Stuecheli J, Starke W J, Irish J D, et al. IBM POWER9 Opens up a New Era of Acceleration Enablement: OpenCAPI[J]. IBM Journal of Research and Development, 2018, 62(4/5): 8:1-8:8.
[11]	Tamimi Sajjad, Stock Florian, Koch Andreas, et al. An Evaluation of Using CCIX for Cache-coherent Host-FPGA Interfacing[C]//2022 IEEE 30th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). Piscataway: IEEE, 2022: 1-9.
[12]	Das Sharma D, Blankenship R, Berger D. An Introduction to the Compute Express Link (CXL) Interconnect[J]. ACM Computing Surveys, 2024, 56(11): 290.
[13]	Sharma D D. PCI Express® 6.0 Specification at 64.0 GT/s with PAM-4 Signaling: A Low Latency, High Bandwidth, High Reliability and Cost-effective Interconnect[C]//2020 IEEE Symposium on High-Performance Interconnects (HOTI). Piscataway: IEEE, 2020: 1-8.
[14]	Guo Chuanxiong, Wu Haitao, Deng Zhong, et al. RDMA Over Commodity Ethernet at Scale[C]//Proceedings of the 2016 ACM SIGCOMM Conference. New York: Association for Computing Machinery, 2016: 202-215.
[15]	Sankar R. Memory Tiering, CXL, and RDMA Networking: Blending Advanced I/O Technologies to Get Past the AI Memory Wall[EB/OL]. [2025-09-02]. .
[16]	Ma Teng, Zhang Mingxing. Revisiting Distributed Memory in the CXL Era[EB/OL]. (2024-01-09) [2025-09-02].
[17]	Rashinkar P, Paterson P, Singh L. System-on-a-chip Verification: Methodology and Techniques[M]. Boston, MA: Springer US, 2002.
[18]	Altera. What Is an SoC-FPGA?[EB/OL]. (2020-08-13) [2025-09-02]. .
[19]	Patel Harsh, Hiraskar Hrishikesh, Tahiliani Mohit P. Extending Network Emulation Support in Ns-3 Using DPDK[C]//Proceedings of the 2019 Workshop on Ns-3. New York: Association for Computing Machinery, 2019: 17-24.
[20]	Sasaki Kanon, Hirofuchi Takahiro, Yamaguchi Saneyasu, et al. An Accurate Packet Loss Emulation on a DPDK-based Network Emulator[C]//Proceedings of the 15th Asian Internet Engineering Conference. New York: Association for Computing Machinery, 2019: 1-8.
[21]	McVoy L W, Staelin C. Lmbench: Portable Tools for Performance Analysis[C]//USENIX Annual Technical Conference. Berkeley: USENIX Association, 1996: 279-294.
[22]	Tirumala A. Iperf: The TCP/UDP Bandwidth Measurement Tool[EB/OL]. [2025-09-02]. .
[23]	Schuh H N, Krishnamurthy A, Culler D, et al. CC-NIC: A Cache-coherent Interface to the NIC[C]//Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York: Association for Computing Machinery, 2024: 52-68.
[24]	Sano Shintaro, Bando Yosuke, Hiwada Kazuhiro, et al. GPU Graph Processing on CXL-based Microsecond-latency External Memory[C]//Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis. New York: Association for Computing Machinery, 2023: 962-972.
[25]	Abdullah R, Lee H, Zhou Huiyang, et al. Salus: Efficient Security Support for CXL-expanded GPU Memory[C]//2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). Piscataway: IEEE, 2024: 1-15.
[26]	Arif M, Maurya A, Rafique M M. Accelerating Performance of GPU-based Workloads Using CXL[C]//Proceedings of the 13th Workshop on AI and Scientific Computing at Scale Using Flexible Computing. New York: Association for Computing Machinery, 2023: 27-31.
[27]	Gouk Donghyun, Kang Seungkwan, Bae Hanyeoreum, et al. Breaking Barriers: Expanding GPU Memory with Sub-two Digit Nanosecond Latency CXL Controller[C]//Proceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems. New York: Association for Computing Machinery, 2024: 108-115.
[28]	Fridman Yehonatan, Suprasad Mutalik Desai, Singh Navneet, et al. CXL Memory as Persistent Memory for Disaggregated HPC: A Practical Approach[C]//Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis. New York: Association for Computing Machinery, 2023: 983-994.
[29]	Sun Yan, Yuan Yifan, Yu Zeduo, et al. Demystifying CXL Memory with Genuine CXL-ready Systems and Devices[C]//Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture. New York: Association for Computing Machinery, 2023: 105-121.
[30]	Cho A, Saxena A, Qureshi M, et al. A Case for CXL-centric Server Processors[EB/OL]. (2023-05-08) [2025-08-02]. .
[31]	Maruf H A, Wang Hao, Dhanotia A, et al. TPP: Transparent Page Placement for CXL-enabled Tiered-memory[C]//Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York: Association for Computing Machinery, 2023: 742-755.
[32]	Zhou Zhe, Xu Shuotao, Chen Yiqi, et al. Polaris: Enhancing CXL-based Memory Expanders with Memory-side Prefetching[C]//Advanced Parallel Processing Technologies. Singapore: Springer Nature Singapore, 2024: 19-39.
[33]	Yang Shaopeng, Kim Minjae, Nam Sanghyun, et al. Overcoming the Memory Wall with CXL-Enabled SSDs[C]//2023 USENIX Annual Technical Conference (USENIX ATC 23). Berkeley: USENIX Association, 2023: 601-617.
[34]	Zhou Zhe, Chen Yiqi, Zhang Tao, et al. NeoMem: Hardware/Software Co-design for CXL-native Memory Tiering[C]// Proceedings of the 57th Annual IEEE/ACM International Symposium on Microarchitecture. New York: Association for Computing Machinery, 2024: 1518-1531.
[35]	Kwon Miryeong, Lee Sangwon, Jung Myoungsoo. Cache in Hand: Expander-driven CXL Prefetcher for Next Generation CXL-SSD[C]//Proceedings of the 15th ACM Workshop on Hot Topics in Storage and File Systems. New York: Association for Computing Machinery, 2023: 24-30.
[36]	Jang Junhyeok, Choi Hanjin, Bae Hanyeoreum, et al. CXL-ANNS: Software-hardware Collaborative Memory Disaggregation and Computation for Billion-scale Approximate Nearest Neighbor Search[C]//2023 USENIX Annual Technical Conference (USENIX ATC 23). Berkeley: USENIX Association, 2023: 585-600.
[37]	Huangfu Wenqin, Malladi Krishna T, Chang Andrew, et al. BEACON: Scalable Near-data-processing Accelerators for Genome Analysis near Memory Pool with the CXL Support[C]//2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). Piscataway: IEEE, 2022: 727-743.
[38]	Huo Pingyi, Devulapally A, Maruf H A, et al. PIFS-rec: Process-in-fabric-switch for Large-scale Recommendation System Inferences[C]//2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). Piscataway: IEEE, 2024: 612-626.
[39]	Ha Minho, Ryu Junhee, Choi Jungmin, et al. Dynamic Capacity Service for Improving CXL Pooled Memory Efficiency[J]. IEEE Micro, 2023, 43(2): 39-47.
[40]	Lee Donghun, Willhalm Thomas, Ahn Minseon, et al. Elastic Use of Far Memory for In-memory Database Management Systems[C]//Proceedings of the 19th International Workshop on Data Management on New Hardware. New York: Association for Computing Machinery, 2023: 35-43.
[41]	Ahn Minseon, Willhalm Thomas, May Norman, et al. An Examination of CXL Memory Use Cases for In-memory Database Management Systems Using SAP HANA[J]. Proceedings of the VLDB Endowment, 2024, 17(12): 3827-3840.
[42]	Gouk Donghyun, Lee Sangwon, Kwon Miryeong, et al. Direct Access, High-performance Memory Disaggregation with DirectCXL[C]//2022 USENIX Annual Technical Conference (USENIX ATC 22). Berkeley: USENIX Association, 2022: 287-294.
[43]	Zhang Xu, Chang Yisong, Lu Tianyue, et al. Rethinking Design Paradigm of Graph Processing System with a CXL-like Memory Semantic Fabric[C]//2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid). Piscataway: IEEE, 2023: 25-35.
[44]	Miller E L, Benetopoulos A, Neville-Neil G V, et al. Pointers in Far Memory:A Rethink of How Data and Computations Should be Organized[J]. Queue, 2023, 21(3): 75-93.
[45]	Xu Yi, Mahar S, Liu Ziheng, et al. CXL Shared Memory Programming: Barely Distributed and Almost Persistent[EB/OL]. (2024-07-17) [2025-09-02]. .
[46]	Zhang Mingxing, Ma Teng, Hua Jinqi, et al. Partial Failure Resilient Memory Management System for (CXL-based) Distributed Shared Memory[C]//Proceedings of the 29th Symposium on Operating Systems Principles. New York: Association for Computing Machinery, 2023: 658-674.
[47]	Wang Chenjiu, He Ke, Fan Ruiqi, et al. CXL Over Ethernet: A Novel FPGA-based Memory Disaggregation Design in Data Centers[C]//2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). Piscataway: IEEE, 2023: 75-82.
[48]	Zhang Jie, Chen Xuzheng, Zhang Yin, et al. DmRPC: Disaggregated Memory-aware Datacenter RPC for Data-intensive Applications[C]//2024 IEEE 40th International Conference on Data Engineering (ICDE). Piscataway: IEEE, 2024: 3796-3809.
[49]	Wang Zhonghua, Guo Yixing, Lu Kai, et al. Rcmp: Reconstructing RDMA-based Memory Disaggregation via CXL[J]. ACM Transactions on Architecture and Code Optimization, 2024, 21(1): 15.
[50]	Tang Wenda, Han Ying, Ai Tianxiang, et al. Yggdrasil: Reducing Network I/O Tax with (CXL-based) Distributed Shared Memory[C]//Proceedings of the 53rd International Conference on Parallel Processing. New York: Association for Computing Machinery, 2024: 597-606.
[51]	Zhang Xu, Lu Tianyue, Chang Yisong, et al. Morpheus: An Adaptive DRAM Cache with Online Granularity Adjustment for Disaggregated Memory[C]//2023 IEEE 41st International Conference on Computer Design (ICCD). Piscataway: IEEE, 2023: 134-141.
[52]	Zhong Yijie, Zhou Minqiang, Shen Zhirong, et al. UniMem: Redesigning Disaggregated Memory Within a Unified Local-remote Memory Hierarchy[C]//Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference. Berkeley: USENIX Association, 2024: 463-477.
[53]	Calciu I, Imran M T, Puddu Ivan, et al. Rethinking Software Runtimes for Disaggregated Memory[C]//Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York: Association for Computing Machinery, 2021: 79-92.
[54]	Fang K, Peng D. NetDAM: Network Direct Attached Memory with Programmable In-memory Computing ISA[EB/OL]. (2021-10-28) [2025-09-02]. .
[55]	Zhong Yuhong, Berger D S, Waldspurger C, et al. Managing Memory Tiers with CXL in Virtualized Environments[C]//18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Berkeley: USENIX Association, 2024: 37-56.
[56]	Tang Yupeng, Zhou Ping, Zhang Wenhui, et al. Exploring Performance and Cost Optimization with ASIC-based CXL Memory[C]//Proceedings of the Nineteenth European Conference on Computer Systems. New York: Association for Computing Machinery, 2024: 818-833.
[57]	Puri Amit, Bellamkonda Kartheek, Narreddy Kailash, et al. DRackSim: Simulating CXL-enabled Large-scale Disaggregated Memory Systems[C]//Proceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation. New York: Association for Computing Machinery, 2024: 3-14.
[58]	Yang Yiwei, Safayenikoo P, Ma Jiacheng, et al. CXLMemSim: A Pure Software Simulated CXL.Mem for Performance Characterization[EB/OL]. (2023-03-10) [2025-09-02]. .
[59]	Wang Yanjing, Wu Lizhou, Hong Wentao, et al. CXL-DMSim: A Full-system CXL Disaggregated Memory Simulator with Comprehensive Silicon Validation[J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2026, 45(4): 1787-1801.
[60]	Boles D, Waddington D, Roberts D A. CXL-enabled Enhanced Memory Functions[J]. IEEE Micro, 2023, 43(2): 58-65.
[61]	Umeike J, Patel N, Manley A, et al. Profiling Gem5 Simulator[C]//2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). Piscataway: IEEE, 2023: 103-113.
[62]	齐乐, 常轶松, 陈欲晓, 等. 基于SoC-FPGA的RISC-V处理器软硬件系统级平台[J]. 计算机研究与发展, 2023, 60(6): 1204-1215.
	Qi Le, Chang Yisong, Chen Yuxiao, et al. A System-level Platform with SoC-FPGA for RISC-V Hardware-software Integration[J]. Journal of Computer Research and Development, 2023, 60(6): 1204-1215.
[63]	JEDEC Solid State Technology Association. JEDEC Memory Module Reference Base Standard-for Compute Express Link (CXL)[EB/OL]. [2025-03-01]. .

名称	多种互连结构	运行速度	延迟、带宽参数可配置	支持结构仿真	贴近真实部署场景
NUMA系统	×	√	×	×	×
软件仿真	√	×	√	√	×
FPGA硬件仿真	×	√	×	√	√
CNetSim	√	√	√	√	√

名称	值	比例/%
LUT	59 009	11.3
Register	85 895	8.2
BRAM	581	59.0
URAM	0	0

R值	理论延迟/ns	实际延迟/ns	误差/%
2.7	4 629	4 906	6.0
3.0	4 166	4 424	6.2
4.0	3 125	3 368	7.8
5.0	2 500	2 710	8.4
6.0	2 083	2 285	9.7
6.4	1 953	2 171	11.2