MEN has developed a new VME board with Intel Xeon D processor for CERN. Thanks to an open source FPGA-based PCIe-VME64x bridge, existing particle accelerator equipment can now be upgraded with a more current processor and remain operational for many years. The plan is to use VME until the LHC's scheduled end in 2032.
CERN is the world's largest research center in the field of particle physics, with foundations dating back to 1954. Today, more than 2.500 employees and more than 12.000 visiting scientists from 85 nations are investigating the building blocks there. of our Universe. Probably the best known is the 27 kilometer long Large Hadron Collider (LHC), which came online in 2008. One of the questions it helps to examine is why our Universe consists mostly of matter and not equally as well. of antimatter. The most advanced equipment is used for this and many other tasks, and huge investments are made year after year. It is also very important to maintain the existing infrastructure for years to come.
Flexible devices for data acquisition and accelerator control
Take, for example, the thousands of so-called boxes that have been installed in the many particle accelerators at CERN over the years. Deployed in the supporting infrastructure of various particle detectors, they are typically used for trigger electronics and data acquisition. The boxes have a typical configuration based on slots with backplane and freely configurable modules. This type of modular electronic system is predestined for use in scientific institutions such as CERN, since such modular systems allow individual circuits to be reused many times and implemented in multiple systems and in various configurations. Once an experiment is finished, the boxes are reused for new experiments in different configurations, securing the initial investment in the long run. One type of these boxes is based on the VME bus, which was first specified in 1981 and has been continuously developed ever since. Currently, more than 900 such boxes are in use at CERN itself, mainly to control accelerators. Boxes in different configurations are used for data acquisition in the experiments and detectors. In the LHCb experiment, for example, they are used to pre-process parts of the native data from about a million sensors so that scientists receive only the relevant data for their analyses. Other boxes are found in many other CERN detectors such as ATLAS, CMS, ALICE, ISOLDE and TOTEM, in some cases they fulfill completely different tasks as, like any other modular backplane system, they can be used extremely flexibly.
New devices for new challenges
Since the tasks change with each experiment, new box configurations are constantly being developed that must provide the ultimate in computing performance. In 2016 alone, and on the accelerators, about 50 new boxes were put into operation. In addition, it is already foreseeable that around 200 new boxes will be installed during the planned “Long Shutdown” from 2019 to 2020, which will be used to completely repair and overhaul the equipment. However, one problem with VME-based systems lies in the fact that the processors do not natively support communication over the VME bus. Therefore, processor boards must provide a PCIe-VME64x bridge to interface with the VME bus. However, only a few manufacturers had discrete components available, while the main vendor had announced end of life for the current component (TSI148). The scale of this problem becomes clear quickly when you consider the number of single board computers (SBCs) with the VME bus installed at CERN: today, more than 900 VME-based SBCs from MEN Mikro Elektronik with Intel® Core™ Duo are installed. and Core™ 2 Duo. While these are significant amounts for high-quality VME boards, this volume alone would not justify manufacturing a discrete component just for this purpose. Therefore, CERN's Beams Department/Control Group, BE/CO for short, was looking for sustainable alternatives for the next few years by issuing a new tender.
In search of a new solution for the VME64x bus
Three possible options have been specified for communication from PCI/PCIe to VME64x. have a sufficient stock of TSI148 chips to be able to produce the number of boards specified in the contract, or use the Tundra Universe II, the predecessor of the TSI148, or use FPGA technology: in that case, CERN asked the bidders to make available Full VHDL sources for FPGA design available through a GPL3 or later license. In addition to the first two options, CERN knew that there were companies with proprietary implementations of VME bridges made on FPGAs. For example, previous generations of SBCs used at CERN (before TSI148-based boards) had a PowerPC processor with an FPGA to connect to a VME bus. Therefore, with the last option in the call for tenders, CERN hoped that at least one of these companies would be ready for their open source implementation. On the other hand, to ensure fair conditions for anyone who submits their offers, CERN did not give preference to any of these options. The selection was based on the price of the offers presented in the tender. Ultimately, the company offering the best prices was chosen to be awarded the contract.
The open source PCIe-VME64x bridge is now available
The result of that tender leads to a solution based on FPGA technology, and therefore all VHDL sources are now available under the GPL3 license or later, and the Linux driver package under the GPL2 license or later on the page from the PCIe-to-VME bridge project from the Open Hardware Repository. The open source PCIe-to-VME bridge is a big step not only for CERN, but also for all other institutions around the world where VME is still in use. First, CERN engineers are no longer dependent on a particular vendor. Even if the currently used FPGA chip becomes obsolete, access to full VHDL sources allows you to port the PCIe to VME bridge to another FPGA. Thanks to the fact that the design is open source, any institute or company can now not only buy an upgraded product with that bridge, but also build any other VME SBC using the same bridge. Using the same VME bridge in SBC also means the same Linux kernel drivers and user VME API for all institutes and enterprises. In the future, this should allow all engineers to collaborate more efficiently in the VME world and have more freedom to share and reuse Linux kernel drivers for VME boards that, for example, CERN engineers design on their own.
New Logic FPGA Contributor
The company that was open to investing a lot of effort together with CERN to design, test and validate a suitable PCIe Gen 3 bridge for VME's 64-bit systems was MEN Mikro Elektronik. The company not only worked together with the CERN team on validation and testing, but also on open source, publishing the PCIe to VME bridge that translates read and write operations in the PCIe address space to read and write transactions to the VME bus. It acts as a PCIe endpoint on one side and a VME master bus on the other. The bridge can generate unique VME cycles and block transfers.
The following types of access are currently supported: VME single cycles: A16, A24, A32 with any of the data widths D8, D16, D32 VME Block Transfers (BLT): A24D16, A24D32, A32D32 plus multiplexed block transfer (MBLT) A24D64 and A32D64 Configuration Space Access CR/CSR VME block transfers are executed using a direct memory access (DMA) engine, where data blocks are transferred between system memory and the VME bus, without going through the CPU. In addition, it is also possible to use DMA with individual cycles, which is especially useful for connected boards that do not support the BLT access mode. In general, this is a faster and more efficient way of exchanging multiple words of data, since the CPU is free to continue its normal operation until the DMA engine finishes with a scheduled task. The bridge also supports some features added in the VME64x extensions.
You can use the geo-address pins and generate a special type of A24 access to read and write the CR/CSR configuration space of VME slaves installed on the same box. However, while none of the fast transfer modes (2eVME, 2eSST) are currently supported, these could be implemented in the future as upcoming VME slave cards may require them; This topic is already being evaluated in MEN. In addition, the VME bus module implemented by MEN can act as both a VME master and a VME slave. This allows not only their use in VME SBCs running as masters, but also to take advantage of them in I/O and other peripheral cards connected as slaves. Although for the SBC VME application, the configuration is focused only on the master VME functionality. Currently, the entire bridge design occupies only 30 percent of the FPGA area of the Intel Cyclone. This means that there is a lot of space available to implement additional new features (such as 2eVME, 2eSST transfers).
Guaranteed long-term availability
By releasing the specification along with the deployment of the first boards with the new FPGA-based PCIe to VME64x bridge, CERN has reached an important milestone for the long-term availability of its VME-based enclosures for data acquisition and control. Of the accelerator. The reference is also a milestone for all other existing users of VME-based systems, because the availability of the appropriate logic is now also guaranteed for them in the long term. According to current estimates, the market for the new boards will still amount to more than USD 200 million in 2020. During the project, the embedded computing specialist MEN Mikro Elektronik once again demonstrated its great expertise in FPGA technology and standardized embedded computing. together with knowledge of CPU VME. With the PCIe-to-VME bridge, customers benefit from long-term availability of existing installations. MEN Mikro Elektronik can also offer customized solutions to each type of customer, for example PCIe to PCI or even PCIe to ISA, which also make legacy hardware from OEMs available for the long term, providing a return on investment. even longer. Along with bridging solutions for internal legacy buses to ensure long-term availability, the company also offers FPGA-based bridging to external interfaces and buses, such as UART, CAN bus, or QSPI controllers using SPI.
This scenario allows OEM customers to create extremely cost-effective shape variants. For example, a single CPU board design can be used for completely different applications. Even when batch sizes are small, it is possible to serve significantly more applications, such as solutions with different fieldbuses or Industrial Ethernet variants, or, specifically, solutions with migration requirements, for example, in railway engineering or aircraft construction. . OEMs can use a single hardware platform across all variants, significantly simplifying service, documentation, and certification.
The first plate with the new bridge
The first board with the new FPGA-based bridge for the VME bus used at CERN is the A25 from MEN Mikro Elektronik. It is equipped with Intel's Xeon D-1500 CPU and, in addition to the new FPGA, combines high cost efficiency with a rich feature set. In addition, the A25 supports system size reduction, long-term reliable operation without forced-air cooling, and multiple computing functions on a single board. With two USB 3.0 ports, up to three Gigabit Ethernet ports, and two RS232 COM ports on the front, the board offers the basics of a multi-purpose industrial computer.
Equipped with up to 8 GByte of DDR4 SDRAM with ECC accompanied by Flash, the need for flexible mass storage expansions is covered by microSD and mSATA slots. In addition, the A25 can be equipped with an XMC/PMC mezzanine card and a PCI Express mini card, which provide additional front I/O (XMC/PMC) for features such as graphics, mass storage, or additional Ethernet ports. The PMC slot supports PCI-X modules up to 64-bit / 133 MHz, while the XMC slot is controlled by a PCI Express x8 link. Modular expansion with I/O mezzanines on a single-seat computer allows custom systems to be configured from open standard components, reducing integration time and cost. The A25 supports operation in a temperature range from -40°C to +60°C. The robust board withstands shock and vibration as all board components are welded, which is a prerequisite for reliable operation and a longer product life.
