- Example PTP solutions
3.1. PTP timing in a new substation automation system
3.2. Replacing an IRIG-B distribution system with PTP
3.3. Network design to support redundancy and PTP
PTP time synchronisation within a substation using a combination of stand-alone slave clocks (PTP translators) and protection relays with built-in support for PTP
conventional time synchronisation with AM and unmodulated IRIG-B time code signals. Ethernet connections to each relay are used for control purposes, however this could be DNP3 or IEC 60870-5-101 over RS485 in an older automation design
General arrangement of extended substation with an extra transformer, switchgear and control building
General arrangement of 330/132 kV substation using conventional time synchronisation equipment
Single line diagram of a 330/132 kV substation with a “breaker and a half” 330 kV switchyard and a folded bus 132 kV switchyard
Network topology for 330/132 kV substation with a central communications building and two protection/control buildings
PTP network traffic transmitted by a two step grandmaster clock
The primary source of time is the Grandmaster Clock with a satellite receiver. It is recommended that the PTP grandmaster clock is also an NTP master clock, as NTP can be used by automation servers, SCADA gateways, energy meters and protection relays that require millisecond-level accuracy.
equipment are installed in separate buildings.
Redundant connections are not used, and only one protection scheme is shown.
GE “UR” series relays have built-in PMU functions and support PTP for high accuracy time synchronisation that is required (rather than requiring IRIG-B as most protection relays do).
Ethernet switches are used to distribute PTP message throughout the substation, along with IEC 61850, DNP3, HTTP, SNMP and any other protocols that are used. PTP traffic is so low in volume, approximately 420 bytes per second, that there is no impact on the rest of the network.
This two step mode of operation creates the most PTP network traffic, and so is the worst case.
Network Design
Balance
flexibility ✅more switches, more ports
reliability ⚠ single of failure
robustness ⚠ if a switch fails, how many items of HV plant will you lose control of?
TC 🆚 BC LAN_Switch
TC - ✅ simple, work with Wireshark ❌Consider the the case of a network failure between the root switch and the 132 kV switch. If the 132 kV switch was a transparent clock, each slave device (the protection relays) will “drift” away from the true time, and from each other, due to unavoidable manufacturing variations in their internal oscillators. The rate of drift depends upon a range of factors, including the quality of the oscillator and local temperature changes. If the outage is prolonged then the clock error between the individual 132 kV protection relays may become significant. This is similar to the situation where an IRIG-B cable was broken in a conventional timing system.
BC ✅ provide a degree of isolation between the upstream grandmaster and downstream slave clocks. This is because they maintain a true clock internally, rather than just estimating residence times. ⭐ If the network connection to the grandmaster is lost the protection relays will remain synchronised to the boundary clock. The local time in the boundary clock will slowly drift away from the grandmaster, and hence the slave clocks will drift too—but at exactly the same rate. The quality of the internal clock references in the protection relays is now less important, as it is only the internal oscillator in the boundary clock that determines the drift rate.
The “Root Switch” is the centre of the substation Ethernet network. This is where “whole of substation” services, such as SCADA gateways (to control centres), operator interfaces (HMIs), security systems and engineering workstations connect to the LAN. In this design there are two additional switches, one each for the 330 kV and 132 kV protection equipment. This reduces the number of Ethernet cables required to communicate with protection relays. Local switches in each control building enables horizontal communication between protection relays (e.g. GOOSE messages for bus zone tripping and CB fail initiation) to remain in service if the network link to the central communications building fails.
The existing substation uses Ethernet for communication with protection relays and uses IRIG-B time codes to synchronise the clocks of the protection relays. Fibre-optic cable is used for both Ethernet and IRIG-B as this provides the best immunity to interference and the safety of galvanic isolation. Isolated Timing Repeaters (ITRs) are used to convert the optical IRIG-B time code signals back to electrical forms that can be used by the protection relays.
The utility has a project in place to add three more “diameters” of breaker-and-a-half switchgear to the 330 kV switchyard, along with an additional 330/132 kV transformer. Another control building will be installed to house the protection relays and other control equipment. While it would be possible to loop the IRIG-B signal from 132 kV Control Building 1, the total path is long and introduces time error due to propagation delay. This “brownfield” expansion is an opportunity to gain experience with PTP.
Very little equipment needs to be replaced. If the GPS master clock cannot support PTP then it must be replaced. The Tekron TCG 01-G selected for this example supports all existing time codes as well as PTP and NTP. If the main Ethernet switch (the “root” switch) does not support the Power Profile then it must be replaced with one that does, such as the GE MultiLink ML3000. The configuration of the old switch should be documented so all VLAN and multicast filter definitions, port configurations and SNMP monitoring settings can be replicated.
The final step is to use a PTP Translator in the new control building rather than an Isolated Timing Repeater (ITR). This converts the PTP signal back to IRIG-B (modulated and/or unmodulated), allowing the standard protection design to be used for the expansion. Any Ethernet switches installed in the new control building need to be Power Profile transparent clocks or boundary clocks. Figure 13 shows the layout of the upgraded substation. It is worth checking to see if the protection relays used in the utility’s standard design have been updated by the manufacturer to support PTP. This provides another opportunity to gain experience with PTP without changing tried and tested protection designs.
Adopting PTP for brownfield developments gives utilities and system integrators the opportunity to gain experience with PTP in a gradual manner. Having PTP infrastructure in place then provides a test-bed for to evaluate new and revised protection relays that have native support for PTP.
If a utility is moving to Ethernet in the substation for the first time then it is prudent to investigate the use of Ethernet switches that support the PTP and the Power Profile. Revisions to protocols can be made through firmware updates in the future, but these are dependent upon having PTP hardware support in the first place.
The use of PTP for time synchronisation within a substation then allows inter-panel communications to use fibre-optic cable. PTP slave clocks, such as Tekron’s PTP Translator, are used to generate conventional time codes at each panel. Local generation of IRIG-B time codes means that each panel can have a different format or time zone, giving greater flexibility that is currently possible with a single IRIG-B source. Figure 15 shows how PTP can be used to distribute time to existing protection relays with translators and to upgraded protection relays with native support for PTP.
No compensation of propagation delay is needed for devices in the new control building as this is taken care of automatically by the peer-delay mechanism specified by the Power Profile. This simplifies the configuration and commissioning of PMUs and other applications that need microsecond-level accuracy.
A refinement to the panel design might be to install a PTP translator in each panel, rather than having inter-panel IRIG-B cabling. Many utilities have adopted the practice of eliminating metallic communications cables between panels, and this can be achieved through the use of PTP over fibre-optic Ethernet—the same Ethernet that is being used to communicate with protection relays.
The Fundamental principles
The failure of any device or network link does not result in the loss of control of more than one bay of HV switchgear.
Fully redundant duplicate protection is used, often referred to as Main1/Main2, A/B or X/Y protection.
Switchgear is controlled by one of the protection relays and not through dedicated bay controllers.
Adv & Disadv.
Rapid Spanning Tree Protocol (RSTP) ring or mesh networks. Supported by most, if not all, substation Ethernet switches. The time required for the network to recover following an outage is not defined. Networks can take some time to stabilise, especially if there are meshes rather than rings.
Parallel Redundancy Protocol (PRP) duplicated networks. No loss of data for a single link or switch failure, and straightforward design. Requires specific support or the use of a “redundancy box” (also called a “redbox”), and an increased number of switches are required.
High-reliability Seamless Redundancy (HSR) ring networks. No loss of data for a single link or switch failure, and avoids need for extra switches. Is limited to a ring topology and requires specific support by connected devices (e.g. PTP clocks and protection relays) or the use of a redbox to connect non-HSR devices into the ring.
Redundant network architecture using PRP for network control, with duplicate protection systems
The example presented in this section uses PRP and avoids “bay switches” or “diameter switches” often used to limit the loss of control following network failure. In some situations PRP can reduce the number of Ethernet switches required compared to an RSTP-based design.
The “X” (this may be called “Main 1” by some utilities) protection is implemented using the GE UR series of protection relays, as these support PTP, PRP and local control of switchgear. The X protection will provide control and phasor monitoring functions in addition to protection. The “Y” (or “Main 2”) protection is implemented using relays from other vendors that support PTP or NTP for time synchronisation.
Figure 16 shows the network topology. The two parallel networks in PRP are referred to as “A” and “B” paths, both of which are active at all times. RSTP works by blocking backup links to avoid circulating messages, and these are shown a dashed line between Root Switch 2 and the Y Switch. Some SCADA gateway computers use “fail over” networking, where a second Ethernet port is kept disabled unless the primary link fails. These standby links are shown as dashed lines too.
It is expected that substation gateways will eventually support PRP natively, allowing both links to remain active at all times. Similarly, Switch Y could provide “Redbox” (redundancy box) functionality for the Y protection relays, taking care of de-duplication of frames.
Substation rated Ethernet switches are now available with high port density, avoiding the need for “breakout” switches in relay panels. In smaller substations the protection switches (X1, X2 and Y in the figure above) may not be required, and conversely in large substations it may be useful to have a set of X1, X2 and Y switches for each voltage level. Regardless of the topology, the use of Ethernet switches with PTP transparent or boundary clock capability will allow PTP clients to be connected at any location in the network.