14.06.2019
On Friday 7 June, EPEX SPOT experienced a technical issue causing a chain of events that eventually led to decoupling of EPEX SPOT markets, cancellation of an erroneous set of local auction market results and the publication of new and final local results.
The chain of events can be divided into two parts: one concerning the decoupling incident, the other concerning an incident on the local auction. The root cause at the beginning of the chain of events was a corrupt order entered into the ETS system unintentionally.
DECOUPLING OF EPEX SPOT MARKETS
From 11h39 onwards, EPEX SPOT experienced a technical issue with its ETS system, initiated by the corrupt order which caused an ETS server lock. Once the server was unlocked, the corrupt order was removed followed by a restart of server. The same corrupt order was again unintentionally introduced, despite EPEX SPOT’s request to the market participant not to do so. This caused a chain of events which led to multiple server locks resulting in a delay of order book submission on EPEX SPOT side. This delay meant that the deadline of 12h40 for partial decoupling, as defined in the PCR procedure, was reached and partial decoupling was declared at 12h49. The following borders were decoupled:
• France-Belgium
• France-Germany
• Germany-Austria
• Germany-Denmark
• Germany-Netherlands
• Great Britain-Netherlands
• Great Britain-Belgium
• Great Britain -France
• Great Britain -Ireland/Northern Ireland
• Great Britain 1- Great Britain 2
• Italy-France
• Italy-Austria
• Netherlands-Belgium
• Norway-Netherlands
• Slovenia-Austria
• Spain-France
• Sweden-Germany
ISSUE ON THE LOCAL AUCTION
This partial decoupling of CWE and GB from Multi-Regional Coupling (MRC) implies that, according to procedure, local auctions are run for each local national market areas. These local auctions are also run on ETS. The fixing measure applied to remedy the issue of the corrupt order resulted in a further IT issue, with market results being calculated based on only a subset of the orderbook for all market areas, which subsequently led to the publication of erroneous market results. Due to the previous issues, the monitoring tool normally used to verify the orders did not function, resulting in an inability to verify the volumes prior to result publication. Considering the time pressure on the related gate closure times for nomination schedules, and as partial decoupling is an unprecedent market event, no second auction process was triggered by EPEX SPOT at this point. Upon later detection, the erroneous results were cancelled, and the orderbook was re-opened until 14h35. EPEX SPOT then calculated market results based on a complete order book set and final results were published at 15h38. At 15h49, EPEX SPOT sent the trade results to ECC which in turn validated all transactions. All nominations and payments were consequently completed and settled correctly.
In order to permanently remedy the underlying root cause, ETS system updates have been issued. After successful testing, these updates were deployed in ETS at 19h30 on the same day, in order to secure the ongoing robustness of the trading system for upcoming auctions.
Regarding the decoupling incident, EPEX SPOT underlines that all its actions were conducted in accordance with the procedures in place. All technical issues on ETS have been fixed and the EPEX SPOT markets have been running smoothly since. We do not expect any similar incidents on these markets going forward.
The issue on the local auction has led us to immediately implement additional manual checks and we will continue to assess and improve the existing checks, procedures and communication towards the market on EPEX SPOT side.
(All timings are in CET)