October 6, 2017

Key Elements of a Comprehensive Hedge Fund Disaster Recovery Plan

Hedge fund managers reasonably tend to focus the better part of their time and resources on fund performance, strategy execution, attracting and retaining top personnel, and developing robust business infrastructure, but increasingly, disaster recovery is—and should be—absorbing a growing share of hedge fund managers’ top of mind business concerns. According to the Identity Theft Resource Center, as of the first week of October, this year has seen over 1,000 data breaches across nearly all industries, including at international financial institutions, investment advisers, third-party service providers, like law firms, and notably, the Securities and Exchange Commission, which didn’t discover until August 2017 that a data breached in 2016 may have resulted in illicit trading. Nearly 200 million individuals (excluding the updated Yahoo breach statistics) have been affected or had personal information compromised by data breaches this year, and though the business costs of those breaches can only be speculative at this point, they surely reach the hundreds of millions.

The personal and corporate costs of small and larger scale cybersecurity breaches are staggering, and in terms of disaster recovery planning, they’re only one component among numerous potential catastrophes, such as hurricanes, earthquakes, power outages, terrorist attacks and other emergencies, that could befall a hedge fund manager and, in fact, also have occurred this year. The need for hedge fund managers to have a comprehensive disaster recovery plan in place that outlines procedures the firm will employ to ensure critical data, business operations and investment activities are uninterrupted, functional and/or recoverable during and after a range of crisis situations has, perhaps, never been greater. This article explains the substance of disaster recovery plans and applicable regulatory requirements; and reviews key elements specific to hedge funds of developing DRPs, including business impact analyses, services restoration timelines, the frequency with which DRPs should be reevaluated and tested, and institutional investor expectations for DRPs.

What is a Disaster Recovery Plan?

A disaster recovery plan identifies the steps a firm will take to implement and support shadow and everyday infrastructure and operations both during a catastrophic event and as the firm recovers. DRPs also identify a firm’s critical applications and outline a strategy and program for their continued operation or quick recovery.

According to Kevin Holl, director of technology at Evanston Capital Management, “A disaster recovery plan describes how a business will deal with outages to their systems or technology. A DR plan is typically complementary to a business continuity plan, which tends to be more about people, office space, facilities, and business processes. For hedge funds, disaster recovery has been part of the landscape for a long time, so managers should have a plan in place.”

Matt Hilsenrad, director of disaster recovery at the Abacus Group, explained, “DR is the technology component of a business continuity plan. The key elements of the plan will address how resilient your design is, how soon your systems are restored, whether and how often you are doing backups and then identifying key people and steps you need to recover from a disaster.”

DRPs have long been part of the hedge fund landscape and so most, if not all, managers should have a DRP in place, Holl continued. “It may be very basic depending on the size and complexity of the firm and its reliance on technology.”

Since DRPs are technology-driven, the events that lead to their implementation can include technical issues such as power outages, disruption of exchanges, computer viruses, equipment failures or cyber hacking; or natural and human triggering events such as hurricanes, earthquakes, pandemics, terrorism or theft.

Disaster recovery makes the assumption that the current business can no longer operate. According to Chris Grandi, founder, chief executive officer and chairman of the Abacus Group. “The three main causes for downtime include loss of power, loss of connectivity and human error. Natural disasters and weather can initiate two of these three outages.”

Disaster Recovery Plan Regulatory Requirements

In June 2016, the SEC proposed a rule that would require registered investment advisers to include business continuity and disaster recovery and transition plans to address operational and other risks related to a significant disruption in the adviser’s operations in order to minimize client and investor harm. The SEC has not yet adopted the rule, and there hasn’t yet been any indication as to when it will.

Rule 206(4)-7 under the Advisers Act, however, requires every investment adviser to adopt and implement written policies and procedures reasonably designed to prevent the adviser from violating the Advisers Act. The rule arguably can be interpreted to encompass disaster recovery planning since all advisers have a fiduciary obligation to their clients that includes taking steps to protect the clients’ interests from risks particular to the business, which can include an inability to provide advisory services during and in the aftermath of a natural disaster or other emergency.

In addition, under Advisers Act Rule 204-2, advisers must maintain books and records, including electronic storage media, “so as to reasonably safeguard them from loss, alteration, or destruction.”

The interpretation that disaster recovery planning is already a requirement of the Advisers Act has some basis in SEC precedent. In 2013, following Hurricane Sandy in 2012, the SEC’s Office of Compliance Inspections and Examinations issued a Risk Alert addressing the need for business continuity and disaster recovery planning and notifying advisers that such preparations are within the ambit of their fiduciary responsibilities. The SEC also issued a joint release with the Commodity Futures Trading Commission and Financial Industry Regulatory Authority explaining the steps firms should take to implement effective business continuity and disaster recovery plans in advance of another hurricane.

The National Futures Association also requires member firms to adopt a written business continuity and disaster recovery plan tailored to their operations, and to maintain them with other firm procedures. The NFA has also a written policy alone is insufficient to meet its requirements, and firms must also implement and follow the policy and communicate it to employees.

Developing a Disaster Recovery Plan

In accordance with regulators’ public statements on disaster recovery planning, each manager’s DRP should be tailored to its particular business operations and risks. “There is no one-size-fits all approach,” said David Alvarado, vice president of advanced client services for Peak 10 + ViaWest. “There is no one plan that is going to meet the needs of all managers. They need to look at what they will need to protect their fund, their people and their investors for their size and for what they are doing.”

A firm’s DRP will also vary depending on the disaster the firm is dealing with. Hilsenrad explained, “You may prepare differently for different types of disasters. Recovery times vary by service. Services like email and phones, for instance, can be back up in minutes following a disaster once you shift from one data center to the next.”

Identify Business-Critical Systems

The first step in developing a DRP is to conduct a business impact analysis to determine which applications are critical to the hedge fund’s operations and how quickly they can be restored in event of a disaster.

Alvarado advised managers, “Identify and prioritize critical business functions, both on the IT side and not. Hedge fund managers want to take an overall look at the business and determine which are the absolute critical applications and business functions that need to be up and running [and which need to be protected].”

Holl summarized, “The process of developing a disaster recovery plan typically begins with reviewing your systems inventory (or creating an inventory if you don’t have one) and ranking your systems and tools according to importance. Business critical systems should be the first ones to restore, or keep up and running, if there is some sort of event. The typical things that bubble to the top of this list are tools that are required to run your portfolio and tools for communicating with your investor base and other outside parties, such as email and phone lines.”

Alvarado agreed that managers should determine the recovery’s sequencing during the business impact analysis. “[Managers should] list these systems from most critical to least critical, so you know which functions you have to get up and running right away and which ones you can take some time to bring back online. As part of that sequencing,” he continued, “you need to look at application interdependencies and network interdependencies, because there may be several applications tied together that may make one particular application work. So by doing this analysis, you can figure out which applications you need back up in order for one particular application to come back up. With the network, you’re doing the same thing and looking for interdependencies to make sure all functions that are needed to make the whole thing work are included in the disaster recovery. You also need to look at the networks that are outside of the main data center that are important so those are covered in the DR plan as well.”

Managers must assess the various recovery points of their data and applications and review the technologies available to meet those various recovery points. “For the critical applications you’re probably looking at replication capabilities or having an ‘active active,’ meaning they have two data centers that share the workload, so if data center one is down then data center two can pick up the workload, and there is no disruption,” said Alvarado. “This is not a cheap option” he added.

Once managers assess the essential functions that have to be restored immediately, they can turn to focusing on restoration of non-essential applications that can be suspended or left offline until critical functions are restored. Such non-essential functions should be itemized, prioritized and brought back online in an orderly manner as needed.

Alvarado said managers can start the process of developing their disaster recovery policies and procedures once the business impact analysis is complete. “It’s not enough to have the equipment and the technology; you need to create a step-by-step guide to recover. The plans have to be so foolproof that anyone that has a technology background can pick them up and can recover the business based on how detailed the plans are. This is important, because during disaster event, the people you have working for you probably have a disaster of their own that is higher in mind than the disaster of the business, and they may have to take care of various situations at home and may not be available to do the recovery work for the firm.”

Since DRPs emphasize the technological impacts of business interruptions, Holl noted that the key members of a firm’s IT department must be involved in the planning process to ensure that the proper systems are used to back up, restore and maintain operations. “The firm’s technology head should take the lead in developing the plan, whether it’s an internal CTO or an outsourced technology partner. Business leadership, typically represented by a COO or president, should be a key member of the team developing the plan as well. We also think it’s a good idea to involve your compliance team to help with regulatory requirements.”

Impact of a Hedge Fund’s Strategy on a Disaster Recovery Plan

A hedge fund’s strategy can have a major impact on its disaster recovery processes because it can determine, at least in part, how often data must be backed up and how much downtime a fund can sustain.

Grandi said that to determine the impact of their strategies on the disaster recovery plan, managers should determine how frequently the fund trades. “If they are very frequent traders that rely heavily on technology, then it’s going to be more critical that they do not lose their ability to trade.”

Alternatively, he continued, “There are long/short equity funds that don’t trade a lot, and the reality is that they don’t need to come up as quickly in disaster recovery as a trading-intensive or algorithmic trading fund that relies heavily on technology to make trades.”

Key Elements of a Disaster Recovery Plan

Once the manager has conducted the business impact analysis and identified critical systems that need to be restored immediately and those that can come back online more slowly, the next steps in developing a disaster recovery plan include selecting a backup location for systems and networks, and communications, not only with employees but with service providers and investors as well.

Select a Backup Location

In order to ensure a fund’s business operations can continue during a disaster, it is critical for hedge funds to back up essential documents and data and store the information offsite. Funds can have this information stored with a service provider via a separate server or other data storage method, or backup systems can be housed in a backup facility.

When choosing a backup location, fund managers need to consider such characteristics as its security, generator services and telecommunications capabilities. A site should have diverse entry points to the building, telecommunications network abilities to allow a firm to support its disaster recovery systems and backup resources such as generators and fuel.

Grandi noted that the SEC used to expect funds to have their DR backup location within at least 90 miles of their primary location. “This was not a written requirement but more of an industry best practice recommendation. The best disaster recovery plan has the backup location in a different time zone.”

Mark Coriaty, chief strategy officer at Eze Castle Integration, advises firms on choosing a backup location and said that it’s a component of the DRP that firms often fail to fully evaluate. “We look at the physical location of your office and your production environment. Eze Castle has an office in Boston, but our data center that we leverage is outside of Boston and our backup location is in California. The rationale for that is so that your main location and your backup location are not impacted by the same disaster, such as a hurricane coming up the coast or a power grid failure. You want to make it geographically different so there are no event-driven impacts to both your production and disaster recovery sites.”

Hedge funds either can designate as backup locations other offices used by their firms or dedicated backup sites that are used exclusively for business continuity and disaster recovery. In addition, to being geographically separate from the manager’s primary site, backup facilities should be capable of becoming operational within the manager’s designated recovery time and outfitted with adequate technology resources to continue operations.

Alvarado added, “The primary data center should be in a location that has multiple networks from various carriers, multiple feeds from different substations of the electrical grid, and the secondary data center in a place that is geographically dispersed from the first so it cannot be impacted by the same things that could impact the primary site. The minimum safe distance depends on the location.”

Holl also noted that no one set of backup location recommendations will be applicable to every hedge fund. He did, however, recommend that, “Power availability is something that needs to be considered. Many firms have moved their production server environments out of their office and into data centers that have diesel backup, redundant power and redundant connectivity. Replication to another companion facility then provides further resilience. Geographic diversity between these sites is important; don’t have a production site and then a disaster recovery site that are a mile down the road from each other. Ideally, production and DR sites are sufficiently separated to mitigate the effects of a localized power outage, network outage or weather-related event.”

Holl added that during some disaster situations, “Your servers may be up and running, but if the city where your office is located has a broad power outage, then your people are not going to be able to connect to those servers. Some firms may decide to put generators at a portfolio manager’s house or other key locations. Workplace recovery services offered by some property management firms are another option.”

Communications

No matter how well-thought-out a DRP is, it can’t be effective unless a firm’s employees or service providers know how to implement it.

“Communication is probably the most important element, as it goes across your entire business,” said Coriaty. “It is critical to the safety of your people and the viability of your working environment.”

Holl agreed and offered discrete communications procedures. “Have a communication plan set up ahead of time, and have that plan distributed to your employees so they know who to reach out to if they have a question or who they should expect to hear from if there is a disaster. It’s important to have those things decided in advance and communicated to your group so they know what to do in a disaster. Make sure your team has up to date contact information. Establish rules around who at the firm has the authority to declare DR events—typically a president, COO, or CTO—to limit ambiguity about where instructions should come from.”

In that regard, managers should ensure consistent and accurate messaging across the firm during a disaster. Coriaty said there are systems available to automatically send out voice shots or emails to let employees know if something has happened, which can be invaluable during a business interruption. Coriaty also advised managers to map out key communications points with individuals in advance to determine who they communicate with internally and externally and how, so managers can ensure these communications channels are still available during the business interruption.

Grandi added, “You should have a disaster recovery call tree with everyone’s contact information so that you can communicate with everyone if something happens.”

The communications component of the DRP should also include employee training, experts who spoke with The Hedge Fund LCD agreed. Employees need to know both the elements of the firm’s disaster recovery plan and their functions during an emergency situation. According to Alvarado, “Training is absolutely mandatory. Everyone needs to be cross-trained. There should not be one group of individuals who has all of the knowledge but two or three groups that have the knowledge to be able to recover the environment and keep it flowing.”

Restoring Services

Alvarado said that the Recovery Point Objectives and Recovery Time Objectives for various applications and systems that hedge fund managers determined when they conducted the business impact analysis become operative when services are being restored.

The RTO, in particular, will determine how quickly a particular service needs to be available, he explained. “A critical application for a hedge fund may not be able to be down for more than a couple of minutes because a single transaction could be worth millions of dollars. So you need to know where in your disaster recovery plan to allocate the most resources to make sure a critical application, like trading, stays up and running or is back up and running very quickly.”

The RPO determines how much data a manager is willing to lose or have to recreate due to a business-disruptive event. “Since a single transaction can be worth millions of dollars, a recovery point objective is really important, because managers need to be able to see what happened with a particular transaction so their RPO may be very small,” said Alvarado. “Determining the RPO will depend on the needs of a particular application. With a trading application, for instance, you want all of the data as quickly as possible, so you want synchronous replication so you don’t lose a heartbeat.” The Service Point Objective will determine when end users have access to the firm’s systems, he added.

Holl advised that managers use worst case scenario analyses when designing restoration timeframes. “Timelines and service recovery objectives should be built with the worst-case scenario in mind. When would you have the tightest timeline and the least tolerance for error? Investor reporting deadlines or filing deadlines are typical examples. You should use this hypothetical to form your tolerance for downtime and design your plan to work within those tolerances.”

Coordinate With Third-Party Service Providers

Because hedge fund managers rely on a network of third parties—from prime brokers to administrators to accountants and lawyers—it’s possible that an event that impacts a hedge fund could also impact any or all of its critical service providers. This possibility makes it business-critical for hedge fund managers to review their service providers’ DRPs and have in place a method for communicating with them during a crisis.

When assessing service providers’ DRP, firms should first make sure there is a plan in praticable place and then review the results of the service provider’s testing around the plan. Service provider DRPs should be reviewed and tested on an ongoing basis at least semi-annually.

“Because hedge funds have such interaction with investors, service providers and others,” explained Alvarado, “they have partnerships to provide services—other brokers—so they need to understand their DR capability, because [service provider] outages could create outages for [managers].”

Holl added, “Proactivity is important when coordinating with your service providers. Have discussions in advance so when something happens you know what to do. The service providers will have their own disaster recovery plans and the best means for getting in touch with them if something happens, and you want to know these processes before you need them.”

Testing and Maintenance

Hedge fund disaster recovery plans must be tested and maintained internally at least annually. They should also be reviewed at least quarterly and updated if necessary, and any changes to the plan must be communicated to employees. The plan also should be tested to take into account changes.

According to Coriaty, “We typically see clients testing bi-annually or quarterly. A full-blown DR test means you are running the company off of your DR facility and then replicating it back to production for the next work day.”

“We do testing twice a year,” Hilsenrad offered. “We simulate what it would be like to bring a hedge fund manager’s servers offline then bring them back online, so they’re familiar with the process”

When testing the DRP, Holl said, “Verifying both the tools you are using to facilitate a failover, as well as the operation of your line of business systems in DR mode, is important. Booting up the DR environment, seeing that data replication is working, and testing your procedures for conducting failovers are key to ensuring that those systems will serve you should a future DR event ever occur.”

Investor Perspective

Disaster recovery plans often are just one element of a firm’s overall compliance program, so managers tend to disclose that the firm has a DRP in place, but only disclose details about the plan when specifically requested. During due diligence, however, investors increasingly are putting more emphasis on all aspects of a hedge fund’s operations, including disaster recovery planning. Issues such as disclosures concerning key business relationships and business sustainability, in particular, encompass disaster recovery. As such, fund managers should be prepared to discuss the key elements of DRPs and show evidence the plans have been tested.

According to Coriaty, “The due diligence process from investors on managers has probably increased 1000% in the last four to five years. We’re getting DDQs directly from investors.”

“Aside from being prudent business practice,” said Grandi, “the investor due diligence expectations are at such a point now that if you did not have a disaster recovery plan, the investor would likely withhold making an investment. It is absolutely critical that all hedge funds have disaster recovery.”

Providing the investor’s perspective, Holl observed, “Hedge fund investors definitely pay attention to disaster recovery. It’s important to try to understand how the planning process works, how it’s reviewed and tested, and any lessons learned from prior tests or actual events.”

Furthermore, Grandi added, investors are not only looking at a manager’s DRP, but they’re also reviewing the DRPs of managers’ service providers. “The due diligence process has gotten very thorough on the infrastructure side. Many institutional investors hire technology auditors to evaluate us and our systems.”