How to Run a Chainlink Node: Understanding OCR and RPC Nodes

In this article, we will explain how to run a Chainlink node in production. Before proceeding, ensure you have read and understood our previous blog posts on:

1. What is a Chainlink Node?

2. How to Set Up a Chainlink Node

3. The Skills Needed to Become a Chainlink Node Operator

We also assume you have read all published Chainlink documentation on running a Chainlink node.

In this article, we cover additional considerations for running Off-Chain Reporting (OCR) nodes, which are the most common types of verified Chainlink nodes. To successfully operate this type of Chainlink node, you will also need to understand how to run RPC nodes. That’s why we discuss this topic later in the article. Let’s begin by understanding what an OCR node is.

What Is an OCR Node in the Context of Chainlink?

An OCR node in the context of Chainlink is a type of node that participates in Chainlink’s decentralized oracle networks (DONs) which is crucial for several Chainlink services, such as Data Feeds. Essentially, any OCR node used for a Chainlink service can also be called a Chainlink node.

The OCR node is responsible for aggregating data off-chain before reporting it to a reference smart contract through RPC nodes. OCR nodes work together to reach a consensus on the data to be reported, ensuring its accuracy, decentralization, and reliability.

As you can see, OCR nodes are central to solving the oracle problem. The oracle problem refers to the challenge of securely providing external data to smart contracts on the blockchain in a decentralized manner. OCR nodes address this challenge by enhancing security, ensuring reliability, improving efficiency, and facilitating scalability.

How Chainlink OCR Nodes Solve the Oracle Problem

Once you become a verified Chainlink node operator, you will most likely start operating OCR nodes that participate in Chainlink Data Feed oracle networks. This involves deploying an OCR node and RPC nodes to a specific network, and obtaining subscriptions to various premium data providers based on your assigned jobs, such as the LINK/USD feed on the Ethereum mainnet.

After adding the job, you need to establish a connection to the peer-to-peer (P2P) network. Then, the OCR nodes initiate the following process:

1. Data Fetching: Each OCR node independently collects data from various external sources. The Chainlink node then calculates the median of the responses.

2. Communication: The nodes then communicate with each other off-chain to share their price observations. This communication takes place in the peer-to-peer network that connects all the nodes participating in this specific DON.

3. Aggregation: The observations of all participating nodes are gathered and at the end of an observation round the median of all individual responses is calculated once again.

4. Consensus: The nodes reach a consensus on the final aggregated data through a majority agreement. This consensus mechanism ensures that the data is accurate and not tampered with by any single node.

5. On-Chain Reporting: If the update requirements of the aggregator contract are met (deviation threshold or time limit exceeded) then the final report is transmitted on-chain by one of the participating nodes and node signatures are verified on-chain.

6. Redundancy and Security: To further enhance security and reliability, multiple OCR nodes perform these steps independently. Even if one or more nodes fail or act maliciously, the network as a whole can still provide accurate and reliable data due to the redundancy and consensus mechanisms.

7. Payment of a Chainlink Node Each node has an assigned payee account, separate from the Chainlink node wallet, to which rewards for each valid observation as well as on-chain transmission get assigned and distributed automatically.

For on-chain reporting of the approved data, we will also cover how to run RPC nodes as a Chainlink node operator later on, since RPC nodes enable interaction with smart contracts and blockchains. Before delving into that, let’s quickly review the essential technical aspects to consider when running a Chainlink OCR node in production.

Technical Requirements for Running Chainlink Production OCR Nodes

Now that you understand the basic functioning of OCR nodes, you also need to grasp the necessary technical aspects of operating them in production.

Required Hardware Specs of an OCR Node

The hardware requirements of an OCR node are 4 cores, 8GB RAM and 20GB SSD disk. It is possible to run several OCR nodes on a single server. For example, you can run 10+ OCR nodes on a server with 10 cores, 50GB RAM and 800GB SSD, and with a good network connection you should not have any problems with P2P and External Adapter traffic.

PostgreSQL of a Production OCR Node

The recommended hardware requirements of the PostgreSQL server for OCR nodes are 4 cores, 16GB RAM and 100GB SSD disk. Chainlink nodes use active connections to read states from and write to the database. The PostgreSQL connection therefore is of paramount importance for the overall performance of the Chainlink node as they are in constant communication, so as soon as the connection drops the Chainlink node is down. Each Chainlink node requires and locks its own database – one single OCR node in action uses about 20 open connections to the database.

By default, a PostgreSQL server allows 100 active connections, so it could host databases for several Chainlink nodes. For high throughput Chainlink nodes you can adjust the database related environmental variables to boost the performance, please make sure to read and understand the official documentation of Chainlink before doing this.
Example settings for a high throughput OCR node are:

1. ORM_MAX_IDLE_CONNS=25

2. ORM_MAX_OPEN_CONNS=40

Hosting of Production OCR Nodes

An RPC node is a computer running software that communicates with the blockchain network. It can be a full node (with a complete copy of the blockchain) or a light node (connected to other nodes for blockchain data).

While it makes sense to run a locally hosted Chainlink node for initial testing purposes to gain a basic understanding of its functioning, production nodes are expected to maintain the highest possible availability which can be reached by leveraging a cloud computing platform or server colocation in a data center to achieve redundancy and automated location independent failover solutions. In the following, we want to provide an overview of the various options and their implications.

1. It might be the fastest and most convenient to deploy an initial Chainlink node stack to a major cloud platform as they provide highly available VMs across availability zones and regions, helpful pre-built services such as managed PostgreSQL databases, pre-configured OpenVPN servers and many more. Each cloud platform has different services, features and pricing models which are constantly changing. It is therefore recommended to independently research and test different options. 

2. Hosting your Chainlink node stack in a data center that offers server colocation is another option. It is also worth looking into bare metal servers. This requires more experience in the fields of system and network administration and a higher maintenance effort than a cloud-based environment. You need to come up with a custom network configuration and failover solutions, as well as operating a redundantly running failover stack, so a minimum of two data centers in different availability zones or regions should be used in production. Data center hosted infrastructure comes with the advantage of having complete deployment and configuration sovereignty. Additionally, renting bare metal servers is usually more cost-efficient than using machines with equivalent specs offered by cloud platforms.

3. A hybrid model could also be applied, where parts of the Chainlink node stack are hosted on a cloud platform and others in a data center. Depending on the network, the major cloud platforms might not even offer servers with the required hardware specs for certain blockchain RPC nodes. Node operators should inform themselves in depth about the decisive factors when choosing hosting options, anticipating as best possible the potential expansion of their operations and planning accordingly for the long term. Migration of infrastructure in production involves a lot of effort.

Container Orchestration and Infrastructure Management of Production OCR Nodes

While it is possible to build Chainlink nodes from source, running nodes in container orchestration tools is commonly recommended and this should also be applied in production. Each node operator must develop their own strategy for the deployment of new services and the general management of infrastructure. Various practices have become established, including the use of Kubernetes, Terraform and Ansible. There are helpful resources from experienced node operators, to which we only link here as addressing them would go into too much detail at this point.

Security Best Practices for Production OCR Nodes

A basic security best practice is secrets management, so it is advisable for organisations to work with a password manager, ideally self-hosted.

1. The use of multi-factor authentication is recommended wherever possible. As of v1.7.0, the Chainlink node operator UI allows the use of 2FA. Since this version, Chainlink nodes also allow the root admin CLI user (and any additional admin users created) to create and assign tiers of role-based access to new users. There are four roles: admin, edit, run, and view. These new API users will be able to independently log in to the Chainlink operator UI and can each have specific roles tied to their account. This comes with the benefit that only the admin user has full access to the Chainlink CLI, so you can grant someone on your team limited access to the Chainlink operator UI by assigning the “edit” role. This user can then create bridges, add and remove job specs and change node configs; however, this user cannot export the wallet keys or manually send transactions via the CLI. With the release of this functionality, Chainlink nodes enable several people with different roles to work together on a Chainlink node without taking an internal security risk.

2. Chainlink nodes can be accessed via SSH tunnelling.

3. Usage of a VPN is highly recommended to further restrict the access of internal resources and routing traffic over the internal network.

4. Database servers should only have internal IPs as they only need to be reachable from the internal network.

5. Chainlink OCR nodes need to be publicly reachable to allow P2P communication, but should only have the OCR P2P port open.

6. Interfaces which are exposed to the internet, such as the Chainlink node operator GUI, should be TLS-encrypted.

High Availability Best Practices of Production OCR Nodes

It is a requirement to operate a minimum of two redundant, geographically distributed hosted Chainlink node instances pointing to the same DB. The PostgreSQL database should run on a server separate from the one hosting the Chainlink nodes. It is highly recommended to create regular backups of the Chainlink node database in order to restore the last state of the node, including its wallet and jobs, in case of issues.

To be prepared for unexpected incidents, it is therefore recommended to also regularly test the restoration process. The monitoring and alerting system allows node operators to keep an eye on all services and processes of the stack to identify problems early and prevent scenarios such as Chainlink node downtime.

Now that you understand the technical requirements for running a Chainlink node, there is one crucial aspect left for successful operation: RPC nodes.

What Is an RPC Node in the Context of Chainlink?

An RPC (Remote Procedure Call) node in the context of Chainlink is a crucial component that allows Chainlink nodes to interact with blockchain networks. Here’s a detailed explanation:

1. Definition: An RPC node is a service that provides access to a blockchain network via remote procedure calls. It acts as an interface for applications and other nodes to interact with the blockchain.

2. Functionality in Chainlink

   • Data Retrieval: Chainlink nodes need to query the blockchain for various types of data such as transaction details, block information, and smart contract states. An RPC node facilitates these queries by providing a standardized API.

   • Transaction Submission: When Chainlink nodes need to send data back to the blockchain (for example, after aggregating off-chain data), they use RPC nodes to submit these transactions.

3. Importance for OCR Nodes

   • Off-Chain Data Reporting: OCR nodes collect and aggregate data off-chain. Once a consensus is reached among OCR nodes, the aggregated data must be reported on-chain. RPC nodes are used to send these transactions to the blockchain.

In summary, RPC nodes are essential for Chainlink operations as they enable the interaction between Chainlink nodes and the blockchain network. They provide the necessary infrastructure for querying data and submitting transactions, which are vital for the accurate and reliable functioning of Chainlink services.

How to Run Blockchain RPC Nodes as a Chainlink Node Operator

As we now know, a Chainlink node requires an active connection to an RPC node to interact with the target blockchain in order to read from and write to it. Verified Chainlink node operators are expected to run multiple RPC nodes for each network on which they are running Chainlink nodes.

RPC nodes require regular maintenance to stay fully synced and operational. They should therefore be redundant and geographically distributed, with a minimum of two RPC nodes per network being highly recommended.

Since v1.3.0, Chainlink nodes come with an internal RPC node failover solution that lets node operators set several RPC nodes in an environmental variable and spares the need for your own RPC node failover proxy. Please read Chainlink’s official documentation about the EVM_NODES setting, which allows operators to set multiple WebSocket and HTTP addresses.

Technical Requirements for Running an RPC Node

Different networks’ RPC nodes have different hardware requirements, but  commonly range between 4-8 cores, 32-64 GB RAM and fast stateful storage of 1 TB+ for the chain database. Solana, as a non-EVM chain, is an outlier which requires 16 cores, 512 GB RAM and 1 TB+ NVMe.

Additional Considerations

Node operators should encrypt traffic via TLS and apply firewall management if their RPC nodes are exposed to the internet. Cloud-based hosting of RPC nodes can get expensive very quickly – data centers usually offer better conditions. It is recommended to get started with spinning up a testnet RPC node using one of many great guides such as Go Ethereum and Avalanche.

Running Chainlink Nodes Is Complex

Running a Chainlink node involves understanding concepts like the OCR protocol and RPC nodes, which are essential for efficient and reliable data handling. In this guide on how to run a Chainlink node, we covered the fundamentals of these components. However, to fully optimize your Chainlink node, you will also need expertise in the field of External Adapters. Stay tuned as we will cover this crucial topic in an upcoming post.