Navigating the Oracle Explosion
A breakdown of the most important metrics relevant to oracle solutions today - 2021-03-22
The Oracle Problem
The oracle problem is intrinsically related to one very simple limitation. Smart contracts, and the blockchains they run on, have no built-in functionality to interact with external sources of data. A useful analogy is computers without modems or internet connections: they have no way of communicating with the wider world. Why are blockchains designed this way? Because if they weren’t, they would effectively lose their greatest attribute—their ability to form consensus on, for example, data values or the state of a network.
However, for blockchains and smart contracts to reach their full potential they need access to data. For example, decentralised flight insurance products need access to flight information and derivative products need access to pricing data. In short, blockchains and smart contracts need a reliable way to interact with external (off-chain) data resources such as data providers, web APIs, enterprise systems, cloud providers, IoT devices, payment systems, and other blockchains. Further, they need to do it in a manner that does not jeopardise a system’s integrity. This is the crux of the oracle problem, and also the reason we have oracles.
What are Oracles?
Oracles can be described as agents (commonly software solutions) that facilitate the retrieval and transfer of information. At its most basic level, the ideal behavior of an oracle can be defined by three steps:
- Accept a request for data.
- Obtain data.
- Return data.
As the blockchain and smart contract space has evolved and matured, tasks carried out by oracles have become more sophisticated. This has led to a surge in use cases that, until recently, had not been envisaged, let alone implemented.
The Rise and Rise of DeFi
In the past 12 months, the growth of all things related to DeFi can be described as nothing less than explosive. New DeFi projects pop up on what seems like a daily basis. Lending, borrowing, hedging, derivatives, and synthetic assets—the list is long. In February 2020, the Total Value Locked (TVL) of the top DeFi protocols stood at slightly over US$1 billion. Today that number stands at roughly US$40 billion.
The popularity of these DeFi protocols has had a knock-on effect. Namely, a need for significant amounts of on-chain data which is used to power these nascent financial projects. This is where oracles step in.
Trusting the Data
At first glance, the oracle problem does not seem overly complicated. Request data, receive data, and plug data into a protocol, dApp, or smart contract. However, there are malicious entities constantly looking for ways to game systems. Poorly designed oracle networks are often viewed as potential weak points just waiting to be exploited. In recent months, the blockchain world has witnessed a spate of financially crippling oracle exploits caused primarily through the manipulation of centralised and/or single-source price oracles via flash loans.
Thankfully, developers are beginning to understand the crucial role secure and resilient oracle networks play in not only in securing a protocol's finances, but also in securing its reputation within what is becoming an incredibly competitive and increasingly crowded space.
“It was important for us to use an oracle network that met the highest quality standards for security, reliability, and accuracy. We require that the data feeds triggering our financing processes be of the same standard of security and functionality as the underlying blockchain” Derek Mayne, Co-Founder and MD of CrescoFin.
Snapshot of Different Oracle Solutions
The oracle problem has spawned a multitude of different oracle architectures and incentivisation schemes for secure data delivery. Below is a small section of the oracle solutions available. All excerpts are taken directly from a projects respective web page.
“API3 data feeds are governed by an open DAO of stakeholders, industry experts and project partners. This allows dAPIs to be operated with maximal transparency, minimal required trust in centralised operators, and no centralised attack surfaces.”
“Based on Cosmos' state-of-the-art SDK, BandChain can process thousands of transactions per second with instant finality and the ability to send oracle data across multiple blockchains.”
“Chainlink's decentralised oracle network provides reliable, tamper-proof inputs and outputs for complex smart contracts on any blockchain.”
“The Open Price Feed is a system to allow various sources (i.e. reporters) to sign off-chain data about prices and allow any Ethereum account to move that data on-chain.”
“iExec makes it easy for anyone to deploy their own Decentralized Oracle, allowing developers to leverage the terabytes of data available on the WEB 2.0 to build a whole new range of useful and impactful DApps without compromising on the security.”
“Kylin Network offers any applications and blockchains (such as parachains and parathreads) instantaneous but reliable and valid on/off-chain market data and social data sources by leveraging the power of Polkadot/Substrate Framework on open networks.”
“The Maker Protocol uses oracles to enable the use of price data of various assets to determine a number of important things like when to liquidate a Vault or how much Dai a given Vault can generate.”
“All risk associated with NEST oracle prices can be computed by a predictable model which quantifies your risk. As possible deviations of NEST oracles are controllable, you can fully determine the risk when you make, market, trade or build other applications on NEST oracles.”
“Oraichain's data oracle aggregates and connects Artificial Intelligence APIs to smart contracts and regular applications. It is the world’s first AI-powered oracle aiming to revolutionize the AI, DeFi and Blockchain industries.”
“The ProvableTM blockchain oracle for modern dApps. Enabling the shift of traditional services such as finance, gambling, and insurance into decentralisation.”
“Tellor is a permissionless community of token holders, data providers, and validators. Together, we cryptographically secure putting real world data on-chain.”
“The Witnet protocol aims to create an overlay network that connects smart contracts to any online data source, in a trustless, decentralised manner.”
“Decentralised bonding curve curation market. Providing access to data providers and other services through algorithmic token generation.”
“Zoracles partners with DeFi projects using Open Oracle with zero-knowledge proofs to provide confidential data to smart contracts.”
So, how do you determine if an oracle network is secure and resilient? Which oracle solution should you trust? Although there are a multitude of ways to assess the efficacy of an oracle network, we decided to limit our checklist to what we believe are the six most important metrics.
1. Centralised Versus Decentralised
Arguably, the single most important design consideration when evaluating oracle solutions is their level of decentralisation. In its simplest form, decentralisation distributes power and influence away from central authorities that can act as single points of failure within a system. Malfunction of these centralised entities, whether unintentional or deliberate, can result in catastrophic consequences for systems.
Considering the ramifications of having “single points of failure” within an oracle system (i.e. unreliable data delivery, manipulated information, or server downtime), it seems almost inexplicable that centralised oracle solutions exist. Yet they do exist, and they are being actively used.
Decentralisation within the context of oracles and oracle networks can be evaluated in many ways, however, there are two fundamental questions that should be asked when assessing any oracle solution.
Firstly, how many different data sources does an oracle network utilise when fulfilling a request for information?
The most high-profile example of a centralised oracle network is Provable (previously known as Oraclize). Provable retained most of the Oraclize feature set, but added a component that allows it to interoperate with decentralised oracle systems, such as Chainlink.
Compound.Finance uses what it calls an “Open Oracle System”. Open Oracle System currently uses only Coinbase Pro data in conjunction with Uniswap V2 on-chain price data. The Uniswap V2 data is described in official Compound.Finance documents as a ‘sanity check’.
In April 2020, Coinbase positioned themselves as a standalone oracle service when they launched their Coinbase price oracles. In reality, the Coinbase Price Oracles are just simple data sources, similar to other signed API’s like CryptoCompare.
The MakerDao medianizer uses an inhouse oracle solution that allows only authorised whitelisted price feed contracts to post price updates. MakerDao calls its oracle nodes “feeds”. Each feed pulls and then aggregates data from multiple exchanges. In the case of MakerDao, the addition and removal of whitelisted price feed addresses is controlled via governance—a topic we will discuss later in the article.
In the case of Chainlink, the number of data sources is not limited and varies according to the requesting protocols’ specific requirements. For example, Chainlink price feeds source data from off-chain, centralised exchanges (e.g. Binance), on-chain decentralised exchange protocols (e.g. Uniswap, Kyber), and data aggregators (e.g. BraveNewCoin, CoinGecko). Read more about Chainlink’s data aggregation here.
Secondly, how many oracles, nodes, workers, or validators contribute information to any given data request?
Again, different protocols adopt different solutions. The MakerDao Medianizer has a total of 24 whitelisted feeds (AKA oracle nodes), each of which pull data from multiple exchanges.
Chainlink’s LINK-USD off-chain reporting price feed currently queries 31 node operators. Each of these node operators pull data from multiple CEXs and DEXS and weights by volume. This prevents issues as described here and ensures market coverage. At the time of writing, there are 121 Chainlink nodes listed on market.link.
Again, Chainlink’s core functionality allows protocols to use as many node operators as they feel necessary. They can select as few or as many as they wish in order to achieve their desired level data security and reliability.
In broad terms, the level of decentralisation of any oracle network can be gauged by simply counting the number of oracles or nodes used for obtaining off-chain data, and by recording the number of different data sources a project utilises. Although this may seem overly simplistic, higher numbers generally equate to greater levels of decentralisation and increased network resilience and security.
The second oracle network design feature to consider is a project's core architecture. Broadly speaking, this feature set can be divided into two main categories.
Projects that use a blockchain as their oracle network (e.g. Band, Witnet, Tellor).
Blockchain agnostic oracle networks that run natively on any chain (e.g. Chainlink).
Projects that fall into the first category include the Band oracle network which runs its own Tendermint based blockchain using the Cosmos SDK (Software Development Kit); Tellor, which has miners competing to add data points to an on-chain data bank; and the iExec’s oracle network dOracles, which runs through its own PoCo (proof of contribution) protocol.
Introducing consensus mechanisms and other blockchain specific tasks into the running of an oracle network increases levels of complexity in systems whose primary goal is data delivery and not blockchain security and integrity.
In addition, Band and dOracles use random selections of validators/nodes for their data queries. However, because on-chain consensus needs to be reached, all validators/nodes need access to all data. This can result in significant monetary outlay, in that every node needs to purchase API subscriptions to every authenticated API. In reality, most oracles using this design limit themselves to free or open APIs to avoid excessive costs.
The second architectural approach sees projects designed in ways that do not constrain them to the consensus mechanisms of a single blockchain. The most high-profile project employing this approach is Chainlink. Essentially, Chainlink is a framework for building oracle networks and each network can be designed to meet the specific requirements of any use case. Chainlink allows for an unlimited number of oracle networks to run in parallel without any cross dependency. It is also designed to run on any blockchain or layer 2 scaling solution. At the time of writing, Chainlink has oracle networks actively running on Ethereum, Polygon, Binance Smart Chain, and XDai.
A key tenet for any oracle solution is its level of transparency. All users of a given oracle network should have the ability to critically assess how, and by who, a data request is completed.
In the case of dOracles, their version of nodes, which are called “workers”, are randomly chosen from a much bigger group and are all assigned the same computation. Each of the workers proposes a result and the result that is proposed by the biggest number of workers is taken as the overall computation result. Read more here.
Band uses a similar process whereby Bandchain validators are selected at random when responding to requests for data. Note that BandChain validators are responsible for performing two main functions on the network. Firstly, they are responsible for proposing and committing new blocks in the blockchain, and secondly they assist with the chain's capability to natively support external data query.
Initially, the act of randomly selecting workers or validators might appear to add extra security to an oracle network. However, this process requires a giant leap of faith: users of the network must trust every single contributing worker, validator, node, or oracle. In reality, the act of randomly selecting nodes lessens transparency within a network—only the wider pool of contributors is ever known at the time of a request.
MakerDao medianizer use something they term “Dark Feeds”. Dark Feeds are anonymous individuals or institutions that run a feed. At the time of writing, the MakerDao medianizer uses 15 Dark Feeds from a total of 24. This high level of anonymity obscures who is delivering data on-chain.
In comparison, users of the Chainlink network can create their own bespoke oracle network. Users can select as many operators, and whichever operators they want when designing their oracle network. In addition, network users can scrutinise the historical performance of every node on the Chainlink network using third party entities such as reputation.link and market.link. This added level of transparency of a node operator's historical performance acts as a strong incentive for reliable and efficient data delivery.
Transparency within a project can also be measured in more subjective ways. A project's key concepts and core workings might be complex and difficult to understand. If this is the case, projects should provide resources so that inquisitive minds can direct technical questions to project developers who will provide honest and valid answers.
As the oracle problem is centered around data and data delivery, it is important to consider what data sources an oracle network can connect to.
Band oracles are currently in Guan Yu (or phase 1) rollout. Phase 1 only supports free APIs. As the protocol moves to Laozi (phase 2), the protocol will support additional private APIs. Until then, developers are unable to call any password-protected (authenticated) API, severely limiting the type and the quality of data available to consuming contracts.
It is also worthwhile noting that there is a distinction between first and third party oracles. First party oracles are operated by the API providers themselves. Third party oracles act as middlemen between the data source and the Blockchain. API3 is an example that limits itself to only first party oracles, while Chainlink supports both first and third party oracles. Flexibility is key when dealing with data sources. Chainlink supports connections to all APIs, including free open APIs, paid authenticated APIs, and proprietary private APIs. This is achieved by using external adapters.
External adapters are services that extend the functionality of Chainlink nodes. They give smart contracts on any blockchain access to the entire spectrum of off-chain data and services that exist outside of public APIs. External adapters can be created and deployed without needing additional support. Market.link lists external adapters that can be freely used by developers.
APIs represent just one source of data of the many that oracle networks are capable of connecting to. Data quality and reliability is a key component to an effective and successful oracle network.
Flexibility within an oracle network is the ability to easily alter parameters of a data request such as the number of oracles, data sources, or frequency of data updates.
There are, of course, other aspects to consider when assessing a project's flexibility. Take infrastructural flexibility for example. The Chainlink network allows anyone to design and deploy an external adapter. But this ethos goes much deeper: anyone can run a node and anyone can list jobs on the open marketplace. There are essentially no restrictions.
But what about quality control? Dishonest node operators or poorly written code. This is where market forces and reputation systems such as reputation.link come into play. Inferior products and node operators with poor reputations will receive less work—it is as simple as that. However, it is important to remember that Chainlink provides the flexibility for anyone to participate in its network. Essentially the network affords infinite flexibility.
Other oracle solutions have taken much more conservative routes. DAO (decentralised autonomous organisation) related governance models can hamper the flexibility and adaptability of a protocol, or at least the ability to move quickly. Voting takes time.
The ability to demonstrate flexibility within a protocol suggests it will evolve with the rapidly moving market forces and technological advances that are constantly reshaping the smart contract space. Oracle solutions that demonstrate degrees of rigidity and centralised control will struggle to adapt and evolve in the long term. Flexibility, and the ability to adapt, must be a key component for any oracle solution wanting to prosper and to survive.
6. Numbers Don’t Lie
Ideally, any evaluation of an oracle network would be based solely on its technological specifications and merits. However, as is often the case in the world of blockchains, outlandish (and at times fabricated) declarations of a project's unrivalled brilliance are common.
For the reasons listed above, it is useful to spend time studying a project's metrics. Although adoption and usage statistics might not be entirely indicative of a project's overall proficiency, they do provide revealing insights into the nature of a project's progress.
Areas to consider include:
- Is the project live on mainnet? How long has the project been actively providing data?
- How many integrations, partners or collaborations does the project have? Can these collaborations be verified, both by press releases and by on-chain transactional records?
- How many data requests are completed in 24 hours, one week or the cumulative total since launch date? Has the number of data requests increased, remained stable or decreased over time?
- How many distinct data feeds does a network supply? How many nodes are active on the network? Are these numbers increasing?
- How does the project function during periods of high market price volatility? Does the network function as designed, or are there interruptions to data delivery?
- Has the oracle network ever suffered from an exploit?
In the case of Chainlink, the reputation.link and market.link websites provide a variety of resources and project specific data that answer many of the questions noted above. With respect to project integrations, sites such as Chainlink Ecosystem list all the announced Chainlink collaborations. Answers to the above questions should be readily available as blockchains are designed to be transparent.
As the popularity of blockchains and smart contracts continues to grow, and as these nascent technologies begin to take market share away from legacy systems, exposure and interest in oracles and the services they provide will grow exponentially. As awareness grows, the number of projects claiming to have solved the oracle problem will also grow. Buzzwords, hype, and outrageous claims will cloud the space. The community requires tools to help sift the wheat from the proverbial oracle chaff.
The oracle problem is real, it is important, and projects that solve it hold the potential to unleash a true technological revolution. Understanding which of the many proffered oracle solutions are capable of delivering data to smart contracts in a secure and decentralised manner is something developers and technology enthusiasts need to be able to objectively evaluate. Working through the six categories listed above will provide a good starting point. For valid, bona fide projects, the information is out there. It can be found and it can help make the world of blockchains and smart contracts a much stronger and secure space.