Skip to content
This repository has been archived by the owner on Aug 8, 2018. It is now read-only.

Making a User Service: Tutorial

Jump to bottom
vbuterin edited this page Oct 16, 2015 · 14 revisions

Prerequisites

This tutorial assumes that:

  • You have already installed pyethapp, and running pyethapp run on the command line works; if you have not done this, see this tutorial for instructions
  • You know how to use the console to send transactions and create contracts; if you do not, see this tutorial.
  • You know how to program in python; if you do not, consider starting here and here
  • For the later section, you know what HTTP GET requests and REST APIs are.

First Steps

One of pyethapp's most powerful distinguishing features is its ability to easily create built-in user services: scripts written in python that run alongside pyethapp, and run code on startup and every time you receive a new block. This allows you to create "server daemons" for applications that require periodic automated support, such as RANDAO, data feeds, "decentralized dropbox" apps, alarm clocks, decentralized cloud computing services, etc.

User services are stored in the contrib sub-folder of the pyethapp data directory (eg. ~/.config/pyethapp/contrib on Ubuntu); if this folder does not yet exist, then create it. To add a new service, in that directory create a python script with any name, as long as that name does not conflict with another python module (eg. helloworld.py is fine, math.py is not). Now, let us create our first service. In a file called helloworld.py in the contrib directory, put the following:

import sys
    
def on_start(app):
    print "Hello, world!"
    sys.exit()

Now, do pyethapp run on the command line. It should start up, and then quit in about five seconds. The last lines of the output should look something like this:

INFO:app	registering service service=factory generated service 0
INFO:app	starting 
Hello, world!

Now, let's make the service not quit immediately, and instead add an on_block method. Let's put the following into helloworld.py:

count = 0

def on_block(blk):
    global count
    count += 1
    print "Total number of blocks seen this run: %d" % count

When you run pyethapp and start syncing, you should eventually see something like this:

Total number of blocks seen this run: 1
new blk <Block(#375193 17e5591f)>
INFO:eth.chainservice	added txs=2 gas_used=42000 block=<Block(#375193 17e5591f)>
Total number of blocks seen this run: 2
new blk <Block(#375194 d6847603)>
INFO:eth.chainservice	added txs=1 gas_used=21000 block=<Block(#375194 d6847603)>
Total number of blocks seen this run: 3
new blk <Block(#375195 c6061bfe)>

And those are the basics; you can put whatever code you want into on_start and on_block and it will run.

HTTP GET request service example

Now, let's use this to do something interesting. What we are going to do is create a centralized data feed service which essentially acts as an HTTP proxy: contracts can call the get function of a particular contract passing any desired URL as an argument, paying a small fee (in our case, 7 finney), and a successful call creates a log containing the URL, a callback ID and a callback address. The proxy service listens for logs, and upon seeing a log of the right type (LogRequest) to the right address it sends an HTTP GET request to the provided URL, and then sends a callback message containing the HTTP GET response to the address that sent the request. Essentially, an instance of this service run by a trusted party allows Ethereum contracts to access any data feed that is available via a REST API over the internet.

First, determine a seed to generate a private key and an address; for this example we will use qwufqhwiufyqwiugxqwqcwrqwrcqr as our seed. The privkey is c7283d309fbac6fd65d2d9664c04dc71bbbdeb945b359501e912ab007da4fd27 and the address is 0xb3cd4c2512402047ef4ddeb1ab9f8df400ce2d3f. Now, we will write a contract in serpent as follows:

event GetRequest(url:string, callback:address, responseId:uint256, fee:uint256)
data nextResponseId

def get(url:string):
    if msg.value >= 7 * 10**15:
        self.nextResponseId = sha3(block.prevhash + sha3(url:str) + msg.sender + msg.value * 2**160)
        log(type=GetRequest, url, msg.sender, self.nextResponseId, msg.value)
        send(0xb3cd4c2512402047ef4ddeb1ab9f8df400ce2d3f, self.balance)
        return(self.nextResponseId)

This contract does several things. First, it contains a get method with the right signature, which other contracts can call. It enforces a minimum fee of 7 finney, which covers gas costs for your return transaction plus a bit more to spare. It then generates a callback ID for the request; the complicated sha3 formula for generating the callback ID ensures that if the blockchain forks and a different set of messages is made and/or in a different order then the new IDs will all be different from the old IDs, so there is no risk that a response that you make to one request will be accidentally matched with a different request.

It is expected that contracts calling get use these IDs as a way of keeping track which function call is a callback to which request; in the future we may well see high-level programming languages for Ethereum that abstract this logic away into something that looks more like nodejs-style callbacks or promises. Finally, it logs the request data (along with the callback ID and the fee paid for the request), passes along the fee to the account that will be returning the callback, and returns the callback ID to the contract that sent the request so that the contract can store it for future use.

Compile the contract, and push it to the blockchain (see "Creating Contracts" in this tutorial for how to do this in detail; for quick reference the one-liner to compile the contract, push it and retrieve the contract address is t = eth.transact(to='', data=serpent.compile('proxy_contract.se'), startgas=500000); t.creates.encode('hex')); from here we'll assume that the contract's address is 0xd53096b3cf64d4739bb774e0f055653e7f2cd710.

Now, just for our own testing purposes, we'll create a caller contract:

event LogResponse(response:string, fetchId:uint256)
data cbids[]

extern main: [get:[string]:int256]

def callback(response:str, responseId:uint256):
    if self.cbids[responseId] and msg.sender == 0xb3cd4c2512402047ef4ddeb1ab9f8df400ce2d3f:
        log(type=LogResponse, response, self.cbids[responseId])

def call(fetcher:address, url:str, fetchId:uint256):
    log(type=LogResponse, text("cow"), 0)
    x = fetcher.get(url, value=msg.value)
    self.cbids[x] = fetchId

Compile and push this contract to the blockchain; we'll assume the address is 0x6acc9a6876739e9190d06463196e27b6d37405c6. This contract is meant for demonstration purposes, so it does not do much interesting; we call it supplying (i) the address of our proxy contract, (ii) a URL, and (iii) any number, and upon receiving the callback, and confirming that it comes from the correct address (ie. the one controlled by our service), it creates a LogResponse containing the HTTP GET response and the number.

An exercise for the reader is to replace the fetchId argument with a recipientAddress argument, and instead of making a log, parse the HTTP GET response to get the USD price of ether (possibly using a different API that is more convenient), and then divide 5 by that value, and use that result, plus the recipientAddress parameter that got passed through the callback chain, to send the recipientAddress exactly 5 dollars worth of ether (a practical application of this is USD-denominated recurring payments). Note that, because many REST APIs provide answers in JSON form, an extension to the service that would make this easier is allowing it to supply a path down the JSON object, eg. price/usd, and resolving this on the server side (which is much cheaper and programmatically easier than resolving it on the blockchain).

Now, let us get to the meat of the task: writing the service that does the HTTP GET fetching.

The entire code is available at https://github.com/ethereum/pyethapp/blob/develop/examples/urlfetcher.py; feel free to simply copy it into your contrib directory, but we will nevertheless go through how the code works. First, the on_start method.

# Called once on startup
def on_start(_app):
    print 'Starting URL translator service'
    global app, my_nonce, chainservice
    app = _app
    chainservice = app.services.chain
    my_nonce = chainservice.chain.head.get_nonce(my_address)

In the code, we had initialized some global variables: app, representing an object for the entire pyethapp application, chainservice, a service that provides access to the chain, and my_nonce, the contract nonce. The on_start method is called with app as an argument; from this we get the app and the chainservice. From the chainservice, we use chainservice.chain to get the blockchain object, and chainservice.chain.head to get the last block in the chain. From there, we can use all of the methods available to blocks (use dir(chainservice.chain.head) to get a listing), including get_nonce to get the nonce of an account.

Now, the start of the on_block method:

# Called every block
def on_block(blk):
    global my_nonce, chainservice
    for receipt in blk.get_receipts():
        for _log in receipt.logs:
            # Get all logs to the proxy contract address of the right type
            if _log.address == my_contract_address:
                log = ct.listen(_log)
                if log and log["_event_type"] == "GetRequest":
                    < ..... continued later ..... >

The on_block method is called every time a block is added to the chain; from that block, we get the list of receipts, from each receipt the list of logs, then filter them just to get logs that are connected to the proxy contract address. Then, we use the ct object, which is of the ContractTranslator type. We initiated it further up in the code, with:

# ContractTranslator object for the main proxy contract
ct = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "get(string)", "outputs": [{"type": "int256", "name": "out"}], "inputs": [{"type": "string", "name": "url"}]}, {"inputs": [{"indexed": false, "type": "string", "name": "url"}, {"indexed": false, "type": "address", "name": "callback"}, {"indexed": false, "type": "uint256", "name": "responseId"}, {"indexed": false, "type": "uint256", "name": "fee"}], "type": "event", "name": "GetRequest(string,address,uint256,uint256)"}])
# ContractTranslator object for the contract that is used for testing the main contract
ct2 = ethereum.abi.ContractTranslator([{"constant": false, "type": "function", "name": "callback(bytes,uint256)", "outputs": [], "inputs": [{"type": "bytes", "name": "response"}, {"type": "uint256", "name": "responseId"}]}])

A ContractTranslator accepts the ABI declaration of a contract, and returns an object that you can use to "interpret" a log, checking if the log is of the right format and if it is parsing the log topics and data into arguments, as well as encoding function arguments and decoding function return values. It is used in the backend of pyethereum.tester; in general, the methods are:

  • ct.listen(log) - converts a log into human-readable form if possible
  • ct.encode('function_name', [arg1, arg2, arg3]) - converts function arguments into transaction data using the contract ABI
  • ct.decode(bytes) - returns a list of output arguments. In practice, this one is used much less outside of the pyethereum.tester environment because transaction outputs are not logged and are less light-client friendly and so relying on them for applications is discouraged.

The output of ct.listen looks something like this:

{'fetchId': 15, '_event_type': 'LogResponse', 'response': 'test'}

Now, let's look at the rest of the code, which runs if log and log["_event_type"] == "GetRequest":

print 'fetching: ', log["url"]
# Fetch the response
try:
    response = make_request(log["url"])
except:
    response = ''
print 'response: ', response
# Create the response transaction
txdata = ct2.encode('callback', [response, log["responseId"]])
tx = ethereum.transactions.Transaction(my_nonce, 60 * 10**9, min(100000 + log["fee"] / (60 * 10**9), 2500000), log["callback"], 0, txdata).sign(my_privkey)
print 'txhash: ', tx.hash.encode('hex')
print 'tx: ', rlp.encode(tx).encode('hex')
# Increment the nonce so the next transaction is also valid
my_nonce += 1
# Send it
success = chainservice.add_transaction(tx, broadcast_only=True)
assert success
print 'sent tx'

The most important piece of the service is the single line response = make_request(log["url"]): make an HTTP request containing the URL from the log, and set the response to be the response of the HTTP request. If the request fails, set the response to ''. Then, we use ct2.encode to encode the response, and the callback ID from the request log, into transaction data, and then create a transaction. The transaction's arguments are:

  • The current nonce
  • A standard gas fee of 60 shannon
  • An amount of gas that scales with the fee paid; the more you pay the more gas the service sends along with the transaction
  • The address to send the callback to
  • A value of zero
  • The transaction data that we encoded earlier

Then, we increment the my_nonce value so that the next transaction is sent with a nonce one higher, and use chainservice.add_transaction(tx, broadcast_only=True) to broadcast the transaction. Using broadcast_only=True is required in order for this to work; otherwise, you may get a (silent) error with the transaction failing because the service is running while the blockchain is "syncing" the last block.

Now, with both contracts on chain, we can use the pyethapp console to try this out. We'll use the ether price API at https://coinmarketcap-nexuist.rhcloud.com/api/eth.

> responder = "6acc9a6876739e9190d06463196e27b6d37405c6"
> proxy = "d53096b3cf64d4739bb774e0f055653e7f2cd710"
> proxy_abi = [{"constant": false, "type": "function", "name": "call(address,bytes,uint256)", "outputs": [], "inputs": [{"type": "address", "name": "fetcher"}, {"type": "bytes", "name": "url"}, {"type": "uint256", "name": "fetchId"}]}, {"constant": false, "type": "function", "name": "callback(bytes,uint256)", "outputs": [], "inputs": [{"type": "bytes", "name": "response"}, {"type": "uint256", "name": "responseId"}]}, {"inputs": [{"indexed": false, "type": "string", "name": "response"}, {"indexed": false, "type": "uint256", "name": "fetchId"}], "type": "event", "name": "LogResponse(string,uint256)"}]
> t = c.call(proxy, "https://coinmarketcap-nexuist.rhcloud.com/api/eth", 123, startgas=500000, value=7 * 10**15)

After a few seconds you should see output like this:

{'url': 'https://coinmarketcap-nexuist.rhcloud.com/api/eth', 'callback': '6acc9a6876739e9190d06463196e27b6d37405c6', 'fee': 7000000000000000, 'responseId': 99851078186838221073812119751183997484609581204551773443199447993715518157603L, '_event_type': 'GetRequest'}
fetching:  https://coinmarketcap-nexuist.rhcloud.com/api/eth
response:  {
  "symbol": "eth",
  "position": "4",
  "market_cap": {
    "usd": "43716923.7072",
    "eur": "38379743.07640649",
    "cny": "277400930.5224298",
    "cad": "56888963.970950484",
    "rub": "2754379969.3105283",
    "btc": "174982.560604"
  },
  "price": {
    "usd": "0.591439",
    "eur": "0.5192331696850001",
    "cny": "3.752911116210001",
    "cad": "0.7696413450170001",
    "rub": "37.263549136710004",
    "btc": "0.00236731"
  },
  "supply": "73916200",
  "volume": {
    "usd": "363197",
    "eur": "318856.094255",
    "cny": "2304626.6118300003",
    "cad": "472629.34569100005",
    "rub": "22883187.03333",
    "btc": "1453.74"
  },
  "change": "-2.44",
  "timestamp": 1444802709.433
}

txhash:  e9e40253ba0cf1d308b4aa3f65ba573c85e8677abf8a4d68ad33fdb0e0237dfa
tx:  f9038b09850df847580083034e5a946acc9a6876739e9190d06463196e27b6d37405c680b903249305414a0000000000000000000000000000000000000000000000000000000000000040dcc1b51da247e3cccba7f319ace046ec81e00c321977043b0c749703fdad332300000000000000000000000000000000000000000000000000000000000002c07b0a20202273796d626f6c223a2022657468222c0a202022706f736974696f6e223a202234222c0a2020226d61726b65745f636170223a207b0a2020202022757364223a202234333731363932332e37303732222c0a2020202022657572223a202233383337393734332e3037363430363439222c0a2020202022636e79223a20223237373430303933302e35323234323938222c0a2020202022636164223a202235363838383936332e393730393530343834222c0a2020202022727562223a2022323735343337393936392e33313035323833222c0a2020202022627463223a20223137343938322e353630363034220a20207d2c0a2020227072696365223a207b0a2020202022757364223a2022302e353931343339222c0a2020202022657572223a2022302e35313932333331363936383530303031222c0a2020202022636e79223a2022332e373532393131313136323130303031222c0a2020202022636164223a2022302e37363936343133343530313730303031222c0a2020202022727562223a202233372e323633353439313336373130303034222c0a2020202022627463223a2022302e3030323336373331220a20207d2c0a202022737570706c79223a20223733393136323030222c0a202022766f6c756d65223a207b0a2020202022757364223a2022333633313937222c0a2020202022657572223a20223331383835362e303934323535222c0a2020202022636e79223a2022323330343632362e36313138333030303033222c0a2020202022636164223a20223437323632392e3334353639313030303035222c0a2020202022727562223a202232323838333138372e3033333333222c0a2020202022627463223a2022313435332e3734220a20207d2c0a2020226368616e6765223a20222d322e3434222c0a20202274696d657374616d70223a20313434343830323730392e3433330a7d1ca04f76b8f659df5d014d66184ec73476cd4d85781d2214829ec47a5fb6f4bcfb34a076985228148bc695f712615ace21643e7528a2f8312e617bdb9136cbfcbfda76
sent tx

Now, for convenience take the txhash above and set it to the response_txhash variable:

> response_txhash = 'e9e40253ba0cf1d308b4aa3f65ba573c85e8677abf8a4d68ad33fdb0e0237dfa'.decode('hex')

We then use eth.chain.index.get_transaction to retrieve the block and index that the transaction is at.

> tx, blk, index = eth.chain.index.get_transaction(response_txhash)

We can take a quick look at the output:

> tx, blk, index
Out[37]: (<Transaction(e9e4)>, <Block(#381276 800a964f)>, 0)

Now, create a ContractTranslator, get the receipt associated with the transaction (by using blk.get_receipts() to get the list of receipts of the block the transaction is in and then using the index provided to get the correct receipt), and then use the ContractTranslator's listen method to convert the log into human-readable form.

> ct = ethereum.abi.ContractTranslator(response_abi)
> ct.listen(blk.get_receipts()[index].logs[0])
{'fetchId': 19, '_event_type': 'LogResponse', 'response': '{\n  "symbol": "eth",\n  "position": "4",\n  "market_cap": {\n    "usd": "43716923.7072",\n    "eur": "38379743.07640649",\n    "cny": "277400930.5224298",\n    "cad": "56888963.970950484",\n    "rub": "2754379969.3105283",\n    "btc": "174982.560604"\n  },\n  "price": {\n    "usd": "0.591439",\n    "eur": "0.5192331696850001",\n    "cny": "3.752911116210001",\n    "cad": "0.7696413450170001",\n    "rub": "37.263549136710004",\n    "btc": "0.00236731"\n  },\n  "supply": "73916200",\n  "volume": {\n    "usd": "363197",\n    "eur": "318856.094255",\n    "cny": "2304626.6118300003",\n    "cad": "472629.34569100005",\n    "rub": "22883187.03333",\n    "btc": "1453.74"\n  },\n  "change": "-2.44",\n  "timestamp": 1444802709.433\n}'}

And there we go: the output of the GET request to the ether price API.

Further exercises

  • Add a feature that allows you to specify not just a URL, but also a path to descend down a JSON object, and then use that to get a particular number (eg. the ETH/USD price) from the API.
  • Write a string parser in serpent or solidity to convert the string returned by the API into a number.
  • Write a contract which uses this to implement on-demand USD-denominated salary payments: an employee earning $5000 per month (ie. $0.001901 per second) can call the contract in order to immediately receive (t - tp) * 0.001901 / p ether, where t is the current block timestamp, tp is the previous time that employee withdrew, and p is the ether price.
Clone this wiki locally