Solana on Akashnet

This write-up will guide you to deploy Solana node on Akashnet

Solana on Akashnet

This write-up will guide you to deploy Solana node on Akashnet.
I'll try to make it short while keeping all necessary steps available for deployment from scratch.

I must point that this is not a production Solana node deployment, but rather the development deployment due to reasons which can be seen in the end of this write-up.

I used Ubuntu 20.04.2 LTS Linux distribution and the following software:
- akash: v0.12.1 (to work with the Akashnet);
- solana: v1.6.10 (to deploy it on the Akashnet);

Meet your new friends:
- Akashnet is a decentralized cloud of future;
- Solana is a high-performance blockchain supporting builders around the world creating crypto apps that scale today.

The idea is simple - leverage the Akashnet decloud in order to deploy Solana container.

It's up to 86% cheaper to run an instance on Akashnet than on Amazon AWS https://akashlytics.com/price-compare (at the time I wrote this write-up)

Ok, I'm not good at talking much, so to save your and my time let's jump straight to the CLI.

1. Install akash client

AKASH_NET="https://raw.githubusercontent.com/ovrclk/net/master/mainnet"
AKASH_VERSION="$(curl -s "$AKASH_NET/version.txt")"
curl https://raw.githubusercontent.com/ovrclk/akash/master/godownloader.sh | sh -s -- "v$AKASH_VERSION"
Please also install jq tool as it will be used in this write-up.

2. Create your akash wallet

akash keys add default

This command will output the mnemonic phrase, make sure to save it in a safe place.
It will also output your address, in my case the address was akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h.

export AKASH_ACCOUNT_ADDRESS=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
More about the Akash wallet https://docs.akash.network/guides/wallet

3. Fund your akash wallet

You will need about 10 AKT (Akash Token) to get you started.
You can purchase them at one of the exchanges mentioned here https://akash.network/token/

You can query the balance of your wallet

export AKASH_NET="https://raw.githubusercontent.com/ovrclk/net/master/mainnet"
export AKASH_NODE="$(curl -s "$AKASH_NET/rpc-nodes.txt" | shuf -n 1)"

akash \
  --node "$AKASH_NODE" \
  query bank balances "$AKASH_ACCOUNT_ADDRESS"
Denomination: 1 akt = 1000000 uakt (akt*10^6), or you can add -o json | jq -r '(.balances[0].amount | tonumber / pow(10; 6))' on the command line ;-)
I'm using production akashnet-2 Akash network. For more on different Akash networks please refer to https://docs.akash.network/guides/version

4. Create certificate file

To create a deployment, a certificate must first be created. To do this, run:

export AKASH_CHAIN_ID="$(curl -s "$AKASH_NET/chain-id.txt")"
akash tx cert create client \
  --from default \
  --chain-id $AKASH_CHAIN_ID \
  --node $AKASH_NODE \
  --gas-prices="0.025uakt" --gas="auto" --gas-adjustment=1.15
certificate needs to be created only once per account and can be used across all deployments. More about the certificate https://docs.akash.network/design/mtls

5. Prepare the deployment manifest file

Create a file named solana.yml with the following contents:

---
version: "2.0"

services:
  solana:
    image: solanalabs/solana:v1.6.10
    expose:
      - port: 8899
        as: 8899
        proto: tcp
        to:
          - global: true

profiles:
  compute:
    solana:
      resources:
        cpu:
          units: 0.1
        memory:
          size: 512Mi
        storage:
          size: 512Mi
  placement:
    akash:
      attributes:
        host: akash
      signedBy:
        anyOf:
          - "akash1365yvmc4s7awdyj3n2sav7xfx76adc6dnmlx63"
      pricing:
        solana: 
          denom: uakt
          amount: 10

deployment:
  solana:
    akash:
      profile: solana
      count: 1

Pricing - Note that there are bids from multiple different providers. In this case, both providers happen to be willing to accept a price of 1 uAKT. This means that the lease can be created using 1 uAKT or 0.000001 AKT per block to execute the container.
pricing sets the limits so providers know how to bid on your deployment. i.e. amount of uakt per block (each block happens every 6 seconds) you are willing to pay.
So in case with 1uakt, you are going to be paying ((1*((60/6)*60*24*30.436875))/10^6)*3.28 = $1.44 per month having 1akt worth $3.28 as of 31 May 2021.

signedBy must be akash1365yvmc4s7awdyj3n2sav7xfx76adc6dnmlx63 as in the doc https://docs.akash.network/guides/deploy
signedBy is a "validation" thing, after a provider is approved by akash they get signed by akash to ensure security and reliability.

More on SDL (Stack Definition Language) used to write the Akash manifest https://docs.akash.network/documentation/sdl

6. Create your deployment

akash tx deployment create solana.yml \
  --from default \
  --node $AKASH_NODE \
  --chain-id $AKASH_CHAIN_ID \
  --gas-prices="0.025uakt" --gas="auto" --gas-adjustment=1.15

If this command ends with Error: RPC error -32603 - Internal error: timed out waiting for tx to be included in a block then no worries, you can query the akash blockchain in order to find your deployment ID:

$ akash query txs \
  --events "message.sender=$AKASH_ACCOUNT_ADDRESS&message.action=create-deployment" \
  --page 1 --limit 999999 \
  | jq -r '.txs[] | [ .tx.body.messages[].id[] ] | @csv' | \
  tail -5
"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h","1172777"

Query the deployment:

I'm explicitly not "export"'ing the following 3 "SEQ" variables as to avoid 'failed to execute message; message index: 0: Deployment closed' error on 'akash tx deployment create' later. If you get this error, then simply unset AKASH_DSEQ.
$ AKASH_DSEQ=1172777
$ AKASH_GSEQ=1
$ AKASH_OSEQ=1

$ akash query deployment get \
  --owner $AKASH_ACCOUNT_ADDRESS \
  --node $AKASH_NODE \
  --dseq $AKASH_DSEQ

deployment:
  created_at: "1172779"
  deployment_id:
    dseq: "1172777"
    owner: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
  state: active
  version: R+bMsK3+N7Wtf8MWmUqoIJTgagXPunxFJqV4EXZahYE=
escrow_account:
  balance:
    amount: "5000000"
    denom: uakt
  id:
    scope: deployment
    xid: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/1172777
  owner: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
  settled_at: "1172791"
  state: open
  transferred:
    amount: "0"
    denom: uakt
groups:
- created_at: "1172779"
  group_id:
    dseq: "1172777"
    gseq: 1
    owner: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
  group_spec:
    name: akash
    requirements:
      attributes:
      - key: host
        value: akash
      signed_by:
        all_of: []
        any_of:
        - akash1365yvmc4s7awdyj3n2sav7xfx76adc6dnmlx63
    resources:
    - count: 1
      price:
        amount: "10"
        denom: uakt
      resources:
        cpu:
          attributes: []
          units:
            val: "100"
        endpoints:
        - kind: RANDOM_PORT
        memory:
          attributes: []
          quantity:
            val: "536870912"
        storage:
          attributes: []
          quantity:
            val: "536870912"
  state: open

As you may have noticed the 5 AKT are now under the escrow_account and your akash wallet balance was reduced by that value. These tokens are going to be used by this deployment as soon as you create a lease. And they will be returned as soon as you close the lease / deployment (minus few of them which were used for the lease itself).

More on deployment https://docs.akash.network/guides/deploy

7. Query the market for bids

Now that we want to actually run our deployment, we will need to see what market offers, accept the bid we feel comfortable with, create the lease and send manifest of our deployment to get it up and running.

$ akash query market bid list \
  --owner=$AKASH_ACCOUNT_ADDRESS \
  --node $AKASH_NODE \
  --dseq $AKASH_DSEQ \
  --state open

bids:
- bid:
    bid_id:
      dseq: "1172777"
      gseq: 1
      oseq: 1
      owner: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
      provider: akash10cl5rm0cqnpj45knzakpa4cnvn5amzwp4lhcal
    created_at: "1172780"
    price:
      amount: "1"
      denom: uakt
    state: open
  escrow_account:
    balance:
      amount: "50000000"
      denom: uakt
    id:
      scope: bid
      xid: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/1172777/1/1/akash10cl5rm0cqnpj45knzakpa4cnvn5amzwp4lhcal
    owner: akash10cl5rm0cqnpj45knzakpa4cnvn5amzwp4lhcal
    settled_at: "1172780"
    state: open
    transferred:
      amount: "0"
      denom: uakt
- bid:
    bid_id:
      dseq: "1172777"
      gseq: 1
      oseq: 1
      owner: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h
      provider: akash1f6gmtjpx4r8qda9nxjwq26fp5mcjyqmaq5m6j7
    created_at: "1172780"
    price:
      amount: "1"
      denom: uakt
    state: open
  escrow_account:
    balance:
      amount: "50000000"
      denom: uakt
    id:
      scope: bid
      xid: akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/1172777/1/1/akash1f6gmtjpx4r8qda9nxjwq26fp5mcjyqmaq5m6j7
    owner: akash1f6gmtjpx4r8qda9nxjwq26fp5mcjyqmaq5m6j7
    settled_at: "1172780"
    state: open
    transferred:
      amount: "0"
      denom: uakt
pagination:
  next_key: null
  total: "0"

Set the AKASH_PROVIDER environment variable to the one you picked from the available market bids:

AKASH_PROVIDER=akash10cl5rm0cqnpj45knzakpa4cnvn5amzwp4lhcal

If akash query market bid list isn't showing any bids, make sure you unset AKASH_PROVIDER environment variable!


8. Create lease / Accept the bid

Create a lease for the bid from the chosen provider above by running:

akash tx market lease create \
  --chain-id $AKASH_CHAIN_ID \
  --node $AKASH_NODE \
  --owner $AKASH_ACCOUNT_ADDRESS \
  --dseq $AKASH_DSEQ \
  --gseq $AKASH_GSEQ \
  --oseq $AKASH_OSEQ \
  --provider $AKASH_PROVIDER \
  --from default \
  --gas-prices="0.025uakt" --gas="auto" --gas-adjustment=1.15
Please note that once the lease is created, the provider will begin debiting your deployment's escrow account, even if you have not completed the deployment process by uploading the manifest in the following step.

9. Send the manifest to deploy Solana node

Now, to get your Solana deployment started, you need to send your manifest to the provider:

akash provider send-manifest solana.yml \
  --node $AKASH_NODE \
  --dseq $AKASH_DSEQ \
  --provider $AKASH_PROVIDER \
  --from default

After few moments, you will be able to see your deployment:

$ akash provider lease-status \
  --node $AKASH_NODE \
  --dseq $AKASH_DSEQ \
  --provider $AKASH_PROVIDER \
  --from default
{
  "services": {
    "solana": {
      "name": "solana",
      "available": 1,
      "total": 1,
      "uris": null,
      "observed_generation": 1,
      "replicas": 1,
      "updated_replicas": 1,
      "ready_replicas": 1,
      "available_replicas": 1
    }
  },
  "forwarded_ports": {
    "solana": [
      {
        "host": "cluster.sjc1p0.mainnet.akashian.io",
        "port": 8899,
        "externalPort": 32509,
        "proto": "TCP",
        "available": 1,
        "name": "solana"
      }
    ]
  }
}

The endpoint URI is cluster.sjc1p0.mainnet.akashian.io:32509,
now why the port is 32509 rather than 8899 you will find out in the end of this write-up (see nodePort).

10. See the logs of your Solana container on Akashnet

You can see your Solana node's mnemonic seed phrase there.

$ akash \
  --node "$AKASH_NODE" \
  provider lease-logs \
  --dseq "$AKASH_DSEQ" \
  --gseq "$AKASH_GSEQ" \
  --oseq "$AKASH_OSEQ" \
  --provider "$AKASH_PROVIDER" \
  --from default \
  --follow
[solana-648bbc8979-hrjq6] solana-faucet 1.6.10 (src:5d4654d2; feat:3533521759)
[solana-648bbc8979-hrjq6] solana-genesis 1.6.10 (src:5d4654d2; feat:3533521759)
[solana-648bbc8979-hrjq6] solana-keygen 1.6.10 (src:5d4654d2; feat:3533521759)
[solana-648bbc8979-hrjq6] solana-validator 1.6.10 (src:5d4654d2; feat:3533521759)
[solana-648bbc8979-hrjq6] + solana address
[solana-648bbc8979-hrjq6] Error: No such file or directory (os error 2)
[solana-648bbc8979-hrjq6] + echo Generating default keypair
[solana-648bbc8979-hrjq6] Generating default keypair
[solana-648bbc8979-hrjq6] + solana-keygen new --no-passphrase
[solana-648bbc8979-hrjq6] Generating a new keypair
[solana-648bbc8979-hrjq6] Wrote new keypair to /root/.config/solana/id.json
[solana-648bbc8979-hrjq6] ==============================================================================
[solana-648bbc8979-hrjq6] pubkey: 2tpqeMBuqQNd2F6pmQmb2Dyj1A6k2yqqCnWXuWh2pX14
[solana-648bbc8979-hrjq6] ==============================================================================
[solana-648bbc8979-hrjq6] Save this seed phrase to recover your new keypair:
[solana-648bbc8979-hrjq6] wagon layer regret misery divorce wild noodle rent actress reflect sister lift
[solana-648bbc8979-hrjq6] ==============================================================================
...
[solana-648bbc8979-hrjq6] [2021-05-31T13:06:27.045978808Z INFO  solana_core::cluster_info] 
[solana-648bbc8979-hrjq6]     IP Address        |Age(ms)| Node identifier                              | Version |Gossip| TPU  |TPUfwd| TVU  |TVUfwd|Repair|ServeR|ShredVer
[solana-648bbc8979-hrjq6]     ------------------+-------+----------------------------------------------+---------+------+------+------+------+------+------+------+--------
[solana-648bbc8979-hrjq6]     127.0.0.1       me|  4558 | 3QDTDmtqV3rLDdkLVPzP5RLg9TdXLFA7QPzv6Eeo72fi | 1.6.10  | 8001 | 8003 | 8004 | 8000 | 8002 | 8006 | 8007 | 48122
[solana-648bbc8979-hrjq6]     Nodes: 1
[solana-648bbc8979-hrjq6]     
[solana-648bbc8979-hrjq6]     RPC Address       |Age(ms)| Node identifier                              | Version | RPC  |PubSub|ShredVer
[solana-648bbc8979-hrjq6]     ------------------+-------+----------------------------------------------+---------+------+------+--------
[solana-648bbc8979-hrjq6]     127.0.0.1       me|  4558 | 3QDTDmtqV3rLDdkLVPzP5RLg9TdXLFA7QPzv6Eeo72fi | 1.6.10  | 8899 | 8900 | 48122
[solana-648bbc8979-hrjq6]     RPC Enabled Nodes: 1
...

11. Query your Solana node on Akashnet

Using Solana's json RPC API you can query your Solana node:

$ curl -s http://cluster.sjc1p0.mainnet.akashian.io:32509 -X POST -H "Content-Type: application/json" -d '
  {"jsonrpc":"2.0","id":1, "method":"getBlockTime","params":[5]}
' | jq -r '.result | todate'
2021-05-31T13:05:39Z

$ curl -s http://cluster.sjc1p0.mainnet.akashian.io:32509 -X POST -H "Content-Type: application/json" -d '
  {"jsonrpc":"2.0","id":1, "method":"getIdentity"}
'
{"jsonrpc":"2.0","result":{"identity":"3QDTDmtqV3rLDdkLVPzP5RLg9TdXLFA7QPzv6Eeo72fi"},"id":1}

$ curl -s http://cluster.sjc1p0.mainnet.akashian.io:32509 -X POST -H "Content-Type: application/json" -d '
  {"jsonrpc":"2.0", "id":1, "method":"getClusterNodes"}
' | jq .
{
  "jsonrpc": "2.0",
  "result": [
    {
      "featureSet": 3533521759,
      "gossip": "127.0.0.1:8001",
      "pubkey": "3QDTDmtqV3rLDdkLVPzP5RLg9TdXLFA7QPzv6Eeo72fi",
      "rpc": "127.0.0.1:8899",
      "shredVersion": 48122,
      "tpu": "127.0.0.1:8003",
      "version": "1.6.10"
    }
  ],
  "id": 1
}

12. Terminating the deployment

When you are done testing, you can terminate  your deployment.

akash tx deployment close \
  --node $AKASH_NODE \
  --chain-id $AKASH_CHAIN_ID \
  --dseq $AKASH_DSEQ \
  --owner $AKASH_ACCOUNT_ADDRESS \
  --from default \
  --gas-prices="0.025uakt" --gas="auto" --gas-adjustment=1.15

13. Depositing more AKT for the deployment

If you are happy with your deployment and would not want it to stop one day, you can deposit more AKT tokens for the escrow account:

akash tx deployment deposit \
  --from default \
  --chain-id $AKASH_CHAIN_ID \
  --node $AKASH_NODE 1000000uakt \
  --dseq $AKASH_DSEQ \
  --gas-prices="0.025uakt" --gas="auto" --gas-adjustment=1.15

14. Production Solana validator node

For production Solana validator node deployment, you will need to modify your manifest file as follows, likely also lifting the cpu, memory, storage limits as recommended.

---
version: "2.0"

services:
  solana:
    image: solanalabs/solana:v1.6.10
    expose:
      - port: 8899
        as: 8899
        proto: tcp
        to:
          - global: true
      - port: 8001
        as: 8001
        proto: tcp
        to:
          - global: true
      - port: 8900
        as: 8900
        proto: tcp
        to:
          - global: true
      - port: 8010
        as: 8010
        proto: udp
        to:
          - global: true
      - port: 8011
        as: 8011
        proto: udp
        to:
          - global: true
      - port: 8012
        as: 8012
        proto: udp
        to:
          - global: true
      - port: 8013
        as: 8013
        proto: udp
        to:
          - global: true
      - port: 8014
        as: 8014
        proto: udp
        to:
          - global: true
      - port: 8015
        as: 8015
        proto: udp
        to:
          - global: true
      - port: 8016
        as: 8016
        proto: udp
        to:
          - global: true
      - port: 8017
        as: 8017
        proto: udp
        to:
          - global: true
      - port: 8018
        as: 8018
        proto: udp
        to:
          - global: true
      - port: 8019
        as: 8019
        proto: udp
        to:
          - global: true
      - port: 8020
        as: 8020
        proto: udp
        to:
          - global: true
    env:
      - SOLANA_RUN_SH_CLUSTER_TYPE=mainnet-beta
      - |
        SOLANA_RUN_SH_VALIDATOR_ARGS=
        --trusted-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2
        --trusted-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ
        --trusted-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ
        --trusted-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S
        --no-untrusted-rpc
        --private-rpc
        --entrypoint entrypoint.mainnet-beta.solana.com:8001
        --entrypoint entrypoint2.mainnet-beta.solana.com:8001
        --entrypoint entrypoint3.mainnet-beta.solana.com:8001
        --entrypoint entrypoint4.mainnet-beta.solana.com:8001
        --entrypoint entrypoint5.mainnet-beta.solana.com:8001
        --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d
        --wal-recovery-mode skip_any_corrupted_record
        --limit-ledger-size
        --dynamic-port-range 8010-8020

profiles:
  compute:
    solana:
      resources:
        cpu:
          units: 0.1
        memory:
          size: 512Mi
        storage:
          size: 512Mi
  placement:
    akash:
      attributes:
        host: akash
      signedBy:
        anyOf:
          - "akash1365yvmc4s7awdyj3n2sav7xfx76adc6dnmlx63"
      pricing:
        solana: 
          denom: uakt
          amount: 10

deployment:
  solana:
    akash:
      profile: solana
      count: 1

You will likely need to expose more ports, such as the ones specified in --dynamic-port-range 8010-8020 argument.

The only caveat is that it is not possible to export non-http/https (80/443) TCP ports, so-called nodePort ports such as 8899/tcp, 8010:8020/udp directly as-is.
The Akashnet providers use Kubernetes under the hood for actually running your containers. The Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767)(nodePort)
This means the operator will have to first deploy the Solana node and then check what ports the provider's Kubernetes assigned to your Akash deployment.
And then use some load balancer / reverse proxy (i.e. nginx / haprorxy / traefik) to forward to these ports.
You would also need to tell Solana validator to use load-balancer's hostname by specifying it via the --public-rpc-address <HOST:PORT> argument to Solana validator node.

However, the goal was to show the Akashnet rather than the complete production Solana node.
The official Solana documentation is not recommending Docker for the production deployment and even despite that the production Solana node should be running on a 12cores, 128Gi RAM, 500Gi+ SSD, Akashnet is not currently providing yet since it's a very new network and there aren't that many providers yet. Not an investment advice.

If you liked this article, please vote for it here https://forum.akash.network/t/twitter-deploy-challenge-solana-on-akash/160

P.S.
Few notes to myself (or to anyone else who wants to pick-up from here and push it further):