KPI 36.OP-1: query-node setup

working-groups/operations-group/Giza-testing/.env

@@ -0,0 +1,62 @@
+COMPOSE_PROJECT_NAME=joystream
+PROJECT_NAME=query_node
+
+# We will use a single postgres service with multiple databases
+# The env variables below are by default used by all services and should be
+# overriden in local env files
+# DB config
+INDEXER_DB_NAME=query_node_indexer
+DB_NAME=query_node_processor
+DB_USER=postgres
+DB_PASS=postgres
+# This value will not be used by query-node docker containers.
+# When running query-node with docker these services will always use the db service
+DB_HOST=localhost
+DB_PORT=5432
+DEBUG=index-builder:*
+TYPEORM_LOGGING=error
+
+## Indexer options
+# Block height to start indexing from.
+# Note, that if there are already some indexed events, this setting is ignored
+BLOCK_HEIGHT=13876
+
+# Query node GraphQL server port
+GRAPHQL_SERVER_PORT=8081
+
+# Hydra indexer gateway GraphQL server port
+HYDRA_INDEXER_GATEWAY_PORT=4000
+
+# Default GraphQL server host. It is required during "query-node config:dev"
+GRAPHQL_SERVER_HOST=localhost
+
+# Websocket RPC endpoint containers will use.
+JOYSTREAM_NODE_WS=ws://localhost:9944
+
+# Query node which colossus will use
+COLOSSUS_QUERY_NODE_URL=http://graphql-server:4000/graphql
+
+# Query node which distributor will use
+DISTRIBUTOR_QUERY_NODE_URL=http://graphql-server:4000/graphql
+
+# Indexer gateway used by processor. If you don't use the local indexer set this to a remote gateway
+PROCESSOR_INDEXER_GATEWAY=http://hydra-indexer-gateway:4000/graphql
+
+# Colossus services identities
+COLOSSUS_1_WORKER_ID=0
+COLOSSUS_1_WORKER_URI=//testing//worker//Storage//${COLOSSUS_1_WORKER_ID}
+COLOSSUS_1_TRANSACTOR_URI=//Colossus1
+
+COLOSSUS_2_WORKER_ID=1
+COLOSSUS_2_WORKER_URI=//testing//worker//Storage//${COLOSSUS_2_WORKER_ID}
+COLOSSUS_2_TRANSACTOR_URI=//Colossus2
+
+# Distributor node services identities
+DISTRIBUTOR_1_WORKER_ID=0
+DISTRIBUTOR_1_ACCOUNT_URI=//testing//worker//Distribution//${DISTRIBUTOR_1_WORKER_ID}
+
+DISTRIBUTOR_2_WORKER_ID=1
+DISTRIBUTOR_2_ACCOUNT_URI=//testing//worker//Distribution//${DISTRIBUTOR_2_WORKER_ID}
+
+# joystream/node docker image tag
+JOYSTREAM_NODE_TAG=latest

working-groups/operations-group/Giza-testing/README.md

@@ -1,6 +1,6 @@
 # [KPI 36.OP-1](https://blog.joystream.org/sumer-kpis/#36.OP-1)
 
-The goal is to install a [GIZA](https://github.com/Joystream/joystream/tree/giza_staging) validator, storage provider and distributor on a fresh VPS.
+The goal is to install a [GIZA](https://github.com/Joystream/joystream/tree/giza_staging) validator, query node, storage provider and distributor on a fresh VPS.
 
 ## Setup
 
@@ -8,9 +8,33 @@ The goal is to install a [GIZA](https://github.com/Joystream/joystream/tree/giza
 
 See [Network deployment](../Network-deployment)
 
+### [Query-node](https://github.com/Joystream/joystream/tree/giza_staging/query-node)
+
+Download these to `~/joystream` (or your repository root):
+- [.env](.env)
+- [import-qn-data.sh](import-qn-data.sh)
+- [deploy-qn.sh](deploy-qn.sh)
+- [reset-qn.sh](reset-qn.sh)
+
+Adapt `JOYSTREAM_NODE_WS` in `.env` and `import-qn-data.sh` if necessary.
+
+```
+cd ~/joystream/query-node
+yarn build
+nano manifest.yml
+# change second to last line to:
+---
+           height: "[13876,13876]"
+---
+cd ..
+sudo docker-compose up
+chmod +x import-qn-data.sh deploy-qn.sh reset-qn.sh
+./reset-qn.sh
+```
+
 ### Storage
 
-See [storage-node](https://github.com/Joystream/joystream/tree/giza_staging/storage-node-v2).
+See [storage-node](https://github.com/Joystream/joystream/tree/giza_staging/storage-node).
 
 ### Distributor
 
@@ -364,3 +388,6 @@ No help:
 | Jan 2 13-17 |     4 | start node, storage provider, distributor      |
 | Jan 4 15-18 |     3 | storage provider and distributor configuration |
 | Jan 4 19-22 |     3 | check commands, validator eras                 |
+| Jan 5 20-23 |     3 | nginx config                                   |
+| Jan 6 18-23 |     5 | content creation                               |
+| Jan 7 12-15 |     3 | query-node deployment                          |

working-groups/operations-group/Giza-testing/deploy-qn.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -e
+
+SCRIPT_PATH="$(dirname "${BASH_SOURCE[0]}")"
+cd $SCRIPT_PATH
+
+# Bring up db
+docker-compose up -d db
+
+# Make sure we use dev config for db migrations (prevents "Cannot create database..." and some other errors)
+docker-compose run --rm --entrypoint sh graphql-server -c "yarn workspace query-node config:dev"
+# Migrate the databases
+docker-compose run --rm --entrypoint sh graphql-server -c "yarn workspace query-node-root db:prepare"
+docker-compose run --rm --entrypoint sh graphql-server -c "yarn workspace query-node-root db:migrate"
+
+# Start indexer and gateway
+docker-compose up -d indexer
+docker-compose up -d hydra-indexer-gateway
+
+# Start processor and graphql server
+docker-compose up -d processor
+docker-compose up -d graphql-server

working-groups/operations-group/Giza-testing/import-qn-data.sh

@@ -0,0 +1,68 @@
+COMPOSE_PROJECT_NAME=joystream
+PROJECT_NAME=query_node
+
+# We will use a single postgres service with multiple databases
+# The env variables below are by default used by all services and should be
+# overriden in local env files
+# DB config
+INDEXER_DB_NAME=query_node_indexer
+DB_NAME=query_node_processor
+DB_USER=postgres
+DB_PASS=postgres
+# This value will not be used by query-node docker containers.
+# When running query-node with docker these services will always use the db service
+DB_HOST=localhost
+DB_PORT=5432
+DEBUG=index-builder:*
+TYPEORM_LOGGING=error
+
+## Indexer options
+# Block height to start indexing from.
+# Note, that if there are already some indexed events, this setting is ignored
+BLOCK_HEIGHT=13876
+#BLOCK_HEIGHT=200000
+
+# Query node GraphQL server port
+GRAPHQL_SERVER_PORT=8081
+
+# Hydra indexer gateway GraphQL server port
+HYDRA_INDEXER_GATEWAY_PORT=4000
+
+# Default GraphQL server host. It is required during "query-node config:dev"
+GRAPHQL_SERVER_HOST=localhost
+
+# Websocket RPC endpoint containers will use.
+#JOYSTREAM_NODE_WS=ws://joystream-node:9944/
+JOYSTREAM_NODE_WS=ws://localhost:9944
+
+# Query node which colossus will use
+#COLOSSUS_QUERY_NODE_URL=http://graphql-server:${GRAPHQL_SERVER_PORT}/graphql
+COLOSSUS_QUERY_NODE_URL=http://graphql-server:4000/graphql
+
+
+# Query node which distributor will use
+#DISTRIBUTOR_QUERY_NODE_URL=http://graphql-server:${GRAPHQL_SERVER_PORT}/graphql
+DISTRIBUTOR_QUERY_NODE_URL=http://graphql-server:4000/graphql
+
+# Indexer gateway used by processor. If you don't use the local indexer set this to a remote gateway
+#PROCESSOR_INDEXER_GATEWAY=http://hydra-indexer-gateway:${HYDRA_INDEXER_GATEWAY_PORT}/graphql
+PROCESSOR_INDEXER_GATEWAY=http://hydra-indexer-gateway:4000/graphql
+
+# Colossus services identities
+COLOSSUS_1_WORKER_ID=0
+COLOSSUS_1_WORKER_URI=//testing//worker//Storage//${COLOSSUS_1_WORKER_ID}
+COLOSSUS_1_TRANSACTOR_URI=//Colossus1
+
+COLOSSUS_2_WORKER_ID=1
+COLOSSUS_2_WORKER_URI=//testing//worker//Storage//${COLOSSUS_2_WORKER_ID}
+COLOSSUS_2_TRANSACTOR_URI=//Colossus2
+
+# Distributor node services identities
+DISTRIBUTOR_1_WORKER_ID=0
+DISTRIBUTOR_1_ACCOUNT_URI=//testing//worker//Distribution//${DISTRIBUTOR_1_WORKER_ID}
+
+DISTRIBUTOR_2_WORKER_ID=1
+DISTRIBUTOR_2_ACCOUNT_URI=//testing//worker//Distribution//${DISTRIBUTOR_2_WORKER_ID}
+
+# joystream/node docker image tag
+JOYSTREAM_NODE_TAG=latest

working-groups/operations-group/Giza-testing/reset-qn.sh

@@ -0,0 +1,8 @@
+docker-compose down -v
+service docker stop
+rm -rf /var/lib/docker/volumes/joystream_query-node-data
+service docker start
+./import-qn-data.sh
+for file in $(rgrep -l mokhtar query-node/); do sed -i s/"\\\u0000"/""/g $file ; done
+./deploy-qn.sh
+docker logs -f -n 100 processor