DWG Handover #722

working-groups/distributors/README.md

@@ -1,31 +1,33 @@
 # Distribution Working Group
 
-With [release of the Giza Testnet](https://blog.joystream.org/giza-released/) selected channels were [migrated from Sumer](https://github.com/traumschule/community-repo/tree/patch-2/working-groups/distributors/Sumer-Giza-migration) previous IPFS based storage system to complete in-house hosting with [storage nodes](https://github.com/Joystream/helpdesk/tree/master/roles/distributors) for bigger portions of assets and local [distribution nodes](https://github.com/Joystream/helpdesk/tree/master/roles/distributors) (read first).
-
-![Giza Testnet](img/joystream-org-giza.png)
-
-Details about inception of the group are reported [here](https://pioneer.joystreamstats.live/#/forum/threads/928?replyIdx=12) (@freakstatic [assigned](https://pioneer.joystreamstats.live/#/forum/threads/917?replyIdx=14&page=2) this task to @l1dev who handed it @ilich).
+- [Intro for Leads](skillshare.md)
 
-- Lead: [@l1dev](https://pioneer.joystreamstats.live/#/proposals/1112) ([Concept](Distribution_Concept.md))
-- Contact: [#distributors](https://discord.com/channels/811216481340751934/933726271832227911)
+## Tasks
+- **Bounty**: [Collect info for group reports](https://discord.com/channels/811216481340751934/943152435432669266/960206661753143367)
+- Develop tools to implement [Quality of Service](giza1.md#quality-of-service), https://github.com/Joystream/community-repo/issues/656
+- Deputy role: Based on available buckets, which channels should be distributed where?
+- Create DP map: [discussion](https://discord.com/channels/811216481340751934/813361923172335648/939636962925641868)
 
+![Telemtry](img/telemetry-map.png)
 
-## Openings
+Image source: https://telemetry.joystream.org/#/Joystream
 
-- [Apply](https://pioneer.joystreamstats.live/#/working-groups/opportunities/distributionWorkingGroup): [Distributor in Central and South America](https://discord.com/channels/811216481340751934/933726271832227911/938557848294678598)
+## Olympia
 
-- Task: Develop tools to implement [Quality of Service](giza1.md#quality-of-service)
+Reports now on [Notion](https://www.notion.so/joystream/Distribution-1f4cfbbb2e934c79bf20b8db7f019d32)
 
-- Deputy role: Based on available buckets, which channels should be distributed where? Weekly update per bucket: `0-0: number[]`
+## Giza
 
-- Create DP map: [discussion](https://discord.com/channels/811216481340751934/813361923172335648/939636962925641868)
+With [release of the Giza Testnet](https://blog.joystream.org/giza-released/) selected channels were [migrated from Sumer](https://github.com/traumschule/community-repo/tree/patch-2/working-groups/distributors/Sumer-Giza-migration) previous IPFS based storage system to complete in-house hosting with [storage nodes](https://github.com/Joystream/helpdesk/tree/master/roles/distributors) for bigger portions of assets and local [distribution nodes](https://github.com/Joystream/helpdesk/tree/master/roles/distributors) (read first).
 
-![Telemtry](img/telemetry-map.png)
+![Giza Testnet](img/joystream-org-giza.png)
 
-Image source: https://telemetry.joystream.org/#/Joystream
+Details about inception of the group are reported [here](https://pioneer.joystreamstats.live/#/forum/threads/928?replyIdx=12) (@freakstatic [assigned](https://pioneer.joystreamstats.live/#/forum/threads/917?replyIdx=14&page=2) this task to @l1dev who handed it @ilich).
 
+- Lead: [@l1dev](https://pioneer.joystreamstats.live/#/proposals/1112) ([Concept](Distribution_Concept.md))
+- Contact: [#distributors](https://discord.com/channels/811216481340751934/933726271832227911)
 
-## Reports
+### Reports
 
 - [Giza 42](giza1.md)
 
@@ -60,7 +62,7 @@ Do not miss to try below queries.
 
 #### Sorting queries
 
-````
+```
 orderBy: [DistributionBucketOperatorOrderByInput!]
 enum DistributionBucketOperatorOrderByInput {
 createdAt_ASC
@@ -96,6 +98,7 @@ Filters need to be applied to the first (or second) level (query node doesn't su
 ```
 query { storageDataObjects(where: { id_eq: "9506" }) {
     id storageBag {
+      id
       distributionBuckets {
         id operators {
           metadata { nodeEndpoint } } } } } }

working-groups/distributors/bucketFamilies/README.md

@@ -0,0 +1,3 @@
+# Bucket Family Templates
+
+Templates for [bucket families](https://github.com/Joystream/helpdesk/tree/master/roles/distributors-lead#set-bucket-family-metadata) used during Giza.

working-groups/distributors/bucketFamilies/distribution-family2.json

@@ -0,0 +1,13 @@
+{
+    "region": "ru_eu-east",
+    "description": "Russia and eastern Europe",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AS"
+        },
+        {
+            "continentCode": "EU"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family3.json

@@ -0,0 +1,10 @@
+{
+    "region": "us-east",
+    "description": "North America (east)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family4.json

@@ -0,0 +1,10 @@
+{
+    "region": "south-america",
+    "description": "Central and South America",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "SA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family5.json

@@ -0,0 +1,10 @@
+{
+    "region": "africa-north",
+    "description": "North Africa",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AF"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family6.json

@@ -0,0 +1,10 @@
+{
+    "region": "asia-east",
+    "description": "East and South East Asia",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AS"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family7.json

@@ -0,0 +1,10 @@
+{
+    "region": "australia-oceania",
+    "description": "Australia and Oceania",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "OC"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family8.json

@@ -0,0 +1,10 @@
+{
+    "region": "us-west",
+    "description": "North America (west)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/distribution-family9.json

@@ -0,0 +1,10 @@
+{
+    "region": "us-central",
+    "description": "North America (central)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/0.json

@@ -0,0 +1,10 @@
+{
+    "region": "eu-1",
+    "description": "Northern Europe",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "EU"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/1.json

@@ -0,0 +1,10 @@
+{
+    "region": "eu-2",
+    "description": "Central Europe",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "EU"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/10.json

@@ -0,0 +1,10 @@
+{
+    "region": "na-1",
+    "description": "North America (W)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/11.json

@@ -0,0 +1,10 @@
+{
+    "region": "na-2",
+    "description": "North America (Central)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/12.json

@@ -0,0 +1,10 @@
+{
+    "region": "na-3",
+    "description": "North America (E)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "NA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/2.json

@@ -0,0 +1,13 @@
+{
+    "region": "as-1",
+    "description": "NE Asia / Eastern Europe",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AS"
+        },
+        {
+            "continentCode": "EU"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/3.json

@@ -0,0 +1,10 @@
+{
+    "region": "as-2",
+    "description": "Eastern Asia",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AS"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/4.json

@@ -0,0 +1,10 @@
+{
+    "region": "as-3",
+    "description": "SE Asia",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AS"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/5.json

@@ -0,0 +1,13 @@
+{
+    "region": "af-1",
+    "description": "Northern Africa / Southern Europe",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AF"
+        },
+        {
+            "continentCode": "EU"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/6.json

@@ -0,0 +1,10 @@
+{
+    "region": "af-2",
+    "description": "Southern Africa",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "AF"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/7.json

@@ -0,0 +1,10 @@
+{
+    "region": "oc",
+    "description": "Australia / Oceania",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "OC"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/8.json

@@ -0,0 +1,10 @@
+{
+    "region": "sa-1",
+    "description": "South America (S)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "SA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/9.json

@@ -0,0 +1,10 @@
+{
+    "region": "sa-2",
+    "description": "Central + South America (N)",
+    "latencyTestTargets": [],
+    "areas": [
+        {
+            "continentCode": "SA"
+        }
+    ]
+}

working-groups/distributors/bucketFamilies/olympia/set-all.sh

@@ -0,0 +1,16 @@
+set -e
+dir=$(dirname $0)
+max=12
+
+# Prepare
+# 1. Update `max` when files were added
+# 2. `chmod +x ~/community-repo/working-groups-distributors/bucketFamilies/olympia/set-all.sh`
+# 3. Set lead key in joystream/distributor-node/config.yml
+
+# Run
+# 1. `cd joystream/distributor-node`
+# 3. `~/community-repo/working-groups-distributors/bucketFamilies/olympia/set-all.sh`
+
+for f in $(seq 0 $max) ; do
+  yarn joystream-distributor leader:set-bucket-family-metadata -f $f -i $dir/$f.json -y
+done

working-groups/distributors/olympia.md

@@ -0,0 +1,69 @@
+# Olympia Distribution WG
+
+https://dao.joystream.org/#/working-groups/distribution
+
+- Lead: [l1dev](https://dao.joystream.org/#/proposals/preview/13) ([Applications](https://dao.joystream.org/#/working-groups/openings/distributionWorkingGroup-0))
+- Budget: [Proposal](https://dao.joystream.org/#/proposals/preview/18) ([Discussion](https://discord.com/channels/811216481340751934/812343711870091285/957682269475201025))
+
+## Bucket Families
+
+| Family | Label | Region | Workers |
+|---|---|---|---|
+| 0 | EU-1 | N EU | @l1 @max @mike |
+| 1 | EU-2 | central EU | @yasir @l1 |
+| 2 | AS-1 | E EU / NW AS | @lkskrn @mike |
+| 3 | AS-2 | E AS | @ilich |
+| 4 | AS-3 | SEA |  @cooloNe |
+| 5 | AF-1 | N AF / S EU | |
+| 6 | AF-2 | S AF | @Alex |
+| 7 | OC | AU / Oceania | @l1 |
+| 8 | SA-1 | S SA | |
+| 9 | SA-2 | N SA + central | @0x2bc @spat |
+| 10 | NA-1 | NA W | @cooloNe |
+| 11 | NA-2 | NA central | |
+| 12 | NA-3 | NA E | @l1 |
+
+## Workers
+
+https://dao.joystream.org/#/working-groups/distribution
+
+[Pin](https://discord.com/channels/811216481340751934/933726271832227911/957589245407678494)
+
+![](https://cdn.discordapp.com/attachments/933726271832227911/957675927326838814/olympia-distribution.png)
+
+### Upgrading
+
+- [QN](https://discord.com/channels/811216481340751934/933726271832227911/957653724174614559), [also](https://discord.com/channels/811216481340751934/933726271832227911/957390875959361597)
+- [Validator](https://discord.com/channels/811216481340751934/933726271832227911/957572146958319616) (`--rpc-cors`)
+
+### Costs
+
+| Member | Bucket | Worker | Location | Size TB | USD / mon |
+|---|---|----|---|---|---|
+| @l1dev | 0:0 | 0 | Helsinki | 2 | 75 |
+| @l1dev | 1:1 | 0 | Nuremberg | .2 | 32 |
+| @l1dev | 12:0 | 0 | Montreal | .15 | 24 |
+| @l1dev | 0:0 | 0 | Sydney | .16 | 20 |
+| @cooloNe | 10:0 | 1 | Los Angeles | 0.8 | 84 |
+| @cooloNe | 4:0| 1 | Singapore | 0.8 | 225 |
+| @Ilich | 3:0 | 2 | Novosibirsk | 0.3 | 44 |
+| @AlexZNet | 6:0 | 3 | Johannesburg | 0.1 | 60 |
+| @xJames |  9:0 | 4 | Sao Paolo | 0.75 | 177 |
+| @Lelik_maxi |  0:1 | 5 | Helsinki | 1.4 | 57 |
+| @lkskrn |  2:0 | 6 | Moscow | 0.3 | 51 |
+| @yasir | 1:0 | 7 | Nuremberg | 0.4 | 35 |
+| @spat_sochi |  9:1 | 8 | Bogota | 0.1 | 69 |
+| @MikeShipa | 0:2 | 9 | Helsinki | 0.48 | 50 |
+| @MikeShipa | 2:1 | 9 | Moscow | 3.5 | 96 |
+| **Total** | | | | | 1200 (300/w) |
+
+## Setup
+
+- [Commands](https://discord.com/channels/811216481340751934/933726271832227911/957567685993050162)
+- Hint: [set dynamic bag policy early to avoid manual bag assignment](https://discord.com/channels/811216481340751934/933726271832227911/957592223422230566)
+
+## Bugs
+
+- multiple workers can be invited to the same bucket ([details](https://discord.com/channels/811216481340751934/933726271832227911/957673382936199240))
+- worker id is not displayed on https://dao.joystream.org/#/working-groups/distribution 
+- only one of three hired workers is displayed on https://dao.joystream.org/#/working-groups/openings/distributionWorkingGroup-4

working-groups/distributors/opening-template.json

@@ -0,0 +1,20 @@
+[
+    {
+        "activateAt": { "CurrentBlock": null },
+        "maxReviewPeriodLength": 432000,
+        "applicationStake": { "mode": "AtLeast", "value": 50000 },
+        "roleStake": {"mode": "AtLeast", "value": 1000000 }
+    },
+    {
+        "version": 1,
+        "headline": "Distributor in Northern Europe",
+        "job": {
+            "title": "EU Distributor",
+            "description": "Requirements: SSD, 1 TB SSD, synced validator and query node."
+        },
+        "application": {
+        },
+        "reward": "40k tJOY per 1200 blocks ($80 / week)",
+        "process": { "details": [] }
+    }
+]

working-groups/distributors/skillshare.md

@@ -0,0 +1,38 @@
+# [47.DT-1 - Olympia - Handover](https://blog.joystream.org/sumer-kpis/#47.DT-1)
+
+There is still a lot room for improvement for the a lead + deputy team.
+
+1. Outline the current routines and workflow of the Storage group: Routines for noticing, and the following actions, when/if: a node goes down, a distributor provides poor service, etc.
+
+None of these are automated yet. There is a script that reports performance stats to jsstats which is displayed on https://joystreamstats.live/distribution but it has to be run by someone (only results older less than a day are displayed): https://github.com/Joystream/community-repo/pull/679 (Open task: [Content benchmark](https://github.com/Joystream/community-repo/issues/656))
+
+2. Outline the current setup, and explain why it is the way it is.
+
+- number of nodes and (family) buckets: Our beautiful planet has 7 [continents](https://en.wikipedia.org/wiki/Continent), we got 9 distribution bucket families. There can be more when more fine grained distribution is needed.
+- geographical distribution: every continent has at least 1 family and bucket for basic global distribution. Central / Northern Europe (1) and Eastern Europe / Western Asia (2) have a some more buckets that could be optimized to save costs.
+- dynamic bag policy: Currently buckets with big enough cache are assigned to new buckets. This is simple and practical for now and should be replaced by some more sophisticated scheme on the new scheme to prepare for bigger upload waves.
+- mode settings: All buckets are accepting and distributing, all disfunctional buckets / operators have been removed which [revealed a bug](https://discord.com/channels/811216481340751934/859752198166282280/954662917045903382).
+- hardware requirements: Recommend are at least 2 cores, 2GB RAM and minimal 100 GB disk space (using external QN), better 200 GB upwards to also host a local validator node and extend the cache limit.
+- there are lots more thoughts in the [first report](https://github.com/Joystream/community-repo/blob/master/working-groups/distributors/giza1.md) that should be thought over and amended by smart OKR.
+
+3. Outline the extra work you, the Lead, do that may not be obvious.
+
+- Organizing: I use an [emacs org-mode table](https://orgmode.org/org.html#Advanced-features) with some formulas to calculate rewards. I can share if there is interest but i doubt anyone would do the same. Also keeping a permanent bash session in tmux to fast find previously used commands with ctrl+r or `history`, also in the mentioned text file.
+
+![](img/emacs-org-mode.png)
+
+- Hiring: This involves a lot of simple steps that are all explained [here](https://github.com/Joystream/helpdesk/tree/master/roles/distributors-lead). Some possible improvements are shared in the [wishlist](https://github.com/Joystream/joystream/issues/276).
+
+- Managing the Workers (at the "human" level): This is the actual fun part of the job because every worker is different and brings their level of versality. Some need no input and some ask every step on the way even when the manual is publicly known. I appreciate both ways because communication can be fun when there is nothing else pressing. I recommend doing max in private messages to burden everyone else and just pin selected compacted instructions / experiences with everyone.
+
+![](img/tmux-history.png)
+
+- Adjusting rewards: This was definitively the most challenging part because there was no input by anyone about intended budget. Rules like the $80 setup bonus were needed for some and had to be granted to all just to be fair. After filling all families and gaining some experience with prices i would probably be more strict about allowed price ranges when hiring. Also better not to hire before the worker set up a node although this would leave the risk (setup fee, work) on them. Every lead has to find their way.
+
+> Is there anything you've learned along the way, that wasn't at all clear (or even wrong) from the guides?
+
+Patience is always is plus, also openness for both sides. I trust my workers to share their costs honestly and always try to pay above that even if it means to overpay in some cases. Surely the system will need to get more fine-grained (thinking of the price per GB, it was just luck that the formula `GB * eras * 10` yielded useful tJOY rewards at current token value, which is about to end with further inflation). Also measuring and recording performance to automatically adjust rewards. A fresh lead will probably bring a bit more enthusiasm than me after 6 months and currently two lead roles + council work.
+
+> If there is anything else you'd wish someone told you when you became the Lead - explain!
+
+I wish someone would have showed interest to assist as Deputy Lead but i guess it is a sign of trust that i will just do the right thing. One more reason to make space for fresh blood. Maybe i will take the time to write something more helpful later. :)