community-repo/working-groups/distributors/archive/giza/migration

41.CC-1 - Finalizing Content Migration

The goal is to check if transferred channels and videos from Sumer testnet are available on Giza distributors.

Abstract

After the upgrade, we migrated a lot of the content from sumer over to giza. This means some follow up work is required. Based on our own, and the community input, the channels that was migrated (including all videos) is available in thi issue. For addressing both this, and the KPI 41.CC-2, using the "old" and the "new" query-node is needed.

Find out if all of the following was in fact migrated as we inteded, or if something was missing: channels videos (he migrations says yes for both, but double check wouldn't hurt) assets - some nuances here: How many ACCEPTED (ignore PENDING/REJECTED) data objects were there in total stored under these channels? How many bytes in total? We know the exact amounts migrated. For 1. Share the queries and scripts used to get the data.

verify.js does following

• load ids.js with channel, video mapping
• query and cache Sumer QN for channels (oldChannels.json)
• query and cache Giza QN for videos (newVideos.json)
• select migrated ids and perform one HEAD request for each asset on https://storage-1.joystream.org/argus (~12 results/s, no throttling, sorry!)
• save results to migration_result.json
• print tables for empty source videos (PENDING) and non-uploads on Giza (see below).

Were there any censored channels or videos that made it into the migration? If yes, re-evaluate, and censor again if needed.

4 channels (54 55 57 133) are marked as isCensored, see Censored source channels.

Are there any channels or videos that wasn't attempted migrated, but should have been? Make a channel on discord, and a forum thread where users can post what they are missing. Review what is being requested, and share the results.

Of 4706 selected videos 4646 were successfully transferred (646,5 GB video data + 82 MB thumbnails).

60 empty source videos were selected for transfer and the same amount is empty on the target provider.

Resources

• Old and new ChannelId mapped: https://github.com/Joystream/community-repo/issues/609
• old QN: https://hydra-sumer.joystream.org/graphql
• new QN: https://hydra.joystream.org/graphql
• initial Distributor setup: https://github.com/bwhm/helpdesk/tree/giza/roles/distributors-lead#initial-configurations---giza

Queries

Channel <> Videos collection Sumer

query {
  channels (where:{id_eq:534 }) {
    id
    createdAt
    updatedAt
    createdInBlock
    title
    category {
      name
    }
    ownerMember {
      id
      handle
      channels {
        id
        title
      }
    }
    coverPhotoAvailability
    avatarPhotoAvailability
    coverPhotoDataObject {
      joystreamContentId
      size
    }
    avatarPhotoDataObject {
      joystreamContentId
      size
    }
    isCensored
    isPublic
    }
    languages {
      iso
    }
    videos {
      id
      title
      mediaAvailability
      thumbnailPhotoAvailability
    }
}

Shorter version here

Assets on JSG provider (Giza)

query{storageBuckets(where:{id_eq:"1"}){id,createdAt,acceptingNewBags,dataObjectsSizeLimit,dataObjectCountLimit,dataObjectsCount,dataObjectsSize,bags{id,objects{id,createdAt,isAccepted,size}}operatorMetadata{id,nodeEndpoint,nodeLocationId,extra,nodeLocation{id,countryCode,coordinates{latitude,longitude}}}}}

=> https://discord.com/channels/811216481340751934/812344681786507274/936573775066054688

Channel <> Videos collection and Bag <> StorageBucket associations Giza

query {
  storageBags { id distributionBuckets{id familyId bucketIndex} }  
  channels {id createdAt updatedAt createdInBlock title
    category { name } isCensored isPublic
    ownerMember { id handle }
    videos { id title }
  }
}

Results

Loaded IDs of 4706 transferred videos in 177 channels. Loaded info of 4706 migrated videos. Loaded info of 713 old channels. 4646 videos were transferred 646485444380 bytes + 82260654 avatars + covers.

Migrated videos with empty source

ID	Title	Size	Upload	Thumbnail
6154	test upload	140534084	PENDING	ACCEPTED
296	asd	253791949	PENDING	ACCEPTED
298	2021-05-25 11-13-38	253791949	PENDING	ACCEPTED
299	2021-05-25 11-13-38	253791949	PENDING	ACCEPTED
300	2021-05-25 11-13-38	253791949	PENDING	ACCEPTED
301	asdasd	253791949	PENDING	ACCEPTED
2040	Как стать Амбассадором Bit.Country	144893665	PENDING	ACCEPTED
120	Showreel animation 2021	44997716	PENDING	PENDING
121	Showreel animation 2021	44997716	PENDING	PENDING
303	Tretyakov Gallery - Black Square	755357428	PENDING	PENDING
206	Любимое кафе с лучшим видом	19237360	PENDING	PENDING
219	Путешествую с Пятницей	14891181	PENDING	PENDING
393	Ночую в заброшке	27508553	PENDING	PENDING
407	Нашел работу в горах	37731273	PENDING	PENDING
413	Где я жил в кемпинге	48027398	PENDING	PENDING
415	Лес обезьян	34139670	PENDING	PENDING
428	Подношения	42596459	PENDING	PENDING
220	Путешествую с Пятницей	14891181	PENDING	PENDING
201	Как живет семья Бали	29098159	PENDING	PENDING
194	Бали открыли	34331478	PENDING	PENDING
216	Прекрасный вид на 360	36176482	PENDING	PENDING
388	Танцы с Пятницей	36599530	PENDING	PENDING
430	С Пятницей на вулкан	36065310	PENDING	PENDING
399	Бали в стиле старых фильмов	79158922	PENDING	PENDING
2601	JOY	6264500	PENDING	ACCEPTED
1255	Superman: Eleventh Hour	34823272	PENDING	ACCEPTED
1270	Superman: Jungle Drums	49888116	PENDING	ACCEPTED
2388	The Great Piggy Bank Robbery	80195964	PENDING	ACCEPTED
5591	I DID A INTENSE WORKOUT CHALLENGE	299462853	PENDING	ACCEPTED
5593	Super-Pump Arm Workout for Mass	263483999	PENDING	ACCEPTED
2473	Cosmos Laundromat	76159780	PENDING	ACCEPTED
2488	Elephants Dream	69682629	PENDING	ACCEPTED
2374	Big Buck Bunny	94054804	PENDING	ACCEPTED
2377	Caminandes1	4999593	PENDING	ACCEPTED
3511	Самые смешные коты	45902976	PENDING	ACCEPTED
3513	Приколы с Котами Топовая Подборка	188260671	PENDING	ACCEPTED
3522	Самые смешные коты рунета	36525391	PENDING	ACCEPTED
2532	Что происходит на дорогах.	74585760	PENDING	ACCEPTED
3252	Секрет обычной ТЁРКИ!	99776942	PENDING	ACCEPTED
3260	Без помощи сварки	166481690	PENDING	ACCEPTED
2731	Добрые поступки на дороге !!	95672013	PENDING	ACCEPTED
2784	Сначало кушал один ......	60910695	PENDING	ACCEPTED
2980	ОШИБКИ при раскладке ПЛИТКИ!	54067149	PENDING	ACCEPTED
2983	Резка кафельной плитки без плиткореза	25852047	PENDING	ACCEPTED
3680	Убойные приколы с животными#6	69258194	PENDING	ACCEPTED
3791	Милые животные	42696740	PENDING	ACCEPTED
3790	Милые белки	303573777	PENDING	ACCEPTED
3789	Спасение котенка	57380709	PENDING	ACCEPTED
4957	ЩЕНОК ЮМИ ЧУ	149896221	PENDING	ACCEPTED
4956	ТАЙНАЯ ЖИЗНЬ ДОМАШНИХ ЖИВОТНЫХ	27674316	PENDING	ACCEPTED
4963	Супер коты	46215175	PENDING	ACCEPTED
4997	Смешные кошки	41456550	PENDING	ACCEPTED
5070	СМЕШНЫЕ ЖИВОТНЫЕ , КОТЫ СОБАКИ И ДРУГИЕ	121691971	PENDING	ACCEPTED
8662	IMG_	42613049	PENDING	ACCEPTED
4684	J. Grunsfeld about working on Hubble	42109276	PENDING	PENDING
5259	Funny Parrot Talking Videos	28707385	PENDING	ACCEPTED
5755	Шоу 15 этажей	259660035	PENDING	ACCEPTED
7724	Joystream	154651016	PENDING	ACCEPTED
4854	Как собирать семантику (запросы)	216125363	PENDING	ACCEPTED
8968	roksolana joy	14286490	PENDING	ACCEPTED

Censored source channels

Found 4 censored channels in migration set:

{"id":"55","createdAt":"2021-05-28T15:41:12.000Z","updatedAt":"2021-06-03T07:31:00.000Z","createdInBlock":733904,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5GpiUscQLtvBzZE1Gk53wvw8gz8zjP7TqFvM5bGhYZUcwCbd","size":193858},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5DyCJaqLMDxC1TgRjuGoFL3VDcw3dV9P1b3NsrdNapqyDPC7","size":15103},"videos":[]}
{"id":"57","createdAt":"2021-05-28T15:44:06.000Z","updatedAt":"2021-06-03T07:31:42.000Z","createdInBlock":733933,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5HW2XWFDEgxmmGN7FQMeX52HHM8EyhbbHSp9YvGm6VCCzAxF","size":172320},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5Fg31k7kmX6Ms5oy75P3KHQ844CfrTo3FvfjeKGQHeb4gi3H","size":14980},"videos":[]}
{"id":"54","createdAt":"2021-05-28T15:40:54.000Z","updatedAt":"2021-06-03T07:24:18.000Z","createdInBlock":733901,"title":"Gingah","category":null,"ownerMember":{"id":"2195","handle":"gingah"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"ACCEPTED","coverPhotoDataObject":{"joystreamContentId":"5Cpg2fhvHtMNcrHrJ3VZ78wo1gCM47M6cKxZwGTiK23jWxzH","size":193858},"avatarPhotoAvailability":"ACCEPTED","avatarPhotoDataObject":{"joystreamContentId":"5FFTicA5vjzgM68QCNzFe8xGXxjVBq1phCRigGdXMMWCGWMU","size":15103},"videos":[]}
{"id":"133","createdAt":"2021-06-08T16:07:12.000Z","updatedAt":"2021-06-20T12:08:00.014Z","createdInBlock":892380,"title":"testing channel","category":null,"ownerMember":{"id":"2269","handle":"member_test_2112"},"isCensored":true,"isPublic":true,"coverPhotoAvailability":"INVALID","coverPhotoDataObject":null,"avatarPhotoAvailability":"INVALID","avatarPhotoDataObject":null,"videos":[]}

Verification

Performed HEAD request on 4697 videos. 4646 of 4706 assets are available.

60 have no content.

Channel	Video	Asset	Status
734	9724		no media
734	9731		no media
734	9750		no media
734	9746		no media
734	9749		no media
734	9843		no media
734	9840		no media
734	9827		no media
734	9857		no media
734	9859		no media
734	9736		no media
734	9845		no media
734	9822		no media
734	9832		no media
747	12100		no media
751	9698		no media
751	9777		no media
751	9697		no media
757	9772		no media
757	9773		no media
757	9774		no media
757	9775		no media
757	9776		no media
758	10667		no media
777	10917		no media
778	10154		no media
778	10169		no media
784	10831		no media
799	10821		no media
799	10822		no media
799	10862		no media
799	10866		no media
801	11340		no media
801	11341		no media
801	11342		no media
801	10888		no media
801	10993		no media
801	11109		no media
801	11111		no media
801	11023		no media
801	11164		no media
801	11603		no media
801	11604		no media
801	11606		no media
801	11170		no media
801	11614		no media
801	11231		no media
801	11222		no media
801	11224		no media
801	11631		no media
801	11308		no media
814	11506		no media
819	11568		no media
821	11702		no media
822	11824		no media
822	11826		no media
828	11884		no media
864	13757		no media
869	13024		no media
889	14004		no media