joystream/content-directory-schemas

Content directory json schemas and inputs

Definitions

In order to make this documentation as clear as possible it is important to make a strict distinction between two types of schemas:

• json-schemas mean files with .schema.json extension. This is a common standard for describing how to validate other json files or objects (ie. a package.json file may be an example of a file that can be supported by a json-schema). A documentation of this standard can be found here: https://json-schema.org/
• runtime-scheams means schemas as they are "understood" by the content-directory runtime module, so schemas that can be added to classes via api.tx.contentDirectory.addClassSchema and linked to entities via api.tx.contentDirectory.addSchemaSupportToEntity

Content directory input

Initializing content directory

In order to intialize the content directory on a development chain based on data that is provided in form of json files inside /inputs directory (classes, schemas and example entities - entityBatches), we can run:

yarn workspace cd-schemas initialize:dev

This will handle:

• Creating a membership for ALICE (if not already created)
• Setting (hiring) ALICE as content curators lead (if not already set)
• Creating classes in the runtime based on inputs/classes json inputs (if the content directory is currently empty)
• Creating schemas in the runtime based on inputs/schemas and adding them to the related classes
• Creating entities based on inputs/entityBatches. Those json inputs allow describing entities and relationships between them in a simplified way and are then converted into one huge api.tx.contentDirectory.transaction call (this is further described in Entity batches section).

Input files naming

In order to get the full benefit of the tooling, in some cases you may need to respect a specific pattern of file naming:

Each input file name should end with Class, Schema or Batch (depending on the input type, ie. LanguageBatch). It is also recommended that each of those file names starts with a class name (currently in entityBatches there's no distinction between schemas and classes, as it is assumed there will be a one-to-one relationship between them)

json-schemas support for json inputs in VSCode

In order to link json files inside inputs directory to json-schemas inside schemas and have them validated in real-time by the IDE, follow the steps below:

If you don't have .vscode/settings.json in the root monorepo workspace yet:

1. Create .vscode directory inside your monorepo workspace
2. Copy vscode-recommended.settings.json into this .vscode directory and rename it to settings.json.

If you already have the .vscode/settings.json file in the root monorepo workspace:

1. Copy the settings from vscode-recommended.settings.json and merge them with the existing .vscode/settings.json

Now all the json files matching *Class.json, *Schema.json, *{EntityName}Batch.json patters will be linked to the correct json schemas. If you edit any file inside inputs or add a new one that follows the naming pattern (described in Input files naming), you should get the benefit of autocompleted properties, validated input, on-hover tooltips with property descriptions etc.

For more context, see: https://code.visualstudio.com/docs/languages/json

Validate inputs and json-schemas via a command

All inputs inside inputs directory and json-schemas used to validate those inputs can also be validated using yarn workspace cd-schemas validate command. This is mainly to facilitate checking the validity of .json and .schema.json files inside content-directory-schemas through CI.

Entity batches

The concept of entity batches (inputs/entityBatches) basically provides an easy way of describing complex input to content directory (ie. many entities related to each other in many ways) without the need to deal with lower-level, hard-to-validate runtime operations like CreateEntity and AddSchemaSupportEntity and trying to glue them together into a huge api.tx.contentDirectory.transaction call.

Instead, the script that initializes the content directory (scripts/initializeContentDir.ts) is able to generate the complex api.tx.contentDirectory.transaction call based on a more human-readable input provided in inputs/entityBatches.

This input can be provided as a simple json array of objects matching { [propertyName]: propertyValue} structure.

For example, in order to describe creating entities as simple as Language, which only has Code and Name properties, we can just create an array of objects like:

[
  { "Code": "EN", "Name": "English" },
  { "Code": "RU", "Name": "Russian" },
  { "Code": "DE", "Name": "German" }
]

(This is the actual content of inputs/entityBatches/LanguageBatch.json)

Related entities

There also exists a specific syntax for defining relations between entities in batches. We can do it by either using "new" or "existing" keyword.

• The "new" keyword allows describing a scenario where related entity should be created along with the main entity and then referenced by it. An example of this could be Video and VideoMedia which have a one-to-one relationship and it doesn't make much sense to specify them in separate batches. Instead, we can use a syntax like:

  {
  "title": "Awesome video",
  /* other Video properties... */
  "media": { "new": {
  "pixelWidth": 1024,
  "pixelHeight": 764,
  /* other VideoMedia object properties... */
  }
  }
  

• The "existing" keyword allows referencing an entity created as part of any other batch inside inputs/entityBatches. We can do it by specifying the value of any unique property of the referenced entity. So, for example to reference a Language entity from VideoBatch.json file, we use this syntax:

  {
  "title": "Awesome video",
  /* other Video properties... */
  "language": { "existing": { "Code": "EN" } }
  }
  

json-schemas and tooling

Entity json-schemas

There is a script that provides an easy way of converting runtime-schemas (based on inputs from inputs/schemas) to json-schemas (.schema.json files) which allow validating the input (ie. json files) describing some specific entities. It can be run with:

yarn workspace cd-schemas generate:entity-schemas

Those json-schemas are currently mainly used for validating the inputs inside inputs/entityBatches.

The generated json-schemas include:

• schemas/entities - json-schemas that provide validation for given entity (ie. Video) input. They can, for example, check if the title property in a json object is a string that is no longer than 64 characters. They are used to validate a single entity in inputs/entityBatches, but can also be re-used to provide "frontend" validation of any entity input to the content directory (ie. input provided to/via joystream-cli).
• schemas/entityReferences - json-schemas that describe how an entity of given class can be referenced. Currently they are used for providing an easy way of referencing entites between batches in inputs/entityBatches. For more details on how entities can be referenced in batches, read the Entity batches section.
• schemas/entityBatches - very simple json-schemas that basically just provide array wrappers over schemas/entities. Those are the actual json-schemas that can be linked to json input files inside inputs/entityBatches (ie. via .vscode/settings.json)

Typescript support

Thanks to the json-schema-to-typescript library, we can very simply generate Typescript interfaces based on existing json-schemas. This can be done via:

yarn workspace cd-schemas generate:types

This command will generate:

• types/entities based on schemas/entities, providing typescript interfaces for entities like Video etc. (note that this interface will include a peculiar way of describing entity relationships, further described in Entity batches section)
• types/extrinsics based on schemas/extrinsics, providing typescript interfaces for input to extrinsics like AddClassSchema and CreateClass

The most obvious use-case of those interfaces currently is that when we're parsing any json files inside inputs using a Typescript code, we can assert that the resulting object will be of given type, ie.:

const createClassInput = JSON.parse(fs.readFileSync('/path/to/inputs/LanguageClass.json')) as CreateClass

Besides that, a Typescript code can be written to generate some inputs (ie. using a loop) that can then can be used to create classes/schemas or insert entities into the content directory.

There are a lot of other potential use-cases, but for the purpose of this documentation it should be enough to mention there exists this very easy way of converting .schema.json files into Typescript interfaces.

Current limitations

Some limitations that should be dealt with in the nearest future:

• Filename restrictions described in Input files naming section
• Some code runs on the assumption that there is only one schema for each class, which is very limiting
• Vector<Reference> property type is not yet supported when parsing entity batches