Creating a new language, Part 2: Editor extensions

As we discussed last time, creating a new syntax is insufficient if your editor won’t recognize and highlight it. Luckily, since our language (which we’re calling pysh because we’re silly people) is a superset of Python with minor syntax changes, the extension is easy to create.

We’ll follow the VS Code tutorial for creating an extension here. We’ll install the tool using:

npm i -g yo generator-code

We’ll run yo code and follow the instructions. The main file here is package.json. We’ll update the contributes section:

"contributes": {
    "languages": [{
      "id": "pysh",
      "aliases": [
        "Pysh",
        "pysh"
      ],
      "extensions": [
        ".pysh"
      ],
      "configuration": "./language-configuration.json"
    }],
    "grammars": [{
      "language": "pysh",
      "scopeName": "source.pysh",
      "injectTo": ["source.python"],
      "path": "./syntaxes/pysh.tmLanguage"
    }]
  }

We’re creating a language with an id of pysh, which has aliases that users can search for in VS Code’s language selector. We specify the extensions we recognize. Next, we specify a grammar for our language. The language field specifies the id of the language we’re using (so pysh). We define a new scope name, which we’ll edit in the grammar later. I’m not sure if the injectTo is needed, but it specifies that our language is an extension to Python, whose scope name is source.python. Our grammar will be located in the path field.

We won’t need to write the grammar from scratch. Instead, we’ll start with a Python grammar. VS Code recognizes grammars in TextMate format, so we’ll use a grammar in that format. We’ll edit this directly. We’ll change the fileTypes field to pysh. At the very end of the file is the scope name that we’ll change to source.pysh. A scope is a part of code that you can recognize via a regular expression, and highlight separately. The top-level scope will be called source.pysh. The latter part of this name can be anything we wish (as long as it matches what’s in package.json), but the source part is intentional: it comes from TextMate’s documentation. We’ll refer to this later as well. Under the patterns key, we’ll add a dict element to the array:

        <dict>
            <key>begin</key>
            <string>(\w[a-zA-Z0-9.]+)?`</string>
            <key>beginCaptures</key>
            <dict>
                <key>1</key>
                <dict>
                    <key>name</key>
                    <string>keyword.pysh.formatter.name</string>
                </dict>
            </dict>
            <key>end</key>
            <string>`|(\n)</string>
            <key>endCaptures</key>
            <dict>
                <key>1</key>
                <dict>
                    <key>name</key>
                    <string>invalid.illegal.unclosed_string.fmt.python.3</string>
                </dict>
            </dict>
            <key>name</key>
            <string>string.quoted.single.line.fmt.python.3</string>
            <key>patterns</key>
            <array>
                <dict>
                    <key>include</key>
                    <string>#string_patterns</string>
                </dict>
                <dict>
                    <key>include</key>
                    <string>#format_specifier_extended</string>
                </dict>
            </array>
        </dict>

I’ll admit that I was cheeky and copied the code from the part of the grammar that recognizes the f-string syntax (since our syntax is very similar to that). The first few lines of code specify that our scope will start with a regex (\w[a-zA-Z0-9.]+)?`, which will match formatter names. This part will have a scope that we’ll call keyword.pysh.formatter.name. The keyword part specifies that VS Code will highlight this as a keyword. Next, we specify that this scope will end with a backtick or a newline. If it’s the latter, then we have an unclosed string, which is illegal syntax. The actual contents inside the scope is treated as a string.

We’ll need to do some admin work before we can compile and publish our extension. In package.json, we’ll update the repository, name, and display name. We’ll follow the documentation to publish our extension. Let’s install the command-line tool:

npm i -g vsce
vsce package
vsce publish

That last command won’t work. We’ll need to go to Azure DevOps and create an organization, and a project within that organization. We’ll need a Personal Access Token, which we’ll generate from the User Settings in the top right. Most of the defaults under New Token are okay, but we’ll set the scope to Custom Defined. We’ll click on Show All Scopes, and under Marketplace, ensure that Manage is selected. We’ll create the token, and we’ll use this to log in to the CLI.

vsce login <publisher-name>

Your publisher name should be in the Azure DevOps dashboard. Use the one from your personal account, not the organization. Finally, vsce publish should work.