Integrating Shotgun with Jira

The Shotgun Jira Bridge is an MIT-licensed project that uses Jira Webhooks on one end and Shotgun Event Daemon triggers on the other to push data back-and-forth between them. The diagram provided in their documentation illustrates this succinctly:

(Image by shotgunsoftware)

Our goals with the bridge are quite simple:

1. Track a ticket using some Shotgun entity linked to a Jira Issue.
2. Comments should be bidirectionally synced, ideally as soon as they are posted.
3. Status changes should also be bidirectional.
4. Optionally, the ticket priority should be displayed on Shotgun. This could be bidirectional or we may want to control this on the Jira side only.

You may have noticed we mentioned "initial setup" earlier. This is because the bridge's features out-of-the-box are rather sparse:

• The ability to sync Tasks from Shotgun as a Jira Issue.
• Status changes are bidirectional.
• Notes on Tasks are synced as comments under the linked Issue.

Some of the limitations are a bit more concerning:

• Replies on Notes are not synced to the Issue.
• New comments on the Issue are not synced over to the Task.
• A Shotgun project is tied to a single Jira project. This is reasonable but worth mentioning. Do note that multiple Shotgun projects can share the same Jira project.

Fortunately the bridge is easily customizable so we'll be able to resolve these problems with a bit of legwork.

For a start, we'd like to add support for the Shotgun Ticket entity. It matches Jira's Issue structure much more and doesn't force us to tie them to Tasks. We have the option to link to Tasks if we wish and a Task could even have multiple Tickets in this setup. You can find more information about Tickets in this article from Shotgun including steps to enable it.

The comment structure for Tickets also matches Jira Issues in that they are linear as opposed to a Task's threaded system using notes and replies.

Before we add our customizations we'll have to get the basic framework setup, so let's do that now.

• A server that is accessible by both the Shotgun and Jira instances.

That's it! We'll be going over the setup from scratch. For our example we'll be using a RHEL 8.2 box, resolved by bridge.aws.pixel-nexus.com. Other distros (or even Windows) should work just fine but will require some tweaking from the steps below.

Some schema changes have to be applied to Shotgun and Jira to track the metadata of related tickets. You can find the specific fields to add in the official documentation, in short you need the following:

• On Shotgun:
  • Project needs Jira Sync URL and Jira Key to know which project to sync with.
  • HumanUser (People) need a Jira Account Key to match them to Jira users.
  • Task and/or Ticket need a Jira Key and Jira URL to track the related Issue as well as a Sync in Jira checkbox to determine which to sync.
  • Note and/or Reply need a Jira Key to track the corresponding comment.
  • Each of the Jira Key fields that we define must enforce unique values.
• On Jira:
  • Shotgun Type to associate to the correct type.
  • Shotgun ID to associate to the specific ID.
  • Shotgun URL for a convenient link to the Shotgun page.

Note that Jira comments do not store metadata to link themselves to the related reply. We'll solve that problem when we configure the bridge.

For convenience, this Python script will create the fields on Shotgun as described above:

#! /bin/python
import shotgun_api3

# Fill in your Shotgun URL and script credentials here
sg = shotgun_api3.Shotgun(
    base_url="<shotgun_url>",
    script_name="<script_name>",
    api_key="<script_key>")


sg.schema_field_create(entity_type='HumanUser',
                    data_type='text',
                    display_name='Jira Account Id')

sg.schema_field_create(entity_type='Project',
                    data_type='text',
                    display_name='Jira Key')
sg.schema_field_create(entity_type='Project',
                    data_type='url',
                    display_name='Jira Sync URL')

sg.schema_field_create(entity_type='Ticket',
                    data_type='text',
                    display_name='Jira Key')
sg.schema_field_create(entity_type='Ticket',
                    data_type='url',
                    display_name='Jira URL')
sg.schema_field_create(entity_type='Ticket',
                    data_type='checkbox',
                    display_name='Sync In Jira')

sg.schema_field_create(entity_type='Reply',
                    data_type='text',
                    display_name='Jira Key')

# You could ignore Task and Note if you don't plan on using it.

sg.schema_field_create(entity_type='Task',
                    data_type='text',
                    display_name='Jira Key')
sg.schema_field_create(entity_type='Task',
                    data_type='url',
                    display_name='Jira URL')
sg.schema_field_create(entity_type='Task',
                    data_type='checkbox',
                    display_name='Sync In Jira')

sg.schema_field_create(entity_type='Note',
                    data_type='text',
                    display_name='Jira Key')
         

For each Project that is to be supported, you'll have to fill in the Jira Sync URL and Jira Key. The Jira Sync URL will always be the same: https://bridge.aws.pixel-nexus.com/sg2jira/default (Replace the hostname with your own). The Jira Key must match the Jira Project you wish to use, eg. SGJ.

This is what it looks like in the Shotgun UI (with the fields visible):

The bridge application itself processes data coming from the Shotgun Event Daemon and the Jira WebHook, so we'll set those up first.

Note: Although the example uses Python 2, it should be Python 3 compatible.

First let's prepare the base directory. We'll be using /etc/sg-jira-bridge for the application and create an sgjira user which will run the services. We'll go ahead and clone the bridge and event daemon repositories as well:

#! /bin/bash
yum install -y python2 git
pip2 install virtualenv
useradd sgjira
basedir=/etc/sg-jira-bridge
mkdir $basedir
git clone https://github.com/shotgunsoftware/sg-jira-bridge.git $basedir/sg-jira-bridge
git clone https://github.com/shotgunsoftware/shotgunEvents.git $basedir/shotgunEvents
         

Note: If you already have an existing Shotgun Event Daemon instance you can skip most of this section. You will only need to add the trigger script to your plugins and the relevant environment variables.

It's always good practice to setup a virtual environment for your Python applications, especially as we'll be running (at least) two Python applications alongside eachother.

#! /bin/bash
basedir=/etc/sg-jira-bridge
virtualenv $basedir/shotgunEvents/venv
# Install requirements using the requirements.txt from the bridge's trigger.
source $basedir/sgeventd-venv/bin/activate
pip install -r $basedir/sg-jira-bridge/triggers/requirements.txt
deactivate
# Prepare configuration
cp $basedir/shotgunEvents/src/shotgunEventDaemon.conf.example $basedir/shotgunEvents/src/shotgunEventDaemon.conf
mkdir -p /usr/local/shotgun/plugins
# Give ownership to 'sgjira'
chown -R sgjira:sgjira $basedir
# TODO: Modify shotgunEventDaemon.conf to match your setup.
         

You'll have to change the server, name, key to match your Shotgun server URL, script name, and script key. You'll also have to add the trigger as a plugin path. If you get stuck, see the wiki on GitHub for more detailed instructions.

Lastly, the trigger uses separate credentials from the daemon. These must be supplied from the environment, so for testing purposes just export these in your shell:

export SGDAEMON_SGJIRA_NAME="<my-script-name>"
export SGDAEMON_SGJIRA_KEY="<my-script-key>"
         

You can then test the daemon using /etc/sg-jira-bridge/shotgunEvents/src/shotgunEventDaemon.py foreground.

If all is looking well the last step is to setup a systemd unit to manage it.

The Jira Webhook is trivial to setup. The settings are:

• Name: "SG Jira Bridge"
• URL: "https://bridge.aws.pixel-nexus.com/jira2sg/default/issue/${issue.key}"
• JQL: project = "jira-project-name"
• Events:
  • Issue (created, updated, deleted)
  • Comment (created, updated, deleted)
• Exclude body must be unchecked.

We'll setup a separate virtual environment for this, similar to the daemon:

#! /bin/bash
basedir=/etc/sg-jira-bridge
virtualenv $basedir/bridge-venv
source $basedir/bridge-venv/bin/activate
pip install -r $basedir/sg-jira-bridge/requirements.txt
deactivate
# Give ownership to 'sgjira'
chown -R sgjira:sgjira $basedir
         

Again, running the webapp requires some credentials. This can come from a .env file, but for testing we'll just export it from our shell:

# Shotgun
 export SGJIRA_SG_SITE="<shotgun-url-here>"
 export SGJIRA_SG_SCRIPT="<my-script-name>"
 export SGJIRA_SG_SCRIPT_KEY="<my-script-key>"
 # Jira
 export SGJIRA_JIRA_SITE="<jira-url-here>"
 export SGJIRA_JIRA_USER="<jira-email-script-is-registered-to>"
 export SGJIRA_JIRA_USER_SECRET="<jira-api-key-here>"
         

Before we can test it, we'll have to import the users. This is something we'll have to do each time new users are added on Shotgun. The bridge provides a convenient script for this:

source $basedir/bridge-venv/bin/activate
python update_shotgun_users.py --settings $basedir/sg-jira-bridge/settings.py --project <shotgun-project-id>
         

You can run the bridge (using the virtual environment still), with

source $basedir/bridge-venv/bin/activate
python $basedir/sg-jira-bridge/webapp.py --settings $basedir/sg-jira-bridge/settings.py --port 9090
         

If you now navigate (or curl) to localhost:9090 you should get a basic HTML page stating The server is alive. If that looks good go ahead and shut it back down and configure a systemd unit for it. The bridge provides a service.py for easy of installation.

As our bridge application is running on the cloud we'll be running it behind Nginx, which will be facilitating TLS for our connection. If your Jira and Shotgun are locally hosted, you could technically skip this step (although I recommend you setup HTTPS anyway).

The certificates we're using are provided by the amazing Lets Encrypt service.

The setup is fairly trivial so we'll skip the basics here, just remember to change the root location to proxy_pass to our web application. That should look like:

# nginx.conf snippet
server {
    server_name bridge.aws.pixel-nexus.com;

    # Server level configuration here...

        location / {
            proxy_pass http://localhost:9090/;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
        }
# ... rest of the config ...
}
        

Lastly, we'll need to enable a rule in SELinux to allow Nginx to connect to the bridge: setsebool httpd_can_network_connect 1. To make this survive a restart we'll have to specify it's permanent as well, which is achieved with the simple -P flag: setsebool -P httpd_can_network_connect 1.