Tuesday, January 28, 2014

Building Android apps with Python and Shotgun

Happy new year!!

Hi, a lot of work starting this 2014, but let's start with the first post of the year.

Our producers are always in the rush, calling clients, delivering projects, etc and all the time they are asking us for tools to manage projects on their smart-phones, so we think that some apps could also be useful for the artists when simple functionality is needed. So we did some research and found that is really easy to prototype Python apps in Android, and this post is for this...

Building Android apps with Python and Shotgun

For me, Android is the most flexible and configurable OS for smart-phones, and also has its own scripting layer thanks to the SL4A project that exposes the Android API to several languages like Python, Perl, JavaScript, and others.

For us, the important one is Python, this because the official Shotgun API is in this language.

So, let's start...

Firs of all you will need to download and install two apk's on your phone, one is the scripting layer for android and the second its the python interpreter, the installation order doesn't matter.

Scripting Layer for Android (SL4A)

Python Interpreter for Android (Py4A)

If you've never installed apk's before you maybe need to set your phone to allow the installation of non market apps (Google Play), this could be different in all the Android versions and distros of all the phones but commonly is like this:
  1. Go to Settings.
  2. Open Security.
  3. Click Unknown sources.
Once you have installed the apps, it's time to install the interpreter itself, this will be performed by the Python for android app, open it and select install. This will download all the Python modules and setup the interpreter.

When this is done, now you can open the SL4A app and see some sample python scripts that uses the Android Python API, very simple to use, here is an example:

import android

droid = android.Android()
name = droid.getInput("Hello!", "What is your name?")
print name  # name is a namedtuple
droid.makeToast("Hello, %s" % name.result)

All this scripts are located in the following path: /sdcard/sl4a/scripts
Here you can create new ones, and organize them in folders if you want, check the SL4A User Guide

To learn about the commands and all the functionality you can access from the API, the best places are the docs, and also all the examples they have.

Now, we need the Shotgun API to perform what we want, this is really easy, only put the shotgun_api3 folder in the following path: /sdcard/com.googlecode.pythonforandroid/extras/python/

By doing this, you can perform "from shotgun_api3 import Shotgun"

At this point you can create scripts that works like command line or auto scripts but, what if we want an app with a graphical user interface?

Well, Android API has it's own way to do this by XML layouts, and is really easy to use. 
Follow this link to check how to use it.

In order to make this more simple, I use Android Studio, where I design the layout in a graphic mode like Qt-Designer, dragging widgets and adjusting properties, then I copied the formatted xml and saved as an xml file to load it into my code as a string:

layout = open("layout.xml","r").read()

And that's it, you can start to build python apps for Android that can interact with Shotgun, let me show you an app to Send Support Tickets:

This example is made of three files: The main python script, the xml layout and a banner image

The banner image:

Here the XML Layout:

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"



                android:text="Write your request:"


                android:text="Check as urgent!"


                    android:text="Send Ticket"

                    android:text="Close App"
            android:text="Status Bar: "

Here the main python script:

from shotgun_api3 import Shotgun
import android
import os
from time import sleep

sg = Shotgun("Your Shotgun Site","Your API name","Your API code")

def eventloop():
  global banner_file
     "Status Bar: Waiting ticket description...")
  while True:
    print event
    if event["name"]=="click":
      if id=="close_app":
      elif id=="sent_ticket":
           "Status Bar: Sending Ticket... Please Wait...")
        ticket_body = droid.fullQueryDetail("ticket_body").result
        ticket_body = ticket_body["text"]

        urgent = droid.fullQueryDetail("urgent_check").result
        urgent = urgent["checked"]

        if urgent == "false":
          ticket_status = "2"
          ticket_status = "1 (High)"

        data = {
          "title": "Support ticket",
          "project": {'id': 123, 'name': 'Support', 'type': 'Project'},
          "description": ticket_body,
          "sg_priority": ticket_status

        # Let the user know we're doing something
        droid.dialogCreateSpinnerProgress('Please Wait...', 
            'Creating Ticket in Shotgun', 

        Ticket = sg.create("Ticket", data)


        droid.dialogCreateAlert("Ticket succesfully created!")

           "Status Bar: The ticket was created...")
           "Status Bar: Waiting ticket description...")

    elif event["name"]=="screen":
      if event["data"]=="destroy":

    elif event["name"]=="about":
      droid.dialogCreateAlert("Autor: Hasiel Alvarez", 
          "Mail: hasielhassan@gmail.com")

dirname, filename = os.path.split(os.path.abspath(__file__))
xml_file = dirname + os.sep + "layouts" + os.sep + "activity_main.xml"
layout = open(xml_file,"r").read()

banner_file = str(dirname + os.sep + "images" + os.sep + "help_banner.png")
banner_file = "file:" + os.sep + os.sep + banner_file[4:] 


From here, there are a lot more things to review:
By using this approach to build mobile apps for Shotgun you are delivering the source code to the users including the api access to Shotgun and this won't be acceptable for most studios.
Also it's difficult to setup the first time, and not all the users can do it.
And maybe the most important one, this approach works for Android and most (if not all) of the producers use iPhones, and for that, currently I haven't found an easy way to script apps in Python on iOS.

But it's good to know and an interesting and really quickly way to prototype apps you need and make changes really easy, so the users can give you feedback and get all the info needed to build the final app, maybe with Phonegap or Kivy.

Wednesday, December 18, 2013

Scene Optimizer

In order to keep our Maya scenes clean and free of common errors to avoid delays in the production, we needed to create a new tool that could check and automatically fix most of this problems.

So, the scene optimizer was born:

In this first version it did the job and did it well, yet it took too much time, (too many loops cycling trough all the nodes in the scene), so we had to optimize the optimizer.

V2 Arrived:

It might look smaller but it has 10 times more power!

Rewritten code that allows many cycles to be avoided.

Changed to commands that allow to process multiple objects at a time.

Removed unnecessary checks for the new version of RenderMan Studio.

Converting some of the checks and fixes to other tools.

Denying the artist to Publish using Shotgun or save the file if it still have errors.

New checks and fixes added.

Obligatory check of some of the functions and optional for others.

What does it check?
In general:
  • That every object is grouped where it belongs (meshes with meshes, joints with joints, etc).
  • That every node has a unique and irrepeatable name.
  • That all the meshes are correctly visible in render times.
  • There are no image planes left in the scene.
  • The project has to be set in the correct path.
In modeling:
  • Meshes has no history left.
  • No color sets from Zbrush.
  • Just a single UVSet, if more then copied to the main set and deleted.
  • No locked normals from Zbrush or Topogun.
  • Nothing has shaders or textures or extra nodes that affect render.
In animation:
  • That every key is has not a stepped tangent or that keys are really close to each other.

The next step for this tool is to combine it with others we already have, like the one that checks for interpenetrations between the meshes:

Saturday, December 14, 2013

Auto Mounting NFS Shares in Mountain Lion

Network File System (NFS) is the most common way for file sharing on Network-attached storage (NAS) server in *NIX systems like, Mac OS X (Darwin), Linux distributions, FreeBSD, etc.
Mountain Lion, can be setup as an NFS client to access shared files across the network.

By default, Autofs mounts network files systems are defined via Directory Services, using /etc/fstab and NFS file systems, all accordingly to its master table, /etc/auto_master.

If you want to define mount points not covered by Autofs default configuration, without resorting to Directory Services or the deprecated /etc/fstab, you will have to add a mapping (via a auto_* file) to auto_master.

This is the way we found to best suit our needs.

1. Create the folder "/etc/automounts"
sudo mkdir -p /etc/automounts

2. Create the file "nfs"
cd /etc/automounts
 sudo touch nfs
sudo vi nfs

3. Add your NFS shared servers, options and mounting paths, for example:

   /chitara/videos             fstype=nfs,soft,intr,rw,tcp
   /chitara/Musica  fstype=nfs,soft,intr,acl,tcp
   /chitara/Tutoriales fstype=nfs,soft,intr,acl,tcp
   /leonidas/datos fstype=nfs,soft,intr,acl,tcp
   /leonidas/proyectos      fstype=nfs,soft,intr,acl,tcp
    /mummra/remisiones   fstype=nfs,soft,intr,acl,tcp

4. Edit the file "/etc/auto_master"
Add the path where nfs file is

This way, we can have a much better control on the workstations running OS X Mountain Lion.

Monday, December 9, 2013

Alembic Pipeline

We have come to realize the amazing potential of the Alembic format in almost every step of the process.

It can clean the hierarchy of the nodes, bake the animations, reduce scene sizes, increase the fps count in the viewport, even reduce render times!!!. So in order to harvest all this power we developed a series of tools to help the artist to manage, export and load the files seamlessly, easy and fast, while keeping them organized and consistent.

We have a standalone version and one integrated with Shotgun and Tank (Pipeline Tookit), both versions do almost the same in the guts. 

Behind the tools:

  • They first search through the hierarchy of all the objects finding patterns of whether they belong to the same character or not, so all the geometry goes in the same group in the end.
  • They search for all possible cameras in the scene finding the right ones for the shot.
  • In the Stand Alone version the user is asked for a time range and a path to save the file, in the Integrated version all this information is loaded from the Shotgun database.
  • Then the files are versioned, labeled and saved properly together with the scene where they come from.
  • The Pipeline Toolkit version is implemented as a hook in the Publish App but ours is designed in a way that versioning is recognized by the Maya Scene Breakdown App (which we also have to tweak for this to work), rather than this one.

With this method we were able to reduce scene complexity and file size to around 1% of the original way. Here is an example demonstrating the principle: (Simple scene, single character, simple animation)

Now the same scene with the same character and the same animation, but cached with Alembic:

Modo to Maya modeling workflow

Let's cut to the chase....

CG animation is a medium where everything has to look good and most of the time be as fast as possible, that does not apply only for the end results, but for all the steps of the process. The tight bond between the concept, texture and animation with the modeling area, makes the last a high demanding and very important step for making a CG animation,  so each one of the models made not only need to look very good but also  they need to be done quickly. 

Having said that, why do not use the technologies available to accelerate the process?. Like pretty much everything in this world, different approaches to sculpt 3D objects have been made over time and each one of them have their advantages, therefore, having the model ready for production could be done more faster than just working with just one 3D package.0

Cluster's modeling artists know how to take advantage of  3D modelers features, but even though there is a waste a time; is tiresome open to a scene with a 3D Modeler, sculpt, save the scene, open other 3D modeler package, open the scene, change... well you get picture, and in fact  pixologic got the picture as well, with that in mind they developed GoZ;  if you do not know what is it: "GoZ is a dynamic bridge between Z-Brush and other 3D packages built around a specific file format, the GoZ file."

The thing we learnt about GoZee

"If you have a file format that both modelers read, you can make a Pipe, preferably an automated pipe".

So GoZ has the GoZ file to make a bridge between ZBrush and other 3D packages (and saving time in the process), but what happens if Z-Brush is out of the pipeline picture?. Cluster's artist wanted exactly that,  they did not want to have ZBrush in  between the Modo-Maya workflow. 

Well we had two options make more clicks or make a program who handles that. Of course for making an automated pipe we needed a way to interact with the 3D package, fortunately  Modo and Maya have (python and MEL).

The way we learnt to make that pipe  is to select all the meshes in the scene, copy each one of them in a temporary scene, save the meshes names in txt file and export one by one those meshes, all that behind the scene! (there are actually a tweaks  in the process like checking if there is a mesh previously saved and asking what to do next, but the basic idea is copy, past and export).

In Maya we read  the txt, locate the exported files and import them.

Let us exemplify:

As you can see there is one mesh in the scene, (but probably you can not see that is called ToMaya), so we run the script and  Modo asks for a few checks, after that the mesh is exported in a file called ToMaya and the name of the mesh is saved in a txt file, Modo sends the signal to run a mel script that loads the exported, like this:

In our approach the exported scenes are an obj file, but it can be done with  any format available in the 3D packages (we should start thinking in make alembic files.)

With all that said, the code, let us know if this was useful 
Thanks for reading, until next time

Wednesday, November 27, 2013

Tractor login authentication through Shotgun

By default the Pixar's Tractor web-based monitor does not authenticate users by using passwords, so for sanity's sake is a good idea to configure an authentication method.
Shotgun recently released an authentication method with its API so we took advantage of this feature for our Tractor monitor login.

This is great because Shotgun is the backbone in our Pipeline and everything is kept centralized in one place, and now every single user with an active Shotgun account can also login into Tractor using his/her same credentials.

To do this we modified the trSitePasswordHash function so that it follows:

trSitePasswordHash function ( passwd , challenge){
   // By default , passwords completely ignore Tractor Dashboard ,
   // And login names are only used for basic event tracking and to
   // Restrict some UI actions . The return value of this function ,
   // Below , JavaScript Should be the ' null' When the default value
   // Password -ignore behavior is Desired .
   // Otherwise , This function should return a string That is the
   // Site- customized encoding of the typed -in user 's plaintext password ,
   // Also possibly Incorporating Given the server -supplied one- time
   // Random challenge string . The matching site-specific server-side
   // Handling of this string MUST be encoded in the file Implemented
   // (Tractor) / config / trSiteLoginValidator.py
   // Return null; / / passwords are ignored (default factory setting )
   // Or
   // Return your_custom_encoding ( passwd , challenge );
   return passwd ;

The previous feature is in the trSiteDashboardFunctions.js file

You also have to modify the main function in trSiteLoginValidator.py

def main () :
   user = raw_input ()
   challenge = raw_input ()
   pw_hash = raw_input ()

      p = pwd.getpwnam (user)
      except KeyError :
      return 1 # unknown user

      SG_SERVER = ' http://yourstudio.shotgunsoftware.com '
      SG_SCRIPT = ' yourScript '
      SG_KEY = ' d34dasdklajklas923849343daskdljaskld '
      sg = Shotgun ( SG_SERVER , SG_SCRIPT , SG_KEY )
      if sg.authenticate_human_user(user, pw_hash ) :
         return 0
         return 1

And you're done! Your users can now login into Tractor using their Shotgun credentials

Thursday, November 14, 2013

How we use git as an update mechanism

The little problem...

As many other studios out in the wild, here at Cluster Studio we have a lot of artists, producers and other employees, who use penguins, fruits and holes in the walls as OSes. Our work as the development team is to provide applications that help them to do their work easier (or harder as they sometimes perceive it). So we needed to find a painless way to distribute software across all the users and all the platforms.

Since some time ago we use git to version control the R&D projects and share code between us, but nothing else beyond this. Fortunately we are changing that, now we use a workflow as the one described on this Bitbucket's article (or at least we try). A better development workflow is something that deserves another complete post on this blog and I think we will never find a definitive solution for that. So let's focus on today's topic DISTRIBUTION.

A local solution...

Since all the artists work at desktop workstations we can mount a shared partition for all of them, so they can launch the standalone applications from there (most of the apps are python scripts). We were doing that since long time ago without any correct version control. So we decided to use git repositories to manage that problem and since, in today's world, everything must be "on the cloud" at least twice to prove it exists, we choose Bitbucket over our old github repositories to host our work.

Why bitbucket?... because their business model fits better for a small development team with a lot of private repositories, like us. We only have to pay for the number of team members and not for the number of private repositories. So we use github for Open Source initiatives (like those of today's post) and Bitbucket for our private parts (well, not that parts).

Bitbucket also has a nice API and a Hook system that let us automate the pulls to our local production repositories. This way the (lazy) members of the development team don't need to manually update both repositories every time. We just make a push to bitbucket and a hook sends a request to our homemade web service that finds the right path to the repository and executes the pull command (kind of magic). You can fork it here.

A little explanation next:

Our weapon of choice for the web service is flask, mainly because its good intentions and great flexibility. Following Miguel Grinberg's tutorial it was easy to develop it. Using wsgi we deploy the web service at Apache.

At Cluster Studio we use Shotgun as our database, including tool tracking, so we get the repositories locations from there. Not much to say here, you can  change it to use any way that fits you to get the local path to your repository.

For those who can't stay...

Some users (usually the producers using laptops, sorry MacBooks Pro) can't stay in our local network all the time but they still need to use our applications, always the latest version, so we came up with the idea of giving them a local repository and develop an auto update system. You can fork it here.

Basically it works by reading a config file to look for the path to the local storage, the URL to the repository and the name of the script, it can include other repositories as dependencies just in case needed. Executes a pull or clone for every repository and then run the given script.

Since executing something from terminal usually is too complicated for our mac users we also have an applescript that runs the python command. Save it as an app and your users will never know the difference.


sh deserves a mention apart because it is a great subprocess interface that helps to handle the system interaction.

Hope you find it useful, interesting or at least as a not to do reference.
We'd love to see how you deal with this in your studio.

The code