Instancing Objects With Particles [maya, python]

I’m putting this in the ‘solved’ category, but here’s one last update on the matrices problem I’ve been tackling! It’s been a great learning tool, as there are a million ways to do it wrong, but they all involve learning how to write better Python, or go deeper into the Maya API.

Here’s a pic of one of the textbook examples that helped get me started with the matrix stuff. My eyes lit up when I saw the words “vertices” and… Actually, no other words excited me. You can see the large letter N on the left though, scaled/sheared a bit. That’s the stuff we’re doing in Maya!

leaf_algebra

Alright, onto the code!

#1 Import Your Business Most of this script is pure Python/PyMel, and everything else that’s been imported is for a specific use. OpenMaya is for the matrix math, because it was much faster than the PyMel counterpart when there are 1000’s of matrix operations. Maya’s particles are accessed through OpenMayaFX, and the rest of the imports are all one line uses; picking some ‘random’ verts, accessing the constant Pi, and timing the whole script with DateTime.

1 import maya.OpenMaya as om
2 import maya.OpenMayaFX as omfx
3 import pymel.core as pm
4 import random
5 import math
6 from datetime import datetime

#8 Stay Classy This first class is basically the whole script. We open with the doc string, then declare some class variables that any instance of this class would have access to, and announce that they’re to be treated as constant (why they’re placed in that location and sans ‘self.’, and why they’re in ALL_CAPS, respectively). In other languages, these are called static constant variables.

8 class ShellsToInstances():
9     """Converts the shells of a mesh to nParticle instance objects."""    
10    MAX_INVERSE_TRIES = 20
11    INSTANCE_NAME = 'base_instance'
12    ORIG_MESH_NAME = 'original_mesh'

leaf_code2a

#14 Instant Variables Lines 16-19 aren’t necessary for the code to run, and could’ve been omitted, but these are the instance variables that the whole class will have access to, and I’m ‘announcing’ them here just so everyone knows what they are.

14 def __init__(self):
15
16     self.large_mesh = None # MeshInfo class object
17     self.instance_mesh = None # MeshInfo class object
18     self.shell_transform_matrices = [] # MTransformationMatrix list
19     self.verts_4 = [] # Four ints; the 4 vert indices for matrix stuff.
20     self.DoIt()
  • ‘large_mesh’ is the mesh we’re acting on,
  • ‘instance_mesh’ is the shell/shape we’ll be instancing around with nParticles.
  • The third variable is going to be a GIGANTIC LIST of transform matrices that we’ll use to position/orient the particle instance objects.
  • ‘verts_4’, is just a short list of the vertex indices that we’ll use across all the shells.
  • Line 20 is where we actually set things into motion!

#22 Do It The class method “DoIt”, called in the last line of __init__, calls every other method and basically runs the script. I’ve been trying speed this thing up, so it opens with a stopwatch to start the clock on line 24. If everything goes smoothly, the clock stops on line 34, and the results are printed. This is a quick ‘n dirty way to test speed, and isn’t a replacement for cProfile (which has been a trick to garner useful info from for me).

22 def DoIt(self):
23    
24     time_start = datetime.now()
25    
26     try:            
27         self.large_mesh = MeshInfo(self._FindMeshToActOn())
28         self.instance_mesh = MeshInfo(self._CreateBaseInstance(self.large_mesh))
29         base_inverse_matrix, self.verts_4 = self._FindInverseMatrixOfShape(self.instance_mesh.shape)
30         self.shell_transform_matrices = self._FindTransformMatricesOfShells(base_inverse_matrix, self.large_mesh, self.verts_4)
31         self._CreateInstanceShells()
32         self._CleanUp()
33        
34         time_end = datetime.now()
35         print '\n#---------------------------------------------------#'
36         print 'total time: ' + str(time_end - time_start)
37    
38     except Exception, e:
39         print e
  • #27 Get The Big Mesh Now we’re actually doing some work! First thing, we’re finding the largest mesh in the scene to act on. If we find it, we return its shape, then cast/instantiate to a helper struct called MeshInfo, that reads things like # of vertices, shells, and some other info we’ll want later on. That class/struct is at the bottom of the script if you want to take a look. It almost isn’t necessary, but did reduce some duplicate code.
  • #28 Duplicate A Shell Using info from the first shell of the mesh we just found, create a new object, move it to the origin, then cast it to another MeshInfo object.
  • #29 Find Neo Using the mesh we just created, create a data matrix of 4 vertex positions (x,y,z). But! Make sure the resulting matrix is invertible so we can use it later for some math. Return that inverse matrix, and the four verts we used.
  • #30 Giant Loop Using the inverse matrix we just found, use it to compare the postion/orientation of the base object to that of every shell in the large_mesh. If we treat that base mesh as ‘zeroed out’, the resulting matrices we find will represent how each shells is positioned/rotated relative to it.
  • #31 Create ‘Em Now that we know the transforms of each shell in the large_mesh, we can create the instance objects (with nParticle object instancing), and transform them so they ‘replace’ each original shell.
  • #32 Clean We’ve done the work, now just clean the scene up a bit.

#49 Except… A few of the methods are peppered with try/except/raise statements. The ‘raise’ keyword is a great way to send a signal that things aren’t going well, and to abort the current operation. Instead of returning a value at the end of a function, an exception is ‘raised’ up to be caught and dealt with. I’m using them here to deal with the scene not having an appropriate mesh to act on. You can see in the DoIt method that all the exceptions are pushed up into that try/except block. I really like this way of doing things, as you don’t have to adjust your return statements to communicate that something’s wrong, .i.e. return a_value, a_message_indicating_success_or_failure. I could have returned ‘None’ instead of raising an exception, which would make this method more versatile, but less… fun?

42 def _FindMeshToActOn(self):
43
44     pm.select(clear = True)
45     geos = pm.ls(type = 'mesh')
46     try:
47         mesh_shape = geos[-1]
48     except:
49         raise Exception('''Can't find a mesh to act on...''')
50        
51     for geo in geos:
52         if geo.numVertices() > mesh_shape.numVertices():
53             mesh_shape = geo
54     mesh_info_obj = MeshInfo(mesh_shape)
55     if mesh_info_obj.num_shells < 2:
56         raise Exception('''This mesh doesn't have enough shells...''')
57        
58     return mesh_shape

#60 Make The Base If we found a good mesh to act on in the previous method, now we can duplicate one of its shells as a new object, then move it to the origin. (This new object is the shape we’ll be instancing around). There are a million ways to do this, with basic Maya commands like polySeparate and maybe polyExtract. But, I’m not using those! They’re too easy! Aaaand, one requirement of this problem is that original mesh can’t be modified, which all of those commands will do. So what’s the solution? How do you duplicate one shell of the large mesh, without actually affecting it? It turns out you can read all of its mesh data, then use it to create a new object from scratch with some cumbersome OpenMaya commands.

There are just 5 things we need to create a mesh from scratch, and that’s what this method is about.

  1. How many verts are there?
  2. How many faces are there?
  3. What are the vertex positions?
  4. How many verts does each face have?
  5. Which faces have which verts?

Once you gather all that info, you can call one command to create a new mesh from scratch, as on line 87. This method uses some shortcuts for interacting with the Maya API. The PyMel command to get MFn and MIt objects are much shorter ways to go! Have a mesh? A face? Just call __apimobject__, __apimit__, or __apimfn__ as needed!

60 def _CreateBaseInstance(self, large_mesh):            
61    
62     mesh_mobj = large_mesh.shape.__apimobject__()
63     mesh_fn = om.MFnMesh(mesh_mobj)
64    
65     face_iter = large_mesh.shape.f[0].__apimit__()
66     face_iter.reset(mesh_mobj)
67    
68     face_vert_counts = om.MIntArray()
69     face_verts = om.MIntArray()
70     while (face_iter.index() < large_mesh.faces_per_shell) and not face_iter.isDone():
71         face_vert_counts.append(face_iter.polygonVertexCount())
72         vert_indicies_on_one_face = om.MIntArray()
73         face_iter.getVertices(vert_indicies_on_one_face)
74         for i in range(vert_indicies_on_one_face.length()):
75             face_verts.append(vert_indicies_on_one_face[i])
76         face_iter.next()
77    
78     position_array = om.MFloatPointArray()
79     for i in range(large_mesh.verts_per_shell):
80         temp_point = om.MPoint()
81         temp_float_point = om.MFloatPoint()
82         mesh_fn.getPoint(i, temp_point)
83         temp_float_point.setCast(temp_point)
84         position_array.append(temp_float_point)
85    
86     mesh2_fn = om.MFnMesh()
87     mesh2_trans = pm.PyNode(mesh2_fn.create(large_mesh.verts_per_shell,
88                                             large_mesh.faces_per_shell,
89                                             position_array,
90                                             face_vert_counts,
91                                             face_verts))
92     pm.xform(mesh2_trans, cp = True)
93     pm.move(0,0,0, mesh2_trans, rpr = True)
94     pm.makeIdentity(mesh2_trans, apply = True)
95     
96     return mesh2_trans.getShape()

#98 Find the Matrix I went over this method in more detail in the previous post, so check that out. I added a few try/except/raise statements though, and they’re used for two different purposes here: There’s a problem and the program should gracefully stop or, There is no problem, but I need to call a method that will probably throw an error until I adjust its input correctly, or give up trying (line 124).

leaf_code2b

#134 Find All The Matrix I think this is the slowest method in the script, purely because of the matrix stuff. Here we’re finding the transformation matrices of each shell in the large_mesh. After testing a bit, I found the PyMel matrix stuff was pretty slow compared to OpenMaya, so the method has been updated to reflect that. I switch between MMatrix and MTransformationMatrix because they each have different methods that are necessary. MMatrix has a transpose() method and supports multiplication, while MTransformationMatrix doesn’t.

134 def _FindTransformMatricesOfShells(self, vert_matrix_orig_inverse, large_mesh, verts_4):
135            
136     shell_ids = range(0, large_mesh.num_verts, large_mesh.verts_per_shell)
137    
138     trans_matrices = [None]*len(shell_ids)
139     all_positions = large_mesh.shape.getPoints(space = 'world')
140
141     for i, shell_id in enumerate(shell_ids):
142        
143         matrix_list = [all_positions[shell_id + verts_4[j]].tolist() + [1] for j in range(4)]
144         matrix_list = [item for sublist in matrix_list for item in sublist]
145         vert_matrix_new = om.MMatrix()
146         om.MScriptUtil_createMatrixFromList(matrix_list, vert_matrix_new)
147         vert_matrix_new = vert_matrix_new.transpose()
148
149         trans_matrix = vert_matrix_new * vert_matrix_orig_inverse
150         trans_matrices[i] = om.MTransformationMatrix(trans_matrix.transpose())
151    
152     return trans_matrices

#138 Big List Since the list of matrices is going to be HUGE, I want to avoid changing/interacting with the list as much as possible, so I declare its initial size here, and just populate it with Nones.

#154 nParticles!! Here’s where we actually create and postion/rotate the instance objects! Why use particles, to that end? Good question! It turns out you can affect the position/rotation of each instance object in just ONE CALL with particles. This is much better than my previous approach of looping through each object and explicitly setting their transforms. The only catch, obviously, is that you need to compile all that transform data into a usable format. That’s why there’s so much list comprehension here and elsewhere in the script – just moving the same numbers around so they’ll play nice with the methods we need to use.

154 def _CreateInstanceShells(self):    
155    
156    # p_ means particle. The variable names are so long!
157    p_translates = [self.shell_transform_matrices[i].getTranslation(om.MSpace.kWorld) for i in range(len(self.shell_transform_matrices))]
158    p_translates_simple = [(i.x, i.y, i.z) for i in p_translates]
159
160    
161    nparticles_transform, nparticles_shape = pm.nParticle(position = p_translates_simple)
162    p_instancer = pm.PyNode(pm.particleInstancer(
163                                                    nparticles_shape, addObject=True, object=self.instance_mesh.shape,
164                                                    cycle='None', cycleStep=1, cycleStepUnits='Frames',
165                                                    levelOfDetail='Geometry', rotationUnits='Degrees',
166                                                    rotationOrder='XYZ', position='worldPosition', age='age'))
167
168    pm.setAttr('nucleus1.gravity', 0.0)
169    nparticles_shape.computeRotation.set(True)
170    pm.addAttr(nparticles_shape, ln = 'rotationPP', dt = 'vectorArray')
171    pm.addAttr(nparticles_shape, ln = 'rotationPP0', dt = 'vectorArray')
172    pm.particleInstancer(nparticles_shape, name = p_instancer, edit = True, rotation = "rotationPP")
173    
174    p_rotates = [self.shell_transform_matrices[i].rotation().asEulerRotation() for i in range(len(self.shell_transform_matrices))]    
175    p_rotates_simple = [[i.x/(math.pi) *180, i.y/(math.pi) *180, i.z/(math.pi) *180] for i in p_rotates]
176    p_rotates_simple = [item for sublist in p_rotates_simple for item in sublist]        
177    
178    vector_array = om.MVectorArray()
179    for i in range(0, len(p_rotates_simple), 3):
180        vector_array.append(om.MVector(p_rotates_simple[i], p_rotates_simple[i+1], p_rotates_simple[i+2]))
181        
182    particle_fn = omfx.MFnParticleSystem(nparticles_shape.__apimobject__())
183    particle_fn.setPerParticleAttribute('rotationPP', vector_array)
184    particle_fn.setPerParticleAttribute('rotationPP0', vector_array)

#161 LOOK! Look how easy it is to create a bajillion particles, in the exact locations you want! One command, and the only argument is the list of positions! You don’t even need to say how many particles – it does that for you with the positions.

pm.nParticle(position = p_translates_simple)

#162 Object Instancing Here, in one command, we’re placing a copy of that mesh we created earlier at every particle location.

pm.particleInstancer(nparticles_shape, addObject=True, object=self.instance_mesh.shape...

#169 Rotation Harder The rest of the method is devoted to getting the particles oriented correctly. It’s a bit tricky, as they need new attributes for their starting rotation (rotationPP0), and general rotation (rotationPP). You can get away with only one of these attributes, but things will go pear-shaped if you scrub the timeline at all.

#186 Almost Done The real work is complete, now we just need to clean things up a bit, and as a final gesture, select the four verts from the instance object we created early. Because it’s been instanced around with our particle commands above, you should see those same verts highlighted on EVERY INSTANCE OBJECT. Pretty sweet, huh?

#196 __repr__ I overwrote the default __repr__ method, only because I didn’t like the memory address of the script object being printed out every time it’s run. Maybe I could make it something cooler.

def __repr__(self):
        
    return ''

leaf_code2c

#201 MeshInfo Lastly, here’s that helper struct I mentioned earlier. Not totally necessary, but actually made things easier/cleaner overall.

201 class MeshInfo():
202     """Simple struct to hold relevant mesh data."""
203     def __init__(self, shape = None):
204         self.shape = shape
205         self.trans = None
206         self.num_shells = None
207         self.num_verts = None
208         self.verts_per_shell = None
209         self.faces_per_shell = None
210        
211         self.populate_info()
212    
213     def populate_info(self):
214         if self.shape is None:
215             return
216         self.trans = self.shape.getParent()
217         self.num_shells = pm.polyEvaluate(self.shape, shell = True)
218         self.num_verts = self.shape.numVertices()
219         num_faces = self.shape.numFaces()
220         self.verts_per_shell = self.num_verts / self.num_shells
221         self.faces_per_shell = num_faces / self.num_shells

Conclusion:

That’s it! I’m done with this! It’s still not as fast as I want, and I think could take about 15 minutes to run on a million objects, but it’s been a fun experiment to play with. I swear I’m done!

Sweet, Sweet Matrices: Part 2

Here’s the code for the matrices problem I talked about in the previous post. This little guy is a PNG (until I find a good way to display code without you having to scroll left/right). The details and explanations for what’s going on are below as well. Check it out!

tree_leaf_code2

#16 Get The Base Shape: This section rips off the first shell in the main mesh, moves it to the origin, and resets its transforms. This new object will be the base shape that we’ll repeatedly compare to the other shells to determine their transformations, as well as be the instance shape that replaces those shells.

16 first_shell_trans, the_mesh_trans = pm.polySeparate(the_mesh_shape, sss = 0, ch = False)
17
18 pm.xform(first_shell_trans, cp = True)
19 pm.move(0,0,0, first_shell_trans, rpr = True)
20 pm.makeIdentity(first_shell_trans, apply = True)

#29 Find Its Invertible Matrix: Now that we have the base instance shape isolated, we need to pick four of its vertices to define a matrix of global vertex positions. Later, for each shell, we’ll repeat the process for those same vertex indices, then compare the two matrices to determine the shell’s translate/rotate/scale/shear values, relative to our base shape.

One hiccup though, is picking the right combination of vertices that form an invertible matrix! Because we’ll be doing some math that requires a matrix with an inverse, this can be a roadblock. The solution here is brute force! Keep picking four verts from the base mesh, at random, until an invertible matrix is found. You can see on line #29 that we start with a list of all the vert indices, then on line #32, we pick four from that list, at random.

On line #40, we’re attempting to find the inverse. If it fails, we try again with another pass through the while loop at #27. (Ignore the flipped_positions variable, I get to that later)

29 verts_all = range(instance_shape.numVertices())    
30 verts_4 = []
31 for i in range(4):
32     verts_4.append(verts_all.pop(random.randint(0, len(verts_all) -1)))
33
34 shell_id = 0
35 positions = [instance_shape.vtx[shell_id + verts_4[i]].getPosition(space = 'world').tolist() for i in range(4)]
36 flipped_positions = [[row[i] for row in positions] for i in range(3)]
37 flipped_positions.append([1,1,1,1])
38 matrix_orig = pm.datatypes.MatrixN(flipped_positions)
39
40 try:
41     matrix_orig_inverse = matrix_orig.inverse()
42     inverse_found = True        
43 except:
44     inverse_try_count = inverse_try_count + 1

#48 How Many Shells ‘N Verts? If we actually found an invertible matrix, then proceed! The first step is to find out how many shells are in the main mesh, so we can loop through them with a list of their starting vertices (0, 20, 40, 60,… last_shell’s_first_vertex).

48 the_mesh_shape = the_mesh_trans.getShape()
49 pm.select(the_mesh_shape)
50 num_shells = pm.polyEvaluate(shell = True)
51 pm.select(clear = True)
52 verts_per_shell = the_mesh_shape.numVertices() / num_shells
53 shell_ids = range(0, the_mesh_shape.numVertices(), verts_per_shell)

#59 Loop Through All The Shells: Here’s where most of the actual work is done. For each shell, grab the world positions of the four specific vertex indices we found at #32, then throw them into a matrix. There are a few details that have to be right though; the positions need to be in column vectors, not row vectors (so we reorder them on #60), and to make this a 4×4 invertible matrix, we need to append a W value of 1 to each vector. I think “homogeneous” might be a term to use here.

59 positions = [the_mesh_shape.vtx[shell_id + verts_4[i]].getPosition(space = 'world').tolist() for i in range(4)]
60 flipped_positions = [[row[i] for row in positions] for i in range(3)]
61 flipped_positions.append([1,1,1,1])
62 matrix_new = pm.datatypes.MatrixN(flipped_positions)

#64 MATH!! Ok, the REAL work is done here, in just a line or two with matrix math! This is why we found an invertible matrix earlier, to use here. We’re determining the transformation that was applied to each shell (again, relative to our base shape), solving for x in xO = N, where x is the transformation that’s applied to the (O)riginal base shape, resulting in (N)ew vertex positions. Just multiply both sides of the equation by O^-1 to find x. That O^-1 is the inverted matrix we found earlier.

(One catch in matrix math though is that you need to do this is in the correct order. Unlike regular multiplication, where xy = yx, that ISN’T the case here.)

64 transformation_matrix = matrix_new * matrix_orig_inverse

#65 Flip It, For Real: The transformation matrix we just found has the right numbers, but they need to be flipped around along the matrix’s diagonal, so they’ll play nice with how Maya’s transformation matrices are ordered.

65 transformation_matrix_transposed = transformation_matrix.transpose()

#66 Make This Whole Thing Slow: We found the answer for this shell, now store it in a list to use later on our instance objects!

66 instance_transforms.append(transformation_matrix_transposed)

#72 Create And Align The Instances! Now, let’s loop through all the transformation matrices we found, and apply them to the instance objects we’re creating.

72 for i, transform in enumerate(instance_transforms):
73     one_instance = pm.instance(instance_shape, name = 'mesh_instance_' + str(i).zfill(3))[0]
74     one_instance.setTransformation(transform)

#79 A thousand apologies:

79 else:
80     for i in range(1000):
81         print 'apology'

Conclusion:
This solution is complicated, and I’d love to see something simpler! I’d also like to increase the speed, so it can actually work on a million+ shells. I think the main culprit is accessing the main mesh, repeatedly, to query four vert positions in each shell. Maybe if I gather that data in one pass? Or declare the size of the arrays/lists involved so they don’t need to be reallocated with each loop that changes their size? There are a few options here, luckily. If those fail, switching over to OpenMaya would be the next step for a hopeful speed increase. I think there’s a function for grabbing multiple vert positions with one call in there! That may be the secret.

Sweet, Sweet Matrices

Here’s a fun Maya Python problem I ran into recently. It took me a long time to figure out that it involved matrices, as it first seemed like a good candidate for, well, I wasn’t sure even where to begin, but I thought some vectors would be involved. After many dead-ends and dabbling in quaternions, I finally discovered the simplest solution (and only one I could find) uses matrix math!

I’m excited about this because linear algebra has haunted me since college. Back then, I mistakenly assumed “Elementary Linear Algebra” was a refresher course, or a well-deserved break from the ‘real’ math classes. I was wrong. It should’ve been called, “MATRICES OR DIE!

Here’s the actual problem: You’ve got a million identical polygonal objects, randomly translated and rotated about, then combined into one mesh. Write some code to replace each piece with an instance object/locator.

Why is this difficult? Because the original objects were combined into one mesh, they no longer have any object transform data. Now, the only thing to query that defines their translation, rotation, scale, and even shear is the vertex positions. So how can you determine the original transform data of those objects? And how can you even tell which vertices belonged to which original objects? AND, how can you do this in a way that doesn’t take 10 hours to process or crash Maya?

The first question to ask is, how can you do this for one object? My initial plan was to define some vectors with a few vertex locations of a ‘zeroed-out’ object, then compare those vectors to that of a transformed object. Even then, I wasn’t sure how to convert the results into a usable transform to apply to a locator. I could align the objects along ONE arbitrary vector, but struggled to complete the second rotation to make the alignment complete.

After a couple weeks dead-ending with vectors and quaternions, I finally discovered an example problem to use in the original course textbook from over 10 years ago.

The solution? Solve this equation for x: Ox = N, where O is the original vertex locations for one object, x is the transform matrix that’s been applied to them, and N is the new vertex locations. Just pick four consistent vert indices from each object, throw them into a column matrix, and solve the equation! The answer is the final transform matrix that was applied to the original objects, respectively. Apply that transform to an instance locator, and voila!

Now just do that a million times, and you’re done!

🙂