Node Dangles

Field Calculator: ArcPy Date to Decimal Function

Map Ranter — Fri, 07 Oct 2011 03:08:35 +0000

One of the standards in our databases is to store dates as 8-digit integer values in the format of yyyymmdd. This requires us to occasionally convert values from date fields into this format.

We can do this in the ArcMap Field Calculator using this arcpy function:

def datetodouble(inNum):
     splitList = str(inNum).split("/")
     return  splitList [2]  +("0"+ splitList [0])[-2:]  +("0"+ splitList [1])[-2:]

Data Extration Failed — Reverse Proxy Time-out

Map Ranter — Fri, 19 Aug 2011 01:55:47 +0000

Testing one of our geodata services, we discovered that it allowed us to extract a portion of our feature class but when we tried to extract the entire data set, we received this Data Extraction error: Data extraction failed. Proxy or Gateway Server did not allow the URL. Check with your LAN administrator that Proxy or Gateway server is configured to allow the URL.

The fact that I was able to extract a portion of the data and I could see the entire geodatabase get made and zipped led me to believe it was more of time-out issue.

Reading through this thread at ArcForum led to some good information. But Thomas’ comment that he was using “IIS7 for my reverse proxy server” and had to change one more setting led me to the solution. In Server Manager, the default Proxy Time-Out is set at 30 seconds by default. I bumped that up, 60 seconds shown below but I ended up going to 300 seconds and the problem was resolved.

Using arcpy in ArcGIS 10 Field Calculater to Create a Unique ID

Map Ranter — Fri, 29 Jul 2011 02:08:13 +0000

One of the common functions I have to do is assign each record in a feature class with a unique identifier–normally just a sequential number from 1 to N. In ArcView 3.x, the formula was simply “rec + 1″ if I wanted to start with the number 1.

In ArcGIS, the process got a little more complex–you had to write a little VBA in Field Calculator as described by ESRI.

While this option still exists in ArcGIS 10, I believe it will disappear when 10.1 comes out and VBA support is completely eliminated. But it is doable using Python which will continue to be supported.

Googling around, I did not find an exact answer but Dave Verbyla, Professor of GIS/Remote Sensing at the University of Alaska has a posted some samples that served as a good starting point.

In the Pre-Logic Script Code box, I declare a variable (counter) and a function. Then in the formula, I call the function.

counter = 0
def uniqueID():
  global counter
  counter += 1
  return counter

While composing this post, I actually wanted a concatenated value; “OP” plus an 8 character numeric sequential number starting at OP00000001 so the actual code is shown below:

ArcGIS Add-In Custom Mouse Cursor

Map Ranter — Wed, 20 Jul 2011 02:47:03 +0000

I was working on a project and wanted my own custom mouse cursor and did not easily find a way to make your own in ESRI’s instructions. But, once you know how to do it, it is pretty easy. In Visual Studio, Add a New Item:

Add a Cursor File:

You can edit your cursor with the editor program in Visual Studio. Once you satisfied with how it looks, make sure that the Build Action on the cursor is “Embedded Resource”.

Then you can set your cursor with two lines of code. Not that my cursor is in my QDI.QdiAddIn Namespace:

Dim pCursorStream As System.IO.Stream = Me.GetType.Assembly.GetManifestResourceStream("QDI.QdiAddIn.NewCursor.cur")
MyBase.Cursor = New System.Windows.Forms.Cursor(pCursorStream)

Debugging a Python Scheduled Task

Map Ranter — Tue, 12 Jul 2011 00:11:13 +0000

I have been working on a python script that I want (NEED) to run as a scheduled task on a remote machine. I got to the point that the script did exactly what I needed when I was interactively running it in a Windows session but had problems when running it as a scheduled task. The debugging process was cumbersome–make a change, schedule a task to run it, log out of the machine, and wait. The log back in and repeat the process.

That got old.

So I wrote a script (tester.py) that calls any other python scripts in the same directory that (1) start with “test_” and (2) there is not a corresponding file with the same base name and “.start” extension. It would launch “test_BaBing.py” as long as there is not a “test_BaBing.start” in the same directory. Tester.py continued to run, looping every 60 seconds, until tester.stop exists.

This made the process easier because I could work on my local machine, editing the problematic script, saving changes and within 60 seconds it would be launched on the remote machine. I could view the results, make additional edits, delete the .start file and it would launch again within 60 seconds.

Within a couple minutes I was able to determine the problem (path related) and fix it.

Happy programmer.

I would recommend using this only while debugging a script–routinely running it could be a security risk since someone could copy a destructive python script into the directory and this would run it.

Download: tester.py

import sys, string, os
import glob
import datetime, shutil
import time, inspect
import getpass

totalstarttime = datetime.datetime.now()

dateString = datetime.date.today().strftime("%Y%m%d_")+datetime.datetime.now().strftime("%H%M%S") #datetime.date.today().strftime("%Y%m%d")
debugfile = inspect.getfile(inspect.currentframe()).replace(".py","_"+dateString+"_Debug.txt")
stopfile = inspect.getfile(inspect.currentframe()).replace(".py",".stop")
newdebugfile = False

codeDir = os.path.dirname(inspect.getfile(inspect.currentframe())).replace("\\","/")

def printit(inText):
    global newdebugfile

    print inText

    if os.path.exists(debugfile):
        if (newdebugfile == False):
            tmpfile = open(debugfile,"w")
            newdebugfile = True
        else:
            tmpfile = open(debugfile,"a")
    else:
        tmpfile = open(debugfile,"w")

    tmpfile.write(inText)
    tmpfile.write("\n")
    tmpfile.close()
    newdebugfile = True

stopFileExists = False
printit("Code Directory: "+codeDir)
printit("Starting at: "+datetime.date.today().strftime("%Y-%m-%d_")+datetime.datetime.now().strftime("%H:%M:%S"))
printit("Stopfile : "+stopfile+"/n")
while (stopFileExists == False):
    for iFile in glob.glob(codeDir+"/test_*.py"):

        thisStartfile = iFile.replace(".py",".start")

        if not (os.path.exists(thisStartfile)):
            printit ("Launching: "+iFile)
            iTmpfile = open(thisStartfile,"w")
            iTmpfile.write("started")
            iTmpfile.close()
            os.system("Start "+iFile)

    if (os.path.exists(stopfile)):
        stopFileExists = True
    else:
        time.sleep(60)

    printit("\nEnd of Loop: "+datetime.date.today().strftime("%Y-%m-%d_")+datetime.datetime.now().strftime("%H:%M:%S")+"\n")    

printit("Done!")

Checking to see if a Field Index Exists Using Arcpy (ArGIS 10.0) redux

Map Ranter — Tue, 05 Jul 2011 18:53:34 +0000

I’ve previously posted python code to check if a field index exists for both ArcGIs 9.3 and ArcGIS 10.0.

Recently I have been working on a process that was using this code but it was not working because it looks for an index with a specific name. It was not working in this case because the name of the indexes was getting incremented as they were being created. For example, I was building an index on the table C5ST, field RelateId ([C5IX].[Relateid]) named I_C5IX_RelateId. That worked fine until we switched our process so now we keep multiple versions of some tables, each with a date-based suffix.

We now have tables name C5St_20110625 and C5St_20110626–the Index-name scheme, however was still creating I_C5IX_RelateId and it worked great on the first one. But when it created the second one, even on a different table, it was automatically name I_C5IX_RelateId_2 even though the name I_C5IX_RelateId was used when trying to create the index.

Before generating relates, our code checks to see if the key fields are indexed, and if they are not, builds an index. Because of the naming situation, multiple, duplicate indexes were being created. Probably not too harmful but it is a little messy.

So I re-wrote the code so that you pass the function the table name and field name that you want to check and it checks to see if there is an index existing for that field and return a Boolean. The one little wrinkle I put in is to account for indexes that span multiple fields–the ” if (iIndex.fields[0].Name.upper() == fieldname.upper()):” statement is checking the index to see if it is on a single field or multiple fields.

def fieldHasIndex(tablename,fieldname):
if not arcpy.Exists(tablename):
return False

tabledescription = arcpy.Describe(tablename)

for iIndex in tabledescription.indexes:
if (len(iIndex.fields)==1):
if (iIndex.fields[0].Name.upper() == fieldname.upper()):
return True

return False

Feature classes and Tables with names starting with “nd_”.

Map Ranter — Mon, 23 May 2011 21:55:24 +0000

Random luck me to discovering a bug related to feature classes whose names start with “nd_”. It appears that you are allowed to create feature classes starting with “nd_” but ArcCatalog will not display them. Further research shows this behavior also occurs for table and for ArcSDE (PostGres) geodatabases, personal geodatabase, and file geodatabases–I am using ArcCatalog 10.0.

I first noticed something odd was occurring while importing a series of shapefiles into a geodatabases. After importing 15 shapefiles, I only had thirteen feature classes despite receiving no errors during the process. The two shapefiles that failed to import were named ND_oil_and_gas.shp and ND_Bendix_Study.shp. Subsequent attempts to import them individually returned an error “Invalid Target Name”.

I discovered in pgAdmin III (Postgres SDE Geodatabase) that the table existed and there was an entry in sde.sde_layers for the feature class but ArcCatalog refused to show it.

I used some un-supported methods to try to resolve the problem and despite some sweating, I failed to find a way to get ArcCatalog to display these feature classes. I did, however, at least found a way to delete them–arcpy can detect that the feature classes exists so it is able to delete them.

At least by deleting them, I can prevent leaving “invisible” feature classes from hanging out in my geodatabase.

I suspect the problems stems from how ESRI has implemented the Network dataset table-naming structure –dirty areas are stored in tables named nd__dirtyareas and nd__dirtyobjects. Possibly the developer working on the ArcCatalog GUI ended up suppressing showing feature classes and tables whose names start with “nd_”.

And, just for posterity’s sake, here is a python code snippet listing the feature classes in a workspace:

import arcpy

arcpy.env.workspace = “c:/temp/_nd/F.gdb”

print arcpy.env.workspace
for fc in arcpy.ListFeatureClasses():
print fc

print “Done!”

Renaming Raster Dataset and arcpy.Exists()

Map Ranter — Tue, 03 May 2011 15:33:48 +0000

Discovered something today. I was working on an arcpy script that copies a raster dataset from a file geodatabase into a Postgres SDE geodatabase and then does some boring routine tasks–building stats, creating a mosaic dataset, adding the raster to the mosaic dataset and making a couple referenced mosaic datasets.

It sometimes has trouble with the initial step of uploading the raster because of the sheer size of if (1m elevation raster for counties) and it failed today on one. It failed today so I used the ArcCatalog GUI to copy the raster and renamed it.

I then proceeded to run launch my script. Before each step, I use arcpy.Exists() extensively to check to see if various items exist before I attempt to create them. It was continuously reporting that my raster set did not exist even though I could see it in ArcCatalog.

Finally, I realized that I needed to close ArcCatalog before arcpy recognized the fact I had renamed something. To note, I was running arcpy from a separate PythonWin window, not from the ArcCatalog session I had renamed the raster dataset with.

Once I closed ArcCatalog, arcpy recognized the renaming and life was good.

I’m also suspicious now about a problem I often have running statistics on my rasters. The ArcTool reports no errors when I create them but for some reason the raster does not show that it has statistics afterwards. I normally have multiple ArcApplication sessions open and now suspect that perhaps this problem is due to sessions not letting go of the connection. Stay tuned for further developments on this.

Sorting a Coded-Value Domain Add-In (ArcGIS 10)

Map Ranter — Tue, 03 May 2011 09:33:01 +0000

I am working on an data-entry application to edit feature classes that contain several coded-value-domains. The problem with some of the domains, however, is that some entries have been added after the initial creation. So the first 25 entries are in alphabetical order and there are some stragglers at the end that are in the order they were appended.

This can be confusing for users–they go to select “Milli Vanilli” and look between “Madonna” and “Motley Crue” but can not find their favorite band there–they have to go to the end of the list to find their selection.

In the past, I have gone through the tedious process of exporting the domain to a table, sorting the table, removing the domain from the necessary field(s), deleting the domian, re-importing the table back in a new domain and finally re-applying the domain to the necessary field(s). Let’s just say I didn’t do this until someone asked a few times and I didn’t have anything more exciting–like a root canal–I could busy myself with.

But this new application contains more domains than any of other datasets so it was time to find a better solution. ESRI does have a Domain Sort Developer Sample. It, however, did not play nice with ArcGIS 10.

So I went ahead and update it from VB 6 to VB.Net/ArcObjects 10. I made an Add-In that can be installed by downloading the .esriaddin file and double-clicking on it. The source code is also available.

This will add an ArcCatalog Toolbar that can be added by going to Customize-Toolbars-Domain Sorter Toolbar.

This will add a toolbar with one button. The button enables whenever you select a geodatabase with at least one coded-value domain.

This brings up a Windows form that lets you sort any domain by either the code or description, ascending or descending. Once you hit “OK” it re-sorts your domain.

The only problem I have had is that only the owner of a domain is allowed to edit it on an SDE geodatabase.

But other than that, the button allows you to easily keep your domains sorted.

http://edndoc.esri.com/arcobjects/9.2/CPP_VB6_VBA_VCPP_Doc/COM_Samples_Docs/Geodatabase/Schema_Creation_and_Management/Sort_a_domain/e826c5a8-9740-4f0b-86b6-d3b834735574.htm

Quick & Dirty arcpy: Batch Splitting Polylines to a Specific Length.

Map Ranter — Sun, 01 May 2011 11:01:50 +0000

For some odd reason, I wanted to split all the arcs in a polyline feature class to a specific length–if a specific feature was longer than the target length, it would become two or more separate polyline records.

Here is the bare-bones script that copies an existing feature class into a new feature class then processes each record, splitting it into multiple records if the polyline is longer than the user-specified tolerance. Some cautionary notes:

This is Quick & Dirty code–minimal error catching or documentation.
I basically tested this against one feature class (the one I wanted to split) once I got it to work, I quit.
There is some rounding error–features may be a tad bit off (a few ten-thousandths of a unit).
I did not test against multi-part features.
The tolerance is the native units of the data–if your data is in meters but you want to split the polylines every mile, enter 1,609.344.

I have included both a toolbox file (.tbx) and python script (.py). After loading the toolbox, you’ll have to change the Source of the script by right-clicking on it, selecting the Source tab, and then navigating to the .py file.

Here is the code for the Googlebots, but you are better off just downloading it.

import arcpy
import sys, math

def printit(inMessage):
    print inMessage
    arcpy.AddMessage(inMessage)

if len(sys.argv) > 1:
    inFC = sys.argv[1]
    outFC = sys.argv[2]
    alongDistin = sys.argv[3]
    alongDist = float(alongDistin)
else:
    inFC = "C:/temp/asdfasdf.mdb/jkl"
    OutDir = "C:/temp/asdfasdf.mdb"
    outFCName = "jkl2d"
    outFC = OutDir+"/"+outFCName
    alongDist = 1000

if (arcpy.Exists(inFC)):
    print(inFC+" does exist")
else:
    print("Cancelling, "+inFC+" does not exist")
    sys.exit(0)

def distPoint(p1, p2):
    calc1 = p1.X - p2.X
    calc2 = p1.Y - p2.Y

    return math.sqrt((calc1**2)+(calc2**2))

def midpoint(prevpoint,nextpoint,targetDist,totalDist):
    newX = prevpoint.X + ((nextpoint.X - prevpoint.X) * (targetDist/totalDist))
    newY = prevpoint.Y + ((nextpoint.Y - prevpoint.Y) * (targetDist/totalDist))
    return arcpy.Point(newX, newY)

def splitShape(feat,splitDist):
    # Count the number of points in the current multipart feature
    #
    partcount = feat.partCount
    partnum = 0
    # Enter while loop for each part in the feature (if a singlepart feature
    # this will occur only once)
    #
    lineArray = arcpy.Array()

    while partnum < partcount:
        # Print the part number
        #
        #print "Part " + str(partnum) + ":"
        part = feat.getPart(partnum)
        #print part.count

        totalDist = 0

        pnt = part.next()
        pntcount = 0

        prevpoint = None
        shapelist = []

        # Enter while loop for each vertex
        #
        while pnt:

            if not (prevpoint is None):
                thisDist = distPoint(prevpoint,pnt)
                maxAdditionalDist = splitDist - totalDist

                print thisDist, totalDist, maxAdditionalDist

                if (totalDist+thisDist)> splitDist:
                    while(totalDist+thisDist) > splitDist:
                        maxAdditionalDist = splitDist - totalDist
                        #print thisDist, totalDist, maxAdditionalDist
                        newpoint = midpoint(prevpoint,pnt,maxAdditionalDist,thisDist)
                        lineArray.add(newpoint)
                        shapelist.append(lineArray)

                        lineArray = arcpy.Array()
                        lineArray.add(newpoint)
                        prevpoint = newpoint
                        thisDist = distPoint(prevpoint,pnt)
                        totalDist = 0

                    lineArray.add(pnt)
                    totalDist+=thisDist
                else:
                    totalDist+=thisDist
                    lineArray.add(pnt)
                    #shapelist.append(lineArray)
            else:
                lineArray.add(pnt)
                totalDist = 0

            prevpoint = pnt                
            pntcount += 1

            pnt = part.next()

            # If pnt is null, either the part is finished or there is an
            #   interior ring
            #
            if not pnt:
                pnt = part.next()
                if pnt:
                    print "Interior Ring:"
        partnum += 1

    if (lineArray.count > 1):
        shapelist.append(lineArray)

    return shapelist

if arcpy.Exists(outFC):
    arcpy.Delete_management(outFC)

arcpy.Copy_management(inFC,outFC)

#origDesc = arcpy.Describe(inFC)
#sR = origDesc.spatialReference

#revDesc = arcpy.Describe(outFC)
#revDesc.ShapeFieldName

deleterows = arcpy.UpdateCursor(outFC)
for iDRow in deleterows:       
     deleterows.deleteRow(iDRow)

del iDRow
del deleterows

inputRows = arcpy.SearchCursor(inFC)
outputRows = arcpy.InsertCursor(outFC)
fields = arcpy.ListFields(inFC)

numRecords = int(arcpy.GetCount_management(inFC).getOutput(0))
OnePercentThreshold = numRecords // 100

printit(numRecords)

iCounter = 0
iCounter2 = 0

for iInRow in inputRows:
    inGeom = iInRow.shape
    iCounter+=1
    iCounter2+=1    
    if (iCounter2 > (OnePercentThreshold+0)):
        printit("Processing Record "+str(iCounter) + " of "+ str(numRecords))
        iCounter2=0

    if (inGeom.length > alongDist):
        shapeList = splitShape(iInRow.shape,alongDist)

        for itmp in shapeList:
            newRow = outputRows.newRow()
            for ifield in fields:
                if (ifield.editable):
                    newRow.setValue(ifield.name,iInRow.getValue(ifield.name))
            newRow.shape = itmp
            outputRows.insertRow(newRow)
    else:
        outputRows.insertRow(iInRow)

del inputRows
del outputRows

printit("Done!")