2014 Review and 2015 Preview

Review 2014

January 2014 - March 2014
Finished the rest of applications for Ph.D programs in US
I was called for a Skype interview of one university, however, I could not received good results with financial aid.

May 2014
Submitted the capstone project report.
Graduate from the University of Maryland.
Though I was hoping and going to keep working at START after graduation, I was suspended because my work authorization card had not arrived. For the time I could not work, I went over some MOOCs and watched the World Cup games.

July 2014
Returned to work at START.

August 2014
Left to Japan to take an entrance exam at the University of Tokyo. Back to Japan for the first time in two years.

September 2014
Trip to France with my mother.

October 2014
Started PhD program in the University of Tokyo. I have been mainly focusing on learning Machine Learning.

Objectives in 2014
Graduate from the University of Maryland <- done
Publish a paper about the human trafficking project <- could not finish due to my leave & difficulties to work off-site
Enter a Ph.D program / get a full-time job <- entered a PhD program.

2015 Preview
January 2015 to March 2015
Call for DC2 candidate will be out.

February to March 2015
Deadlines for popular international conferences in Computer Science. It would be great if I could make it to one of them.

July 2015
Progress report and presentation to graduate committee.

Objectives in 2015
Apply to DC2 or any similar financial aids.
Publish a paper or give a presentation at an international conferences.

How to allow ArcGIS to build a large transportation network dataset

Sometimes when building a large network dataset (in my case 100M ~ edges) in ArcGIS, "Build network dataset" tool returns "read/write error". I at first thought it was because the feature class source was somehow corrupt, but it turned out to be insufficient temporary directory space.

Problem: ArcMap or ArcCatalog returns “read/write error” while building a transportation network dataset.
Cause: insufficient temporary directory space
Solution: Secure enough free space in TEMP directory.

A new window opens and displays the contents of the System Temporary Directory. When building a very large network dataset (for example, all of the streets in France), hundreds of gigabytes of temporary space may be required to process a task of this size. If there is not enough disk space in the current TEMP directory to perform the task, it is necessary to change the TEMP directory to a location that has sufficient memory. The following steps show how to make this change.

1. Right-click Computer and click Properties.
2. Click Advanced system settings on the left panel to display the System Properties dialog box, and click the Advanced tab.
3. Click the Environment Variables button.
4. In the Environment Variables dialog box, under User variables for , both TMP and TEMP may listed. This is the common location; each different login account is assigned its own temporary location.
5. Click the Edit button. In the Edit User Variable dialog box, under Variable value, specify the path to the directory that Windows and many other programs uses for temporary files. Change this to a location that has sufficient disk space to process the network dataset. 
6. Repeat this process for both the TMP and TEMP directories.
7. Restart Windows to begin using the new value for temporary files. Restart any running programs for the new value to take effect.

http://support.esri.com/de/knowledgebase/techarticles/detail/40743

Another lesson to keep free space in the boot drive.

Making off-road transportation network using ArcGIS

Objective:

To create a off-road network (walking, off-road vehicle) from scratch and incorporate into traditional transportation network.

In order to model off-road transportation methods such as walking and driving off-road vehicle, the network dataset consists of traditional road network made by Navteq, horizontal, vertical, and diagonal line segment covering the whole extent.

Data:

Raster data for weighting (such as DEM, land cover)

Procedure:

Create "Fishnet" polygon to cover the target extent
Run Polygon to Raster with desired resolution
Run Raster to Point to create nodes for the network
Run Extract Values to Point to store raster values to points
Create a table containing pairs of coordinates to be connected

import arcpy
arcpy.env.overwriteOutput = 1 # enable overwriting

try:
    # load the point featrure class
    arcpy.env.workspace = arcpy.GetParameterAsText(0)
    #fc = "C:/Users/koitaroh/Docments/ArcGIS/Default.gdb/WeightPoint_Test"
    fc = arcpy.GetParameterAsText(1)
    outtable_path = arcpy.GetParameterAsText(2)
    
    class_field = "WeightPoint"
    fields = ["OBJECTID", "grid_code", "POINT_X", "POINT_Y"]
    xcoord = 0.0
    ycoord = 0.0
    # print("INITIALIZED")

    # create a insert cursor
    cursor2 = arcpy.da.InsertCursor(outtable_path, ("weight", "origin_x", "origin_y", "destination_x", "destination_y"))
    # create a search cursor
    # select * --- where x between a+10 and a-10m AND y between b+10m and b-10 (or 11m?)
    with arcpy.da.SearchCursor(fc, fields) as cursor1:
        for row in cursor1:
            weight = row[1]
            xcoord = row[2]
            ycoord = row[3]
            cursor2.insertRow((weight, xcoord+10, ycoord, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord, ycoord+10, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord-10, ycoord, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord, ycoord-10, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord+10, ycoord-10, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord+10, ycoord+10, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord-10, ycoord+10, xcoord, ycoord))
            cursor2.insertRow((weight, xcoord-10, ycoord-10, xcoord, ycoord))
            #rows = arcpy.da.SearchCursor(fc, "POINT_X" between x+10 and x-10 AND "POINT_Y" between y+10 and y-10 )
        del cursor1
        del cursor2
    # if the point is not same to the original
    

    # add row to a table

    # export as a table
    
except:
    print arcpy.GetMessages(2)

https://github.com/koitaroh/Workspace/blob/master/CreateIllicitGrid.py
Run XY to Line to create polyline feature

Join the weight attributes in the table to the polyline feature
Add and calculate attributes for length, and speed
Create a new network dataset and build it
Test the performance of the network dataset

Test walking network in Yuma County

Walking network prototype (red: shortest path, blue: most efficient path)

2013 Review and 2014 Preview

Review 2013

March 2013
ASPRS Annual Conference
April 2013

May 2013
Started an internship at START

November 2013
Got a part-time job at START
December 2013
Submitted applications for Ph.D. programs

Preview 2014
January 2014
Finish the rest of applications for Ph.D programs

March 2014
Get admission results

May 2014
Submit the capstone project report
Graduate from the University of Maryland

September 2014
Enter a Ph.D program

Objectives in 2014
Graduate from the University of Maryland
Publish a paper about the human trafficking project
Enter a Ph.D program / get a full-time job