The Dell Business Intelligence Project Using USPTO Data: Episode 5

The Dell Business Intelligence Project Using USPTO Data: Episode 5

Author: Robert Pound,RogerLopez,UmeshSunnapu

Links:Enterprise Solutions, Business Intelligence and Appliance Solutions

Episode 5 | Dell Boomi HTTP to CSV

This episode combines the previous processes to extract and map an XML file from the USPTO Web site to a CSV file on a local disk.

The USPTO project uses Dell Boomi, Dell Quickstart Data Warehouse Appliance and Toad products to analyze publicly available data. For more information about the goal and scope of the project, as well as a breakdown of the episodes, go to Episode 1.

Dell Boomi HTTP to CSV process

In Episode 3, we created a Dell Boomi process to extract data from USPTO. In Episode 4, we highlighted mapping from XML to CSV. Now, we combine these processes to connect to the HTTP file and translate it directly to CSV. The following sections outline this episode:

• Combine processes
  •  Copy First Project
  • Add map from XML to CSV project
•  Modify XML profile
•  Format malformed XML
  • Comment headers
  • Add root tag
• Run process
  • Modify Atom

Combining processes                                                 

Most components used in this episode are the same components from the first project. Begin by copying the first process. Then add in the map components from the second process. Adding the map components loads the data to CSV. These steps bring us one step closer to loading the data into the database.

To copy the previous projects:

1.     In the Component Explorer, select the blue triangle next to First project and select Copy.

                                                 Figure 1.      Use Component Explorer to copy previous projects.

2.    In the Copy Component window, select the same folder used in the previous project (in this example Dell-REP), and clear the Copy component dependents? check box.

                                                                              Figure 2.      Selecting Dell-Rep folder.

3.    Click OK.

4.    In Component Explorer, double-click the process named First Project 2.

5.    Rename the process to HTTP to CSV.

6.    Click Save.

Adding a map

To add a map:

1.    Drag and drop a map shape to the newly created process.

2.    Click the  icon to view all components.

3.    Select the XML to CSV map created in the previous process.

                                                                        Figure 3.      Selecting the XML to CSV map.

4.    Click OK.

Modifying components

Modifying the filename

The process output is a CSV file. The first task is to modify the fileName shape.

1.    Highlight the fileName shape and click Configure.

2.    Change the name to fileName test.csv and then select the  icon under Parameters field.

3.    Change the filename to test.csv and select OK.

4.    Click OK.

Modifying the XML profile

Next, modify the map to parse all the patents in the file we downloaded from the site. For the Dell Boomi XML parser to extract data, the XML file must have a single parent element that encapsulates the data. To do this, add the element <root> to the file. After adding the <root> element to the file, modify the XML profile.

To modify the XML profile:

1.    Go to Component Explorer and double-click the USPTO_XML profile.

                                                                               Figure 4.      The USPTO_XML profile.

You must manipulate the nodes because there is no way to add parent nodes to the current top parent node.

2.    In the us-patent-grant element drop-down menu, select Add Child Element. This creates an element called element at the bottom of the profile.

                                                                     Figure 5.      Selecting the Add Child Element.

3.    Select element and rename it to us-patent-grant.

4.    Set Min Occurs to 1 and Max Occurs to unbounded. This change makes sure that you retrieve multiple patents.

                                                                                Figure 6.      Renaming the element.

5.    Select the topmost us-patent-grant element and rename it root.

6.    Set the Min Occurs and Max Occurs to 1. The profile should now look like Figure 6.

                                                                                      Figure 7.      The root element.

7.    Move all elements except root under the newly created us-patent-grant element by dragging the elements to the us-patent-grant element. Once the elements are added, a plus sign appears.

                                                         Figure 8.      After adding elements, a plus sign appears.

8.    Drop the element and it should now be added as a child element to us-patent-grant.

                         Figure 9.      Dropping the element adds it as a child element under us-patent-grant.

9.    Repeat this process for all remaining elements until the profile looks like Figure 9.

                                                         Figure 10.     Final view of the us-patent-grant element.

10.  Click Save and Close.

Formatting malformed XML

To deal with the malformed USPTO XML, add two shapes:

• One shape to deal with the multiple DTD headers
• One shape to add to the <root>

Before creating shapes, adjust the process to make room.

To adjust our process to make room:

1.    Disconnect the arrow from the Unzip and fileName shapes.

2.    Connect the arrow from XML to CSV to filename.

Your process should look like Figure 10:

                                                              Figure 11.     Making room by adjusting our process.