Using the SharePoint 2010 Object Model to import Twitter Data

As a continuation from my first post about my talk at SharePoint Saturday Atlanta 2010, this part will focus on using the .Net Client Object Model to import data into SharePoint 2010.  This is a great way to avoid the BCS/Enterprise features if you have just a Foundation site. 

Here is a high level of things about the Client Object model:

• Provided for Silverlight, ECMA/Javascript, and .Net
• Lightweight server version (limited objects/less property objects within a Site Collection and down)
• Built into SharePoint Foundation, so available for all of the WSS users out there.
• Develop solutions off of the server, both in Visual Studio, and while running, the solutions do not need to be on the server
• Following point #3, no deployment.
• Connect to the Site via the URL
• Batch queries, and limited included properties means less chatter and less data transfer on requests.
• Manage installed Site Collection and Site features
• Manage most Site Collection and Site properties
• Manage web parts and pages
• Manage Site Collection and Site Security

The list goes on…

Now there are some limitations too:

• Can only work within a Site Collection
• Cannot create Web Applications or Site Collections
• Cannot interact with SharePoint Service Applications
• Cannot install or deploy solutions
• Cannot install features

For the SharePoint Saturday presentation, I focused on the .Net Client Object Model specifically. 

You can find the .Net Client Object model dlls in the 14 hive ISAPI folder (\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\ISAPI)

Microsoft.SharePoint.Client.dll
Microsoft.SharePoint.Client.Runtime.dll

Drop them into your VS project as references and add the using statement for the namespace “Microsoft.SharePoint.Client” and you are ready to code.

Pass in the Url to the ClientContext constructor to setup your object.  As seen above, I am going to use the Web (Just like a little brother of SPWeb).  Something to note, you cannot interact with the properties of your objects until you can called Load from the Context, and also ExecuteQuery.  “Load” simply adds the call to fill the object to the batch, then ExecuteQuery actually fills out the object details.

Next, I am going to test the list to check and make sure it was not empty. 

As long as I have the list, I am going to query it for items.  When working in the Client Object model, it is better to trim down the items you return with a CAML query (since it filters server side, and transfers less data), but in my case, I want all of the list items back.  There is a CreateAllItemsQuery() method, that does just that for me. 

Then just pass your query into your list… and call Context.Load() to batch up the request.  Here I have restricted the fields I am going to return from the list to save data in the items being returned.  Since I am only going to check the TweetId value of items before I insert new Tweet data into the list, I have only selected TweetId and ID.

I go ahead and call ExecuteQuery() to fill my listItems collection.  Next, I am going to drop the list items returned into a List object of the (SP)ListItem.  Since I plan to create new items, I new up a ListItemCreationInformation object.  This object, along with a handfull of others like it, allow for creation of new Webs, Lists, ListItems, Content Types, Fields, etc…  Finally, I am going to keep a count of new items I am adding, just to show for a sanity check and something to show during the demo.

Finally, I am going to run through the items I have, check to see if I can find the TweetId, and if not, add the Tweet to the List.

I add some values to the fields that exist on the list, and call listItem.Update().  This method does not commit the change immediately, rather batches up the inserts/changes until we call ExecuteQuery() again.

Execute the push back to SP, and comment to the console, just to let me know how many things went in, from the items I got from Twitter.  This whole class is wrapped into a single DLL for my purpose, and I call it from a console application that can run on a Windows scheduled job, takes parameters for SiteUrl, Twitter Search Term, etc.  This was also used in a Windows forms application to show the insert of data.

Finally, when you start working with this off the server, and happen to not be on the domain… well what happens?

Luckily, the other end of the wire is basically a WCF endpoint, so you only need to work with it like you have used web services in the past.

After you create the context object, set the ClientAuthenticationMode to Anonymous, Default (windows), or Forms.  And set the credentials to our old friend the NetworkCredential. (This was my case for a windows domain.)

Compile everything up, make sure to drop everything into try-catch blocks, and you now have Twitter data in your list… if you have not created your list yet, you can create a list from the custom template, just add the columns listed in the code above, and it should work just fine.  Alternatively, you could create the list using the Client Object model as well!  Just check for the list, if not there, create it in code, then connect to it.

Another alternative is post #3 which will come out tomorrow or Friday, in which I cover the last part of my talk, BCS .Net connector for the Twitter data.

Reading Twitter Search Data in .Net

For the recent SharePoint Saturday Atlanta event, I spoke about consuming Twitter data in .Net to push into SharePoint 2010 via the .Net Client Object Model.  Twitter search has evolved to use a REST web service call which produces either JSON or ATOM.  Since I had not yet used JSON anywhere, I decided to take the bullet and learn it, or rather at least understand it.

JSON is short for JavaScript Object Notation, all the information you might need to know can be found at http://json.org/.

Basically it is lightweight and constrains to a standard, and is being used even by the Client Object model itself.

The Twitter search term we used was #SPTwitter.  We had colleagues of ours post items with the hash tag and turn on geolocation.  Twitter Search API documentation can be found at http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search.  When requesting the term, I used q for the query and rpp (requests per page) set to 100 to return a max of 100 results.  I could use page to get more results, but 100 was more than enough to show on the Bing Maps/Silverlight example my co-worker was giving in his talk.

Code? Yep, here is what I used to query the REST service:

Now we have the JSON response in a string, and we can parse it into our .Net object.  Wait, we need a .Net object AND something to de-serialize the JSON.

There are quite a few options that we could use and after working with a few of them I decided on JSON.Net.  It was pretty straight forward, and is still being maintained.

JSON.Net - http://json.codeplex.com/

So I created a Tweet Class and a Container TweetCollection Class.  The search results are returned in a collection, so we will have to also add some attributes to that class.

Notice the JsonProperty Attribute for “results”.  This matches the collection of return results.  Here is a sample:

From this example, we can see that the results collection will have individual Tweet/Status data.

So I started with a simple class, Tweet.  I added the Profile Image Url and from_user fields to test it out.

Now that we have a simple class, and a collection for it, time to test the de-serialization.

The code above fills our collection and objects with real Twitter data!  Not so hard, well, at least someone wrote the parser for us!

Next step was simple enough, just fill the rest of our object with the other data from Twitter.  What about the GeoLocation data?  It is stored in a format like the following: "geo":{"coordinates":[33.9154,-84.3518],"type":"Point"}… but can also be listed as null (see example data above).  So the object geo, contains an array (coordinates) of two doubles and a key value pair for the “type”.

This turned out to be a little more complex than I thought… I followed the same pattern, geo, contained coordiates, and the type.  So I created a Geo class, with coordinates and the type.

continued…

 

One thing to notice is that the type of coordinates did not serialize correctly.  This type being null sometimes also provided some problems.  I did a little research, and determined that a JsonConverter should work for what we needed.  I created the following, using the JsonConverter abstract class, extracting the methods… ReadJson, WriteJson (we are not serializing it back into JSON, so I left this empty, and finally CanConvert.

Wrap that into a console or Windows app, and check your console!

Check back tomorrow for part 2, writing to SP using the Object Model.

SharePoint 2010 and Office 2010 RTM

I was at the SharePoint Saturday Speakers' dinner tonight, and Dan Attis (MVP) walked up and said "Did you hear, RTM was today... and so I googled binged it, http://arstechnica.com/microsoft/news/2010/04/office-2010-sharepoint-2010-go-rtm.ars

Cannot wait to see what has actually been fixed, and what is still broken.  I have had a couple conversations with Doug Ware about the first version not being up to par, but we will have to see if they had enough feedback to get things fixed.

I am speaking tomorrow at SPSATL, .Net Client Object Model, Twitter, and BCS Connector.  Hope to get a recording of the demo (not live) uploaded a little after the presentation, and several posts coming along with the material.

SharePoint Saturday Atlanta 2010 – Thoughts

I was prepping my speech and slides with my co-workers and realized, that I will have to extend my speech.  My co-worker is speaking on Silverlight and Bing Maps integrated with SharePoint 2010, and relies on the Silverlight client object model.  The main focus of my presentation was the .Net client object model to import twitter data with geolocations into SharePoint 2010.  Since he is speaking before me, I will add a BCS .net connector to the mix.  Will have to wait and see how much fun this is going to be.  More to follow.

Visual Studio 2010 released!

Just got an email that it is officially released! http://mobile.microsoft.com/visualstudio/en-us/Default.mspx

make sure to register for the upcoming events in or near your city. Check it out - https://microsoft.crgevents.com/Register2010/Content/Event_Selection.aspx