REST Web Services calls with C#

It’s really easy to call REST based web services from C#.Net. Let’s see how to do it. We’ll be calling Yahoo Web Services as an example here.

Make REST Calls With C#

The .NET Framework provides classes for performing HTTP requests. This HOWTO describes how to perform both GET and POST requests.

Overview

The System.Net namespace contains the HttpWebRequest and HttpWebResponse classes which fetch data from web servers and HTTP based web services. Often you will also want to add a reference to System.Web which will give you access to the HttpUtility class that provides methods to HTML and URL encode and decode text strings.

Yahoo! Web Services return XML data. While some web services can also return the data in other formats, such as JSON and Serialized PHP, it is easiest to utilize XML since the .NET Framework has extensive support for reading and manipulating data in this format.

Simple GET Requests

The following example retrieves a web page and prints out the source.

C# GET SAMPLE 1


using System;
using System.IO;
using System.Net;
using System.Text;
// Create the web request
HttpWebRequest request = WebRequest.Create(“http://developer.yahoo.com/”) as HttpWebRequest;
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
Console.WriteLine(reader.ReadToEnd());
}

Simple POST Requests Some APIs require you to make POST requests. To accomplish this we change the request method and content type and then write the data into a stream that is sent with the request. C# POST SAMPLE 1

// We use the HttpUtility class from the System.Web namespace
using System.Web;
Uri address = new Uri("http://api.search.yahoo.com/ContentAnalysisService/V1/termExtraction");
// Create the web request
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
// Set type to POST
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// Create the data we want to send
string appId = "YahooDemo";
string context = "Italian sculptors and painters of the renaissance"
+ "favored the Virgin Mary for inspiration";
string query = "madonna";
StringBuilder data = new StringBuilder();
data.Append("appid=" + HttpUtility.UrlEncode(appId));
data.Append("&context=" + HttpUtility.UrlEncode(context));
data.Append("&query=" + HttpUtility.UrlEncode(query));
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data.ToString());
// Set the content length in the request headers
request.ContentLength = byteData.Length;
// Write data
using (Stream postStream = request.GetRequestStream())
{
    postStream.Write(byteData, 0, byteData.Length);
}
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
    // Get the response stream
    StreamReader reader = new StreamReader(response.GetResponseStream());
    // Console application output
    Console.WriteLine(reader.ReadToEnd());
}

HTTP Authenticated requests

The del.icio.us API requires you to make authenticated requests, passing your del.icio.us username and password using HTTP authentication. This is easily accomplished by adding an instance ofNetworkCredentials to the request.

C# HTTP AUTHENTICATION

// Create the web request
HttpWebRequest request
= WebRequest.Create("https://api.del.icio.us/v1/posts/recent") as HttpWebRequest;
// Add authentication to request
request.Credentials = new NetworkCredential("username", "password");
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
    // Get the response stream
    StreamReader reader = new StreamReader(response.GetResponseStream());
    // Console application output
    Console.WriteLine(reader.ReadToEnd());
}

Error Handling

Yahoo! offers many REST based web services but they don’t all use the same error handling. Some web services return status code 200 (OK) and a detailed error message in the returned XML data while others return a standard HTTP status code to indicate an error. Please read the documentation for the web services you are using to see what type of error response you should expect. Remember that HTTP Authentication is different from the Yahoo!Browser-Based Authentication.

Calling HttpRequest.GetResponse() will raise an exception if the server does not return the status code 200 (OK), the request times out or there is a network error. Redirects are, however, handled automatically.

Here is a more full featured sample method that prints the contents of a web page and has basic error handling for HTTP error codes.

C# GET SAMPLE 2

public static void PrintSource(Uri address)
{
    HttpWebRequest request;
    HttpWebResponse response = null;
    StreamReader reader;
    StringBuilder sbSource;
    if (address == null) { throw new ArgumentNullException("address"); }
    try
    {
        // Create and initialize the web request
        request = WebRequest.Create(address) as HttpWebRequest;
        request.UserAgent = ".NET Sample";
        request.KeepAlive = false;
        // Set timeout to 15 seconds
        request.Timeout = 15 * 1000;
        // Get response
        response = request.GetResponse() as HttpWebResponse;
        if (request.HaveResponse == true && response != null)
        {
            // Get the response stream
            reader = new StreamReader(response.GetResponseStream());
            // Read it into a StringBuilder
            sbSource = new StringBuilder(reader.ReadToEnd());
            // Console application output
            Console.WriteLine(sbSource.ToString());
        }
    }
    catch (WebException wex)
    {
        // This exception will be raised if the server didn't return 200 - OK
        // Try to retrieve more information about the network error
        if (wex.Response != null)
        {
            using (HttpWebResponse errorResponse = (HttpWebResponse)wex.Response)
            {
                Console.WriteLine(
                "The server returned '{0}' with the status code {1} ({2:d}).",
                errorResponse.StatusDescription, errorResponse.StatusCode,
                errorResponse.StatusCode);
            }
        }
    }
    finally
    {
        if (response != null) { response.Close(); }
    }
}

Further reading

Related information on the web.

XML with C#

There is excellent support for XML in Microsoft .Net framework, and there is a very good and easy article on it at Yahoo Developer Network.

I am pasting its content here for convenience. Enjoy 🙂

Using Returned XML with C#

Once you have retrieved data from a web service you will need to do something with it. This HOWTO describes the various built-in methods .NET provides to use XML returned by a web service.

Overview

The .NET Framework provides excellent support for XML. Combined with the data binding support of WinForms and ASP.NET applications you have an easy and powerful set of tools. ASP.NET 2.0 takes data binding another step further by providing the DataSource control which lets you declaratively provide data access to data-bound UI controls.

Returned Data to a String

The simplest way to view the returned data is to get the response stream and put it into a string. This is especially handy for debugging. The following code gets a web page and returns the contents as a string.

C# STRING SAMPLE

public class StringGet { public static string GetPageAsString(Uri address) { string result = “”; // Create the web request HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest; // Get response using (HttpWebResponse response = request.GetResponse() as HttpWebResponse) { // Get the response stream StreamReader reader = new StreamReader(response.GetResponseStream()); // Read the whole contents and return as a string result = reader.ReadToEnd(); } return result; } }

USING XMLREADER

XmlReader provides fast forward-only access to XML data. It also allows you to read data as simple-typed values rather than strings. XmlReader can load an XML document without having to use HttpRequest, though you won’t have the same amount of control over the request. If you use HttpRequest, you can just pass the stream returned by the GetResponseStream() method to XmlReader. Fast write-only functions are provided byXmlTextWriter.

With .NET 2.0 you should create XmlReader instances using the System.Xml.XmlReader.Create method. For the sake of compatibility and clarity the next sample uses the .NET 1.1 creation method.

C# XMLREADER SAMPLE

using System.Xml;
// Retrieve XML document
XmlTextReader reader = new XmlTextReader("http://xml.weather.yahoo.com/forecastrss?p=94704");
// Skip non-significant whitespace
reader.WhitespaceHandling = WhitespaceHandling.Significant;
// Read nodes one at a time
while (reader.Read())
{
	// Print out info on node
	Console.WriteLine("{0}: {1}", reader.NodeType.ToString(), reader.Name);
}

USING XMLDOCUMENT

XmlDocument gives more flexibility and is a good choice if you need to navigate or modify the data via the DOM. It also works as a source for the XslTransform class allowing you to perform XSL transformations.

C# XMLDOCUMENT SAMPLE

// Create a new XmlDocument
XmlDocument doc = new XmlDocument();
// Load data
doc.Load("http://xml.weather.yahoo.com/forecastrss?p=94704");
// Set up namespace manager for XPath
XmlNamespaceManager ns = new XmlNamespaceManager(doc.NameTable);
ns.AddNamespace("yweather", "http://xml.weather.yahoo.com/ns/rss/1.0");
// Get forecast with XPath
XmlNodeList nodes = doc.SelectNodes("/rss/channel/item/yweather:forecast", ns);
// You can also get elements based on their tag name and namespace,
// though this isn't recommended
//XmlNodeList nodes = doc.GetElementsByTagName("forecast",
//                          "http://xml.weather.yahoo.com/ns/rss/1.0");
foreach(XmlNode node in nodes)
{
	Console.WriteLine("{0}: {1}, {2}F - {3}F",
	node.Attributes["day"].InnerText,
	node.Attributes["text"].InnerText,
	node.Attributes["low"].InnerText,
	node.Attributes["high"].InnerText);
}

Using XPathNavigator/XPathDocument

XPathDocument provides fast, read-only access to the contents of an XML document using XPath. Its usage is similar to using XPath with XmlDocument.

C# XPATHDOCUMENT SAMPLE

using System.Xml.XPath;
// Create a new XmlDocument
XPathDocument doc = new XPathDocument("http://xml.weather.yahoo.com/forecastrss?p=94704");
// Create navigator
XPathNavigator navigator = doc.CreateNavigator();
// Set up namespace manager for XPath
XmlNamespaceManager ns = new XmlNamespaceManager(navigator.NameTable);
ns.AddNamespace("yweather", "http://xml.weather.yahoo.com/ns/rss/1.0");
// Get forecast with XPath
XPathNodeIterator nodes = navigator.Select("/rss/channel/item/yweather:forecast", ns);
while(nodes.MoveNext())
{
	XPathNavigator node = nodes.Current;
	Console.WriteLine("{0}: {1}, {2}F - {3}F",
	node.GetAttribute("day", ns.DefaultNamespace),
	node.GetAttribute("text", ns.DefaultNamespace),
	node.GetAttribute("low", ns.DefaultNamespace),
	node.GetAttribute("high", ns.DefaultNamespace));
}

Using a DataSet

Using a DataSet from the System.Data namespace lets you bind the returned data to controls and also access hierarchical data easily. A dataset can infer the structure automatically from XML, create corresponding tables and relationships between them and populate the tables just by calling ReadXml().

C# DATASET SAMPLE

using System.Data;
public void RunSample()
{
	// Create the web request
	HttpWebRequest request
	= WebRequest.Create("http://xml.weather.yahoo.com/forecastrss?p=94704") as HttpWebRequest;
	// Get response
	using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
	{
		// Load data into a dataset
		DataSet dsWeather = new DataSet();
		dsWeather.ReadXml(response.GetResponseStream());
		// Print dataset information
		PrintDataSet(dsWeather);
	}
}
public static void PrintDataSet(DataSet ds)
{
	// Print out all tables and their columns
	foreach (DataTable table in ds.Tables)
	{
		Console.WriteLine("TABLE '{0}'", table.TableName);
		Console.WriteLine("Total # of rows: {0}", table.Rows.Count);
		Console.WriteLine("---------------------------------------------------------------");
		foreach (DataColumn column in table.Columns)
		{
			Console.WriteLine("- {0} ({1})", column.ColumnName, column.DataType.ToString());
		}  // foreach column
		Console.WriteLine(System.Environment.NewLine);
	}  // foreach table
	// Print out table relations
	foreach (DataRelation relation in ds.Relations)
	{
		Console.WriteLine("RELATION: {0}", relation.RelationName);
		Console.WriteLine("---------------------------------------------------------------");
		Console.WriteLine("Parent: {0}", relation.ParentTable.TableName);
		Console.WriteLine("Child: {0}", relation.ChildTable.TableName);
		Console.WriteLine(System.Environment.NewLine);
	}  // foreach relation
}

Further reading

Related information on the web is listed below.

How to kill your time by hardcoding?

My colleague asked me to help him solve an issue he was facing. I sat with him and after spending more than half an hour I realized that someone has hardcoded the server name :@ while we were assuming that we have already moved to another server.

The main thing to realize here is that if we have spent few more minutes in making the server name configurable at first place then we could have saved hours that we spent later just to find out that server name was hard-coded. Not doing this results in frustration, waste of time, de-motivation, brings down the moral of the team, and finally kills productivity and you lose credibility.

Hidden Reasons for slipping project deadlines

There can be so many factors that affects the deadline and project can be delayed. There is an excellent article 10 ways to get a slipping project back on track on TechRepublic explaining the remedies. But in my opinion tips given by the author are where there are already processes defined and a proper organizational hierarchy is maintained, not just maintained but also everyone at its position has its role and responsibilities defined and he works according to his role. At least to some extent. But what when this is not the situation? Let me explain here by first commenting on the TechRepublic post.

Overtime was on the top in that post. Author has already accepted that this is an option when you are at the end of the project, and there are better strategies if you are in early stages of project. I agree with that. You cannot work overtime for a long period. Few overtimes are acceptable. I just wanted to mention this here.

Few other reasons of slipping the project deadline is setting the deadlines blindly, assuming ideal situation and not keeping the buffer, specially when there is a learning curve when you are working on something new.

Another problem is that if there are lots of senior guys and everyone is imposing his own decisions discarding other’s. Having senior guys is good but everyone should have separate responsibilities and no one should interfere with other. Discussing matters, coordination, and giving opinion is something else. Not having this leads to unstability.

Another major big problem is with the management when talking in terms of investment. Besides company’s plans few things are obvious where anyone can see that investment has been done at wrong place or investment is not done where it was supposed to be. Like hiring skilled people and paying them good. Finding cheap human resource does not always work. You should know when you have to invest in human resource and when not. And if you already have good human resource then appraise them before you loose a good resource and someone else steals it from you. Keep in mind as long an employee works for a company, he becomes more valuable asset since he has learned so many things and new employee will take time to learn those things. If you can understand this then you already know that time is money. And if employee leaves in the middle of a project, then obviously there is a big chance of slipping the deadline. You have to put extra resource, reschedule accordingly, assign resource etc etc.

Another one is related to human resource and team. One should have experts and should have responsibilities and roles defined. Don’t expect one to do everything from project management, to design and architecting, to learning, to implementation, to support. Project Manager is one guy, System Architect is another, Developer is another, Designer is another, Tester is another, Network or System Admin is another (sometimes two), and Support personal is one another. If you don’t have enough money to hire all these people then don’t jump into the big business, otherwise you will fail. Start with small and gradually build your team. This is a long term investment.

Besides human resource, other resources are also important, including facilities. Not providing sufficient resources to the team wastes time and at the end deadlines are slipped. Employees also get frustrated. Even a very small resource or facility has its effect. Not upgrading systems, network issues, not providing healthy environment, and even not having standards and policies are the reasons. Having flexibility is something else and not having proper company policies is something else. Not having these demotivates employees and one cannot give his 100% then.

One another important thing is forecasting and future planning. If you don’t have the plan and has approach like we’ll see when that will happen, then don’t be surprised if you fail to accomplish your goals.

When working on projects you should have dedicated resources, not the ones that come on and off. I am typically talking about human resources here. Since working on project and to complete it in time needs concentration and focus. There can be resources who come on and off but don’t assign them critical tasks. Use them for assistance only.

Processes! Just having separate development and testing teams doesn’t mean you are following processes. There are processes for within individual teams as well. If processes are not defined then this is also a reason for not meeting the deadlines.

Besides all this I also recommend reading Peter Steven’s blog post 10 Ways to Save a Slipping Project.

Threads in Perl without using thread module

Threads are not good in Perl. If you want to know why, you can read PerlMonks article here. But C/C++ threads are good 🙂 How can you leverage the benefit of C/C++ threads in Perl? Well, if you have a C/C++ library which is multi-threaded then you can write a Perl wrapper on C/C++ library. For writing a wrapper you can use Inline::C or Inline::CPP module, that provides a very gentle and easy way to wrap up your C/C++ calls.

What happens is that if you call a C/C++ function which uses threads from a Perl script/process, it will make your Perl process multi-threaded. You can see this by using -m option of ps if you are on Linux platform.

I was working on a Perl based project and I used a multi-threaded C/C++ library. I was curious how will it work. I ran the Perl process and when I see the process in ps with -m option, it was showing two processes, actually one process and second was the thread. When I was using a single threaded library it used to show only a single process in ps. I also tested the functionality which was supposed to use the multiple threads and it worked as expected.

I think this is a better way to use threads in Perl if you really need threads, instead of using thread module of Perl. If you are not desperate, then try to stick with single threaded model of Perl, or use Perl wrapper over C/C++ multi-threaded library.

How to setup PHP development environment in Eclipse?

As I am currently working on Joomla (http://www.joomla.org/) a CMS built in PHP, I came across a very nice tutorial about how to setup your development environment if you want to work on Joomla. Although the tutorial is specific to Joomla, PHP, and Eclipse, but in general it is a very good tutorial and one can learn how to setup a development environment for any project irrespective of technology used. Especially novice programmers will learn a lot from it.

Author first explained how to install and configure XAMPP, then configure PHP and XDebug to debug the PHP applications. Then author explained how to install Eclipse, setup your workspace, configure it for debugging. Then author created a test project and ran it in debug mode. Then in the end author explained how to install and configure subclipse (Eclipse SVN plug-in) and how to import a project from SVN repository.

This is a very nice tutorial in general and specifically for those who work in Eclipse and PHP. Wanna read it? go ahead: http://docs.joomla.org/Setting_up_your_workstation_for_Joomla!_development

Why code bad instead of clean code?

I was reading the book Clean Code – A Handbook of Agile Software Craftsmanship, Rober C. Martin Series. A section in chapter one caught my attention and remind me of code and reasons that I got for writing bad code and designing bad solution. Now I think this is a good argument against it which I am quoting here from the book.

“Have you ever waded through a mess so grave that it took weeks to do what should have taken hours? Have you seen what should have been a one-line change, made instead in hundreds of different modules? These symptoms are all too common. Why does this happen to code? Why does good code rot so quickly into bad code? We have lots of explanations for it. We complain that the requirements changed in ways that thwart the original design. We bemoan the schedules that were too tight to do things right.
We blather about stupid managers and intolerant customers and useless marketing types and telephone sanitizers. But the fault, dear Dilbert, is not in our stars, but in ourselves. We are unprofessional.
This may be a bitter pill to swallow. How could this mess be our fault? What about the requirements? What about the schedule? What about the stupid managers and the useless marketing types? Don’t they bear some of the blame?
No. The managers and marketers look to us for the information they need to make promises and commitments; and even when they don’t look to us, we should not be shy about telling them what we think. The users look to us to validate the way the requirements
will fit into the system. The project managers look to us to help work out the schedule. We are deeply complicit in the planning of the project and share a great deal of the responsibility for any failures; especially if those failures have to do with bad code! “But wait!” you say. “If I don’t do what my manager says, I’ll be fired.” Probably not. Most managers want the truth, even when they don’t act like it. Most managers want good code, even when they are obsessing about the schedule. They may defend the schedule and requirements with passion; but that’s their job. It’s your job to defend the code with equal passion.
To drive this point home, what if you were a doctor and had a patient who demanded that you stop all the silly and-washing in preparation for surgery because it was taking too much time?2 Clearly the patient is the boss; and yet the doctor should absolutely refuse to comply. Why? Because the doctor knows more than the patient about the risks of disease and infection. It would be unprofessional (never mind criminal) for the doctor to comply with the patient.
So too it is unprofessional for programmers to bend to the will of managers who don’t understand the risks of making messes.”

Breaking problem into code

Let us consider an example. If you have two boxes say BoxA and BoxB, and there are few balls of different colors, let’s say 5 balls (red, blue, green, yellow, and white) in BoxA, and you want to move one ball, say red ball from BoxA to BoxB. What will be the steps? If you can make a flow chart of it then its good, but if you find it difficult to make a flow chart of it then you need to do some hard work to become a programmer.

I asked this to someone and guess how he solved it. Here is his solution. Take out all balls from BoxA, then pick the red ball, put it in BoxB, and then put all remaining balls back in BoxA.

Fine! it worked. But why would you take out all balls if you can only take out red ball and move it to next box?

Let’s move this to programming side. In the above mentioned scenario we will have two database tables, boxes, and balls. boxes will have boxid as PK, name, and other details. Keep it simple so we don’t get lost in other details. balls table has ballid as PK, name, boxid as FK, and other details. boxes table has two records with 1 and BoxA as its boxid and name. balls table will have 5 records with id between 1 to 5 and red, blue, green, yellow, and white as their names, and all balls will have 1 in boxid field which is FK to boxes table.

Now if we adopt the first solution then it will run following queries (these are pseudo queries not actual SQL queries):

– select all balls where boxid=1 (so we can have a list of all balls before deleting them from database)
– delete from balls where boxid=1
– insert into balls values ballid=1, name=red ball, boxid=2
– insert into balls values
(ballid=2, name=blue ball, boxid=1)
(ballid=3, name=green ball, boxid=1)
(ballid=4, name=yellow ball, boxid=1)
(ballid=5, name=white ball, boxid=1)

Or even worse if we use multiple insert queries. It will run from 4 to 7 queries.

Now let’s see what happens if we just perform operation on the red ball and don’t touch other balls.

– update balls set boxid=2 where ballid=1; /* ball with id 1 is the red ball */

Wow, just one query and we’re done! We saved 3 to 6 queries. Imagine this scenario for a high traffic website or any other application, we can improve the performance with a big difference by just implementing correct logic to solve a problem.