Use .NET program to renew GSA crawling header

Now, I gonna guide you how to submitting form datas to GSA by using HTTP header as example.

In my previous article [Using .NET HttpWebRequest to pass through GSA login form], I talked about how to use .NET HttpWebRequest object to pass through GSA(Google Search Appliance) authentication framework. Now, I gonna guide you how to submitting form datas to GSA. (I will use HTTP header setting page under GSA admin console as example.)

Imagine that you have a system which only allows authenticated user and you want to configure GSA to crawl into it. That system does not integrate with Microsoft Active Directory server and has its own authentication framework, so we plan to get a cookie for GSA from that system. However, it is a ASP.NET web application running .NET 2.0 run time and does not provide indefinite duration cookie, so any cookie that we assign to GSA will be expired after a period of time. Although we could manually renew GSA cookie but it is much preferable that we could develop a program doing it for us. So here is the solution.

First, you need to follow the procedure that I introduced in [Using .NET HttpWebRequest to pass through GSA login form] to login GSA admin console page. Then, because we will submitting our data to HTTP Header setting page through HTTP POST, so we need to build the data string first. There has two input columns on HTTP Header page of Google Admin Console, one is "user agent name" and the other is "HTTP header". Please reference the full string from below:

string strContent = "actionType=httpHeaders&userAgent=[user agent name]&addHTTPHdrs=" + Server.UrlEncode("[HTTP header content]") + "&httpHeadersSave=+%E6%9B%B4%E6%96%B0%E6%A8%99%E9%A1%8C%E8%A8%AD%E5%AE%9A";

Please pay attention on a special key value of "HttpHeadersSave". You need to add it into full string or form submit won't be successful. Also, the HTTP header may contain special character so we use UrlEncode method of Server object to encode it. Once we have the full data string, we need to tranfer it to byte array:

ASCIIEncoding asciienc = new ASCIIEncoding();
byte[] byteContent = asciienc.GetBytes( strContent );

Finally, we will use HttpWebRequest object again to send out the request. The objects "req", "cc" and "s" below are HttpWebRequest, CookieContainer and Stream classes we created when logon GSA admin console. (Please reference the article [Using .NET HttpWebRequest to pass through GSA login form] for the details.)

req = HttpWebRequest.Create("http://[GSA URL]/EnterpriseController") as HttpWebRequest;
req.CookieContainer = cc;
req.Referer = "http://[GSA URL]/EnterpriseController?actionType=httpHeaders";
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
req.ContentLength = byteContent.Length;

s = req.GetRequestStream();
s.Write(byteContent, 0, byteContent.Length);
s.Flush();
s.Close();

That's all! You could based on the codes above to develop programs submitting data to all kind of GSA admin console pages.

[re-post this article due to database crash.]