How do I login, navigate, and return data from a protected website, so far everything I'm doing isn't working
and while I've found many articles and other information about how to GET and POST using HttpWebRequest and Response, I find myself having a hard time getting things to work like I would expect them to work.
I've been playing with several ideas that I have found but so far, nothing is working... I'll post my code:
private void start_post()
{
string username = txtUser.Text;
string password = txtPassword.Text;
string strResponce;
byte[] buffer = Encoding.ASCII.GetBytes("username="+username+"&password="+password);
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create(txtLink.Text);
WebReq.Method = "POST";
//WebReq.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
WebReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
WebReq.Headers.Add("Translate", "F");
WebReq.AllowAutoRedirect = true;
WebReq.CookieContainer = cookieJar;
WebReq.KeepAlive = true;
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = buffer.Length;
Stream PostData = WebReq.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse WebResp = (HttpWebResponse)WebReq.GetResponse();
//txtResult.Text = WebResp.StatusCode.ToString() + WebResp.Server.ToString();
Stream answer = WebResp.GetResponseStream();
StreamReader _answer = new StreamReader(answer);
strResponce = _answer.ReadToEnd();
//txtResult.Text = txtResult.Text + _answer.ReadToEnd();
answer.Close();
_answer.Close();
foreach (Cookie cookie in WebResp.Cookies)
{
cookieJar.Add(new Cookie(cookie.Name.Trim(), cookie.Value.Trim(), cookie.Path, cookie.Domain));
txtResult.Text += cookie.Name.ToString() + Environment.NewLine + cookie.Value.ToString() + Environment.NewLine + cookie.Path.ToString() + Environment.NewLine + cookie.Domain.ToString();
}
if (strResponce.Contains("Log On Successful") || strResponce.Contains("already has a webseal session"))
{
MessageBox.Show("Login success");
foreach (Control cont in this.Controls)
{
cont.Visible = true;
}
}
else
{
MessageBox.Show("Login Failed.");
}
}
Here in code, I'm able to get all the way to the bottom, and still get a login failed when I navigate to http://www.comicearth.com (my own site, php and apache) I created a form and from that form, I type in the password and username. When it do this, it says failed, which is ok. I'm also using Fidder to watch to see what all is happening.
So from this, I know I'm doing something wrong from the code below.
However, when I navigate to another web application, I get the follow error on the line:
HttpWebResponse WebResp = (HttpWebResponse)WebReq.GetResponse();
"Content-Length or Chunked Encoding cannot be set for an operation that does not write data."
I'm trying to find what is wrong, and everything I have said that it's because of a 302 redirect...
So, looking in Fiddler I can see a huge difference between when I try to post the data and when I login via a web page. So I know I'm not doing enough, but I don't know where to look.
My goal is to build an application that is able to login to the website, and then via their search option pull out necessary data that currently we have our users do manually, if I can automate some of the tedious work, it would really help everyone out. However, I'm currently stuck on logging in, understanding cookies, etc... Also, the website uses frames, I don't know if that will be an issue, but I figured I'd post that info, just in case it is another hurdle I haven't come across yet.
Let me know if you need me to see more code, currently I'm using the httpwebrequest and httpwebresponse and I've read other informaiton about web client.
I've downloaded and have played around with htmlagilitypack but at this time not to sure I'm 100% good on how that all works as well.
If you know of any good articles, or other information that covers this topic more in depth or have anything that I can try, let me know.
Thanks so much for your time.
Update with new code, see my comment below as well: - Ok, I found out that because of the redirect I was getting that one error message: "Content-Length or Chunked Encoding etc..." and so I turned the allowAutoRedirect = false and now I look for the "location" tag, and redirect myself etc, which got rid of this message, however, I'm still not getting logged into the site, which is disappointing and I can't figure out why at the moment. :S
private void start_post2()
{
string username = txtUser.Text;
string password = txtPassword.Text;
Uri link = new Uri(txtLink.Text);
string postArgs = string.Format(@"userId={0}&password={1}", username, password);
byte[] buffer = Encoding.ASCII.GetBytes(postArgs);
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create(txtLink.Text);
WebReq.Method = "POST";
//WebReq.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
WebReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
//WebReq.ClientCertificates.Add("Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5");
WebReq.AllowAutoRedirect = false;
WebReq.Accept = "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
WebReq.Accept = "*/*";
//WebReq.Headers.Add(HttpRequestHeader.Cookie, cookieJar);
WebReq.CookieContainer = cookieJar;
WebReq.KeepAlive = true;
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = buffer.Length;
Stream PostData = WebReq.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse WebResp = (HttpWebResponse)WebReq.GetResponse();
if (WebResp == null) throw new Exception("Response is null");
foreach (Cookie cookie in WebResp.Cookies)
{
cookieJar.Add(new Cookie(cookie.Name.Trim(), cookie.Value.Trim(), cookie.Path, cookie.Domain));
//txtResult.Text += cookie.Name.ToString() + Environment.NewLine + cookie.Value.ToString() + Environment.NewLine + cookie.Path.ToString() + Environment.NewLine + cookie.Domain.ToString();
}
if (!string.IsNullOrEmpty(WebResp.Headers["Location"]))
{
string newLocation = WebResp.Headers["Location"];
//Request the new location
WebReq = (HttpWebRequest)WebRequest.Create(newLocation);
WebReq.Method = "GET";
WebReq.ContentType = "application/x-www-form-unlencoded";
WebReq.AllowAutoRedirect = false;
WebReq.CookieContainer = cookieJar;
WebReq.CookieContainer.Add(WebResp.Cookies);
buffer = Encoding.ASCII.GetBytes("userId=" + username + "&password=" + password);
WebReq.ContentLength = buffer.Length;
PostData = WebReq.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
WebResp = (HttpWebResponse)WebReq.GetResponse();
foreach (Cookie cookie in WebResp.Cookies)
{
cookieJar.Add(new Cookie(cookie.Name.Trim(), cookie.Value.Trim(), cookie.Path, cookie.Domain));
//txtResult.Text += cookie.Name.ToString() + Environment.NewLine + cookie.Value.ToString() + Environment.NewLine + cookie.Path.ToString() + Environment.NewLine + cookie.Domain.ToString();
}
}
else if (!string.IsNullOrEmpty(WebResp.Headers["Set-Cookie"]))
{
// thinking...
}
foreach (Cookie cookie in cookieJar.GetCookies(link))
{
MessageBox.Show(cookie.Name.ToString() + Environment.NewLine + cookie.Value.ToString() + Environment.NewLine + cookie.Path.ToString() + Environment.NewLine + cookie.Domain.ToString());
}
StreamReader sr = new StreamReader(WebResp.GetResponseStream());
string responseHtml = sr.ReadToEnd().Trim();
SearchPatient(WebReq, username, password);
}