DotNet Guru avatar

How to decode an encoded HttpWebResponse?

dotnetguru

Published: 31 Jan 2024 › Updated: 31 Jan 2024How to decode an encoded HttpWebResponse?

How to decode an encoded HttpWebResponse?

I have this piece of code to fetch a Page HTML from an URL, however the response content looks encoded.

Code:

HttpWebRequest xhr = (HttpWebRequest) WebRequest.Create(new Uri("https://www.youtube.com/watch?v=somevideoid"));
xhr.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
//xhr.CookieContainer = request.Account.CookieContainer;
xhr.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
xhr.Headers["Accept-Encoding"] = "gzip, deflate, br";
xhr.Headers["Accept-Language"] = "en-US,en;q=0.5";
xhr.Headers["Upgrade-Insecure-Requests"] = "1";
xhr.KeepAlive = true;
xhr.UserAgent = "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)";
xhr.Host = "www.youtube.com";
xhr.Referer = "https://www.youtube.com/watch?v=6aCpYxzRkf4";
var response = xhr.GetResponse();
string html;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
    html = reader.ReadToEnd();
}

These are the response headers:

X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000
Content-Encoding: br
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Date: Sat, 24 Nov 2018 11:30:38 GMT
Expires: Tue, 27 Apr 1971 19:44:06 EST
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=it for more info."
Set-Cookie: PREF=f1=50000000&al=it; path=/; domain=.youtube.com; expires=Thu, 25-Jul-2019 23:23:38 GMT
Server: YouTube Frontend Proxy

The response generated by the server is in br encoding and I need to decode it. support for br encoding is not included in default system compression packages and you will have to install the Brotli.net nuget package.

I added this in my code to cover the 3 main encoding types gzip, br and defalte

HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
Stream responseStream = response.GetResponseStream();

if (response.ContentEncoding.ToLower().Contains("gzip"))
    responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
else if (response.ContentEncoding.ToLower().Contains("deflate"))
    responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);
else if (response.ContentEncoding.ToLower().Contains("br"))
    responseStream = new BrotliStream(responseStream, CompressionMode.Decompress);

Leave How to decode an encoded HttpWebResponse? to:

Written by

github.com/dotnetguru1 codewars.com/users/dotnetguru1

Read more #hive-169321 posts


Best Posts From DotNet Guru

We have not curated any of dotnetguru's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.

More Posts From DotNet Guru