Published: 31 Jan 2024 › Updated: 31 Jan 2024
How to decode an encoded HttpWebResponse?
I have this piece of code to fetch a Page HTML from an URL, however the response content looks encoded.
Code:
HttpWebRequest xhr = (HttpWebRequest) WebRequest.Create(new Uri("https://www.youtube.com/watch?v=somevideoid"));
xhr.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
//xhr.CookieContainer = request.Account.CookieContainer;
xhr.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
xhr.Headers["Accept-Encoding"] = "gzip, deflate, br";
xhr.Headers["Accept-Language"] = "en-US,en;q=0.5";
xhr.Headers["Upgrade-Insecure-Requests"] = "1";
xhr.KeepAlive = true;
xhr.UserAgent = "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)";
xhr.Host = "www.youtube.com";
xhr.Referer = "https://www.youtube.com/watch?v=6aCpYxzRkf4";
var response = xhr.GetResponse();
string html;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
html = reader.ReadToEnd();
}
These are the response headers:
X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000
Content-Encoding: br
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Date: Sat, 24 Nov 2018 11:30:38 GMT
Expires: Tue, 27 Apr 1971 19:44:06 EST
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=it for more info."
Set-Cookie: PREF=f1=50000000&al=it; path=/; domain=.youtube.com; expires=Thu, 25-Jul-2019 23:23:38 GMT
Server: YouTube Frontend Proxy
The response generated by the server is in br encoding and I need to decode it. support for br encoding is not included in default system compression packages and you will have to install the Brotli.net nuget package.
I added this in my code to cover the 3 main encoding types gzip, br and defalte
HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
Stream responseStream = response.GetResponseStream();
if (response.ContentEncoding.ToLower().Contains("gzip"))
responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
else if (response.ContentEncoding.ToLower().Contains("deflate"))
responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);
else if (response.ContentEncoding.ToLower().Contains("br"))
responseStream = new BrotliStream(responseStream, CompressionMode.Decompress);Leave How to decode an encoded HttpWebResponse? to:
Read more #hive-169321 posts
Best Posts From DotNet Guru
We have not curated any of dotnetguru's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.
More Posts From DotNet Guru
- React vs. Angular: Unraveling the Web Development Dilemma
- Mastering Async/Await: Tackling Deadlocks in Advanced .NET Applications
- Solving the 'System.NullReferenceException' in .NET Applications
- How to decode an encoded HttpWebResponse?
- Algorithms and Codewars
- Execution pipeline in plug-and-play architecture in Asp.NET 5
- Execution context in plug-and-play architecture in Asp.NET 5
- Adapter pattern in plug-and-play architecture in Asp.NET 5
- Preparation for plug-and-play architecture in Asp.NET 5
- Introduction to plug-and-play architecture in Asp.NET 5
- Hello Programming & Dev community