When working with web scraping or manipulating HTML content in Go, you might often need to extract the content inside thetag and convert it into a string. This can be particularly useful when you want to process or analyze the body content of web pages. In this blog post, we’ll walk through how to achieve this using Go.<\/p>\n
Before we dive into the code, make sure you have Go installed on your machine. If not, you can download it from the official Go website.<\/p>\n
We’ll also be using the following packages:<\/p>\n
net\/http for making HTTP requests.
\ngolang.org\/x\/net\/html for parsing the HTML content.
\nYou can install the html package from golang.org\/x\/net using the following command:<\/p>\n
bash<\/strong> First, we need to fetch the HTML content of the web page. We’ll use the http package for this.<\/p>\n <\/p>\n Next, we’ll parse the HTML content and extract the content inside the Explanation<\/strong> Parsing HTML:<\/strong> We parse the HTML content using html.Parse.<\/p>\n Extracting Body Content:<\/strong> We traverse the parsed HTML nodes to find the tag. Once found, we extract its inner content by rendering each child node of the tag back to a string.<\/p>\n Running the Code go run main.go In this blog post, we’ve shown how to fetch HTML content from a web page and extract the content inside the When working with web scraping or manipulating HTML content in Go, you might often need to extract the content inside thetag and convert it into a string. This can be particularly useful when you want to process or analyze the body content of web pages. In this blog post, we’ll walk through how to achieve […]<\/p>\n","protected":false},"author":1,"featured_media":10788,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[7],"tags":[],"class_list":["post-10786","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-golang"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/posts\/10786"}],"collection":[{"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/comments?post=10786"}],"version-history":[{"count":1,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/posts\/10786\/revisions"}],"predecessor-version":[{"id":10790,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/posts\/10786\/revisions\/10790"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/media\/10788"}],"wp:attachment":[{"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/media?parent=10786"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/categories?post=10786"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bacancytechnology.com\/qanda\/wp-json\/wp\/v2\/tags?post=10786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\ngo get golang.org\/x\/net\/html
\nStep-by-Step Guide<\/p>\nStep 1: Fetch the HTML Content<\/h3>\n
package main\r\nimport (\r\n \"fmt\"\r\n \"net\/http\"\r\n \"io\/ioutil\"\r\n)\r\n\r\nfunc fetchHTML(url string) (string, error) {\r\n resp, err := http.Get(url)\r\n if err != nil {\r\n return \"\", err\r\n }\r\n defer resp.Body.Close()\r\n body, err := ioutil.ReadAll(resp.Body)\r\n if err != nil {\r\n return \"\", err\r\n }\r\n return string(body), nil\r\n}\r\n\r\nfunc main() {\r\n url := \"http:\/\/example.com\"\r\n htmlContent, err := fetchHTML(url)\r\n if err != nil {\r\n fmt.Println(\"Error fetching HTML:\", err)\r\n return\r\n }\r\n fmt.Println(htmlContent)\r\n}\r\n<\/pre>\nStep 2: Parse the HTML and Extract the Body Content<\/h3>\n
\r\npackage main\r\n\r\nimport (\r\n \"fmt\"\r\n \"net\/http\"\r\n \"io\/ioutil\"\r\n \"golang.org\/x\/net\/html\"\r\n \"bytes\"\r\n)\r\n\r\nfunc fetchHTML(url string) (string, error) {\r\n resp, err := http.Get(url)\r\n if err != nil {\r\n return \"\", err\r\n }\r\n defer resp.Body.Close()\r\n\r\n body, err := ioutil.ReadAll(resp.Body)\r\n if err != nil {\r\n return \"\", err\r\n }\r\n\r\n return string(body), nil\r\n}\r\n\r\nfunc extractBodyContent(htmlContent string) (string, error) {\r\n doc, err := html.Parse(bytes.NewReader([]byte(htmlContent)))\r\n if err != nil {\r\n return \"\", err\r\n }\r\n\r\n var bodyContent string\r\n var f func(*html.Node)\r\n f = func(n *html.Node) {\r\n if n.Type == html.ElementNode && n.Data == \"body\" {\r\n for c := n.FirstChild; c != nil; c = c.NextSibling {\r\n var buf bytes.Buffer\r\n html.Render(&buf, c)\r\n bodyContent += buf.String()\r\n }\r\n }\r\n for c := n.FirstChild; c != nil; c = c.NextSibling {\r\n f(c)\r\n }\r\n }\r\n f(doc)\r\n return bodyContent, nil\r\n}\r\nfunc main() {\r\n url := \"http:\/\/example.com\"\r\n htmlContent, err := fetchHTML(url)\r\n if err != nil {\r\n fmt.Println(\"Error fetching HTML:\", err)\r\n return\r\n }\r\n bodyContent, err := extractBodyContent(htmlContent)\r\n if err != nil {\r\n fmt.Println(\"Error extracting body content:\", err)\r\n return\r\n }\r\n fmt.Println(bodyContent)\r\n}\r\n<\/pre>\n
\nFetching HTML Content: We make an HTTP GET request to the specified URL and read the response body.<\/p>\n
\nTo run the code, simply save it to a file, for example main.go, and execute it using the following command:<\/p>\nbash<\/code><\/p>\n
\nReplace http:\/\/example.com with the URL of the web page you want to process.<\/p>\nConclusion<\/h2>\n