Package jdchum/sitex
reads the text content from websites ignoring styling, behavior, and structure. This package can be used to search site text for key words and phrases as well as monitoring text for changes.
go get -u github.com/jdchum/sitex
package main
import (
"io/ioutil"
"github.com/jdchum/sitex"
)
const url = "https://en.wikipedia.org/wiki/Go_(programming_language)"
func main() {
// Get the site's text
text, err := sitex.GetSiteText(url, " ")
if err != nil {
panic(err)
}
// Output the text to disk
err = ioutil.WriteFile("out.txt", []byte(text), 0644)
if err != nil {
panic(err)
}
}
Attempts to parse all human-readable text from a webpage. "Invisible" text such as HTML tags, JavaScript, and CSS are ignored.
url
- URL of the webpage to fetch and parsesep
- Separator to place between chunks of parsed text
Returns the text parsed from the webpage or an error if one occured.
Text is parsed as-is from the initial content returned by the server. This means that content requiring additional network requests or user interactions is not available to the parser.
- Unicode support
- Parse visible text from attributes
- Follow server redirects
- Parse embedded iframes
- Parse embedded PDF text
MIT licensed. Copyright (c) 2019-2020 Joshua Chumbley. See the LICENSE file for details.