PHP library to get information from any web page (using oembed, opengraph, twitter-cards, scrapping the html, etc). It's compatible with any web service (youtube, vimeo, flickr, instagram, etc) and has adapters to some sites like (archive.org, github, facebook, etc).
Requirements:
- PHP 7.4+
- Curl library installed
- PSR-17 implementation. By default these libraries are detected automatically:
- laminas/laminas-diactoros
- guzzle/psr7 (Only the unreleased version 2.x, installed as
dev-master) - nyholm/psr7
- sunrise/http-message
If you need PHP 5.5-7.3 support, use the 3.x version
http://oscarotero.com/embed/demo
This package is installable and autoloadable via Composer as embed/embed.
$ composer require embed/embed
use Embed\Embed;
$embed = new Embed();
//Load any url:
$info = $embed->get('https://www.youtube.com/watch?v=PP1xn5wHtxE');
//Get content info
$info->title; //The page title
$info->description; //The page description
$info->url; //The canonical url
$info->keywords; //The page keywords
$info->image; //The thumbnail or main image
$info->code->html; //The code to embed the image, video, etc
$info->code->width; //The exact width of the embed code (if exists)
$info->code->height; //The exact height of the embed code (if exists)
$info->code->aspectRatio; //The aspect ratio (width/height)
$info->authorName; //The resource author
$info->authorUrl; //The author url
$info->cms; //The cms used
$info->language; //The language of the page
$info->languages; //The alternative languages
$info->providerName; //The provider name of the page (Youtube, Twitter, Instagram, etc)
$info->providerUrl; //The provider url
$info->icon; //The big icon of the site
$info->favicon; //The favicon of the site (an .ico file or a png with up to 32x32px)
$info->publishedTime; //The published time of the resource
$info->license; //The license url of the resource
$info->feeds; //The RSS/Atom feedsuse Embed\Embed;
$embed = new Embed();
//Load multiple urls asynchronously:
$infos = $embed->getMulti(
'https://www.youtube.com/watch?v=PP1xn5wHtxE',
'https://twitter.com/carlosmeixidefl/status/1230894146220625933',
'https://en.wikipedia.org/wiki/Tordoia',
);
foreach ($infos as $info) {
echo $info->title;
}The document is the object that store the html code of the page. You can use it to extract extra info from the html code:
//Get the document object
$document = $info->getDocument();
$document->link('image_src'); //Returns the href of a <link>
$document->getDocument(); //Returns the DOMDocument instance
$html = (string) $document; //Returns the html code
$document->select('.//h1'); //SearchYou can perform xpath queries in order to select specific elements. A search always return an instance of a Embed\QueryResult:
//Search the A elements
$result = $document->select('.//a');
//Filter the results
$result->filter(fn ($node) => $node->getAttribute('href'));
$id = $result->str('id'); //Return the id of the first result as string
$text = $result->str(); //Return the content of the first result
$ids = $result->strAll('id'); //Return an array with the ids of all results as string
$texts = $result->strAll(); //Return an array with the content of all results as string
$tabindex = $result->int('tabindex'); //Return the tabindex attribute of the first result as integer
$number = $result->int(); //Return the content of the first result as integer
$href = $result->url('href'); //Return the href attribute of the first result as url (converts relative urls to absolutes)
$url = $result->url(); //Return the content of the first result as url
$node = $result->node(); //Return the first node found (DOMElement)
$nodes = $result->nodes(); //Return all nodes foundFor convenience, the object Metas stores the value of all <meta> elements located in the html, so you can get the values easier. The key of every meta is get from the name, property or itemprop attributes and the value is get from content.
//Get the Metas object
$metas = $info->getMetas();
$metas->all(); //Return all values
$metas->get('og:title'); //Return a key value
$metas->str('og:title'); //Return the value as string (remove html tags)
$metas->html('og:description'); //Return the value as html
$metas->int('og:video:width'); //Return the value as integer
$metas->url('og:url'); //Return the value as full url (converts relative urls to absolutes)In addition to the html and metas, this library uses oEmbed endpoints to get additional data. You can get this data as following:
//Get the oEmbed object
$oembed = $info->getOEmbed();
$oembed->all(); //Return all raw data
$oembed->get('title'); //Return a key value
$oembed->str('title'); //Return the value as string (remove html tags)
$oembed->html('html'); //Return the value as html
$oembed->int('width'); //Return the value as integer
$oembed->url('url'); //Return the value as full url (converts relative urls to absolutes)Additional oEmbed parameters (like instagrams hidecaption) can also be provided:
$embed = new Embed();
$result = $embed->get('https://www.instagram.com/p/B_C0wheCa4V/');
$result->setSettings([
'oembed:query_parameters' => ['hidecaption' => true]
]);
$oembed = $info->getOEmbed();Another API available by default, used to extract info using the JsonLD schema.
//Get the linkedData object
$ld = $info->getLinkedData();
$ld->all(); //Return all data
$ld->get('name'); //Return a key value
$ld->str('name'); //Return the value as string (remove html tags)
$ld->html('description'); //Return the value as html
$ld->int('width'); //Return the value as integer
$ld->url('url'); //Return the value as full url (converts relative urls to absolutes)Some sites like Wikipedia or Archive.org provide a custom API that is used to fetch more reliable data. You can get the API object with the method getApi() but note that not all results have this method. The Api object has the same methods than oEmbed:
//Get the API object
$api = $info->getApi();
$api->all(); //Return all raw data
$api->get('title'); //Return a key value
$api->str('title'); //Return the value as string (remove html tags)
$api->html('html'); //Return the value as html
$api->int('width'); //Return the value as integer
$api->url('url'); //Return the value as full url (converts relative urls to absolutes)Depending of your needs, you may want to extend this library with extra features or change the way it makes some operations.
Embed use some PSR standards to be the most interoperable possible:
- PSR-7 Standard interfaces to represent http requests, responses and uris
- PSR-17 Standard factories to create PSR-7 objects
- PSR-18 Standard interface to send a http request and return a response
Embed comes with a CURL client compatible with PSR-18 but you need to install a PSR-7 / PSR-17 library. Here you can see a list of popular libraries and the library can detect automatically 'laminas\diactoros', 'guzzleHttp\psr7', 'slim\psr7', 'nyholm\psr7' and 'sunrise\http' (in this order). If you want to use a different PSR implementation, you can do it in this way:
use Embed\Embed;
use Embed\Http\Crawler;
$client = new CustomHttpClient();
$requestFactory = new CustomRequestFactory();
$uriFactory = new CustomUriFactory();
//The Crawler is responsible for perform http queries
$crawler = new Crawler($client, $requestFactory, $uriFactory);
//Create an embed instance passing the Crawler
$embed = new Embed($crawler);There are some sites with special needs: because they provide public APIs that allows to extract more info (like Wikipedia or Archive.org) or because we need to change how to extract the data in this particular site. For all that cases we have the adapters, that are classes extending the default classes to provide extra functionality.
Before creating an adapter, you need to understand how Embed work: when you execute this code, you get a Extractor class
//Get the Extractor with all info
$info = $embed->get($url);
//The extractor have document and oembed:
$document = $info->getDocument();
$oembed = $info->getOEmbed();The Extractor class has many Detectors. Each detector is responsible to detect a specific piece of info. For example, there's a detector for the title, other for description, image, code, etc.
So, an adapter is basically an extractor created specifically for a site. It can contains also custom detectors or apis. If you see the src/Adapters folder you can see all adapters.
If you create an adapter, you need also register to Embed, so it knows in which website needs to use. To do that, there's the ExtractorFactory object, that is responsible for instantiate the right extractor for each site.
use Embed\Embed;
$embed = new Embed();
$factory = $embed->getExtractorFactory();
//Use this MySite adapter for mysite.com
$factory->addAdapter('mysite.com', MySite::class);
//Remove the adapter for pinterest.com, so it will use the default extractor
$factory->removeAdapter('pinterest.com');
//Change the default extractor
$factory->setDefault(CustomExtractor::class);Embed comes with several predefined detectors, but you may want to change or add more. Just create a class extending Embed\Detectors\Detector class and register it in the extractor factory. For example:
use Embed\Embed;
use Embed\Detectors\Detector;
class Robots extends Detector
{
public function detect(): ?string
{
$response = $this->extractor->getResponse();
$metas = $this->extractor->getMetas();
return $response->getHeaderLine('x-robots-tag'),
?: $metas->str('robots');
}
}
//Register the detector
$embed = new Embed();
$embed->getExtractorFactory()->addDetector('robots', Robots::class);
//Use it
$info = $embed->get('http://example.com');
$robots = $info->robots;If you need to pass settings to the CurlClient to perform http queries:
use Embed\Embed;
use Embed\Http\Crawler;
use Embed\Http\CurlClient;
$client = new CurlClient();
$client->setSettings([
'cookies_path' => $cookies_path,
'ignored_errors' => [18]
]);
$embed = new Embed(new Crawler($client));If you need to pass settings to your detectors, you can use the setSettings method:
//Create the extractor
$info = $embed->get($url);
//Pass settings for example.com site
if ($info->getUri()->getHost() === 'example.com') {
$info->setSettings(['example_api_key' => 'xxx']);
}Note: The built-in detectors does not require settings. This feature is only for convenience if you create a specific detector that requires settings.
If this library is useful for you, say thanks buying me a beer 🍺!