Implementing API in Drupal 8 is quite easy when we are going into headless mode. But implementing sitemap.xml and robots.txt is quite challenging. Since both systems are disconneted and altering API to serve these data is super expensive.

Here is a quick trick to enable sitemap.xml and robots.txt file creation in Drupal 8 and server it to front end server.

Enable and configure sitemap module in Drupal 8

There are many modules available in Drupal.org

XML Sitemap is a good option but their Drupal 8 version not stable. Simple sitemap is alternate solution and it's got most of the features.

Create a route in Express server.

Create a new route in your index.js - This will be the same file where the express module included and the server related configuration specified.

// Specify Drupal side sitemap location
    
let sitemapxml = 'example_com/sitemap.xml'; 
    
app.get('/sitemap.xml', function (req, res) {
	let sitemap = request(sitemapxml);
	req.pipe(sitemap); 
	sitemap.pipe(res);
});

In the above example we used the pipe method. Piping is a great method in which you can read data from the source URL and write to destination URL without managing the flow yourself.

The similar method can be used to serve robots.txt from Drupal to expresJS/NodeJS server.