BBC News scraping using Scrapy and MongoDB

The goal of this coding challenge is to create a solution that crawls for articles from a news website (e.g. http://www.bbc.com), cleanses the response, stores it in a mongo database, then makes it available to search via an API. The main goals of this challenge are:

Crawl articles on a news website such as using a crawler framework such as Scrapy in Python.

Cleanse the articles to obtain only information relevant to the news story, e.g. article text, headline, article url, etc.

Store the data in a hosted Mongo database, e.g. MongoDB Atlas, for subsequent search and retrieval.

Finally, writing an API that provides access to the content in the mongo database to search for articles by keyword.