Need advice building a search engine
(I don't have hadoop experience so excuse me if this is irrelevant)
I am building a search engine that would scrape a number of websites (not the whole internet) I am planning to write the website in django and use scrapy for crawling/scraping.
I am thinking of using elasticsearch with hadoop as it seems straightforward to integrate and there are a lot of documentations and resources around.
I also have user inputted data. reviews, ratings, content moderation and stuff. I am wondering if this could be handled by hadoop as data from here would require joins and complex queries. I am also planning to build analytics data off the hadoop data. So, I am thinking of using mariadb or mysql for this. Is it ok to use two separate databases and how would I integrate the data for both?
I am also planning to add message notifications.
I am open to any ideas on how to go about this. Or maybe share what you are currently using if you're building something similar. thanks!