News page download
Issue Description
We need to implement a functionality where a Selenium bot visits the identified articles from Swiss news websites, expands any live tickers or comments, and then downloads the entire HTML for further analysis.
Expected Behavior
- The bot navigates to the specified article link.
- It identifies and expands any live tickers or comments sections.
- It downloads the entire HTML of the page, including the expanded sections.
Current Behavior
Currently, we don't have a bot that can perform this action.
Definition of Done (DoD)
-
Bot can successfully navigate to the article link. -
Bot can identify and expand live tickers and comments sections. -
Bot can download the entire HTML of the page, including expanded sections. -
All downloaded HTMLs are correctly saved for further processing. -
The bot handles errors and exceptions gracefully, logging them for review. -
Code has been reviewed and approved. -
All tests pass.
Additional Information
The bot should be designed to handle any changes in the website's layout with minimal adjustments to the code. It should also manage scenarios where the live tickers or comments sections are not present or not accessible.
Screenshots
[If applicable, add screenshots to help explain your problem.]
Edited by Christof Weickhardt