June 28th, 2013, 07:37 AM
Web app to view a list of urls as snapshots
As I receive thousands of URLs to be added to our firewall blacklist every day, I find it impossible to verify which deserve blacklisting. As a result, I receive hundreds of complaints that good sites were blocked. So in order for me to test any URL I have to open it in a browser that is in front of the firewall. For hundreds of thousands of urls this is really impossible.
Therefore, I thought of web application that can be developed which will take a list of e.g. 1000 URLs at a time and opens them all as something like a photo gallery with each screen as 175 x 130, filling the displaying monitor. Each opened screen can be ticked to be selected for deletion. On deletion of selected screens, the originating URLs are also deleted from the providing list.
The outcome of this process is a shorter and "genuine" blacklist that will eliminate users complaints. This solution was just the outcome of my thinking (and suffering!). Is it possible? If not, I welcome any other solution that will facilitate viewing thousands of miniature sites that can be selectively selected for deletion if they should not be blacklisted.
June 28th, 2013, 12:33 PM
Sure it's definitely possible. On a Linux server you could do it with xvfb, firefox, a plugin capable of controlling firefox - like selenium, and a screenshot capture program. That will get you your screenshots. Then as far as analyzing them goes, that would be a fairly simple task for any programming language. I'm not sure about automatically adding them to your firewall rules; that depends on what firewall you're using.
June 28th, 2013, 04:27 PM
Thank you E-Oreo for such promising reply. In fact, I'm using a firewall running pfSense with squid and squidGuard, but actually checking urls is to be done separately on a machine (pc) that is in front of the firewall, ie its traffic is not filtered so as to be able to capture true snapshots of the urls related sites. The intended app is to run on this pc.
Originally Posted by E-Oreo
That pc will have a copy of the blacklist text files which will provide urls for the intended application. Admin will decide which urls are to be tested by the intended application. Max number of snapshots at a time can be set eg 1000.
Displayed snapshots may have a checkboxes to select them for deletion from the providing list as they are viewed. Committing deletion should be confirmed with the admin at the end of each viewing session. The resulting list will be manually added to the firewall (how primitive!).
What do you think after this clarification? Your opinion is highly appreciated by me, thank you.