Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.4 While outright duplication of original expression will in many cases be illegal in the United States the courts ruled in Feist Publications v. Rural Telephone Service that duplication of facts is allowable. U.S. courts have acknowledged that users of "scrapers" or "robots" may be held liable for committing trespass to chattels56 which involves a computer system itself being considered personal property upon which the user of a scraper is trespassing. The best known of these cases eBay Inc. v. Bidder's Edge Inc. resulted in an injunction ordering Bidder's Edge to stop data mining from the eBay Web site. This case involved automatic placing of bids known as Auction Sniping. However to succeed on a claim of trespass to chattels the plaintiff must demonstrate that the defendant intentionally and without authorization interfered with the plaintiff's possessory interest in the computer system and that the defendant's unauthorized use caused damage to the plaintiff. Not all cases of web spidering brought before the courts have been considered trespass to chattels.7

One of the first major tests of screen scraping involved American Airlines and a firm called FareChase.8 AA successfully obtained an injunction from a Texas trial court stopping FareChase from selling software that enables users to compare webfares if it also searches AA's website. The airline argued that FareChase's websearch software trespassed on AA's servers when it collected the publicly available data. The injunction was appealed in 2003.

Southwest Airlines has also challenged screen-scraping practices and has involved both FareChase and another firm Outtask in a legal claim.9 Southwest Airlines charged that the screen-scraping is Illegal since it is an example of "Computer Fraud and Abuse" and has led to "Damage and Loss" and "Unauthorized Access" of Southwest's site. It also constitutes "Interference with Business Relations" "Trespass" and "Harmful Access by Computer". They also claimed that screen-scraping constitutes what is legally known as Misappropriation and Unjust Enrichment and is also a breach of the website's user agreement. Outtask denied all these claims and claimed that the prevailing law in this case should be US Copyright law and that under copyright the pieces of information being scraped would not be subject to copyright protection.

Both these cases have yet to go to the Supreme Court of the United States.

Although these are early scraping decisions and the theories of liability are not uniform it is difficult to ignore a pattern emerging that the courts are prepared to protect proprietary content on commercial Web sites from uses which are undesirable to the owners of such sites. However the degree of protection for such content is not settled and will depend on the type of access made by the scraper the amount of information accessed and copied the degree to which the access adversely affects the Web site owners system and the types and manner of prohibitions on such conduct.10

While the law in this area becomes more settled entities contemplating using scraping programs to access a public Web site should also consider whether such action is authorized by reviewing the terms of use and other terms or notices posted on or made available through the site. In the latest ruling in the Cvent Inc v. Eventbrite Inc. In the United States district court for the eastern district of Virginia the court ruled that the terms of use should be brought to the users' attention In order for a Browse wrap contract/license to be enforced.11 In the plaintiff's website during the period of this trial the terms of use link is displayed among all the links of the site at the bottom of the page as most sites on the internet. This ruling contradicts the Irish ruling described below. The court also rejected the plaintiff's argument that the browsewrap restrictions were enforceable in view of Virginia's adoption of the Uniform Computer Information Transactions Act (UCITA)--a uniform law that many believed was in favor on common browsewrap contracting practices.12

Outside of the United States in a February 2006 ruling the Danish Maritime and Commercial Court (Copenhagen) found systematic crawling indexing and deep linking by portal site ofir.dk of real estate site Home.dk not to conflict with Danish law or the database directive of the European Union.13 In February 2010 the Irish High Court in the case of Ryanair Limited v Billigfluege.de GmbH. This case established a precedent by acknowledging that the Terms of Use on Ryanairs website including the prohibitions contained in those Terms of Use against screen scraping. The case is currently under appeal in the Supreme Court.14 In Australia the Spam Act 2003 outlaws some forms of web harvesting although this only applies to email addresses.151617 Technical measures to stop bots The administrator of a website can use various measures to stop or slow a bot. Some techniques include: If the application is well behaved adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way. Blocking an IP address. This will also block all browsing from that address. Bots sometimes declare who they are and can be blocked on that basis; 'googlebot' is an example. Some bots make no distinction between themselves and a human browser. Bots can be blocked by excess traffic monitoring. Bots can be blocked with tools to verify that it is a real person accessing the site like a CAPTCHA. Commercial anti-bot services: Several companies such as Pramana SiteBlackBox and Sentor offer anti-bot services for websites. A few Web Application Firewalls have limited bot detection capabilities as well. Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers. Sometimes bots can be blocked with carefully crafted Javascript code. Using CSS sprites to display such data as phone numbers or email addresses.18 Tools Apache Camel Automation Anywhere cURL Data Toolbar Firebug Gogybot (scraping library for Microsoft .Net) Greasemonkey HtmlUnit HTTrack iMacros SimpleTest Methabot nokogiri watir Wget WSO2 Mashup Server yahoo pipes Yahoo! query language (yql) See also 80legs Corpus linguistics Data scraping Mashup (web application hybrid) opensocial Scraper site Screen scraping Spamdexing Text corpus Web crawling Notes "How to compare and choose scraping tools". www.fornova.net. 2010-10-27. http://www.fornova.net/blog/p18. Retrieved 2010-10-27.  "Vertical aggregation platforms". www.fornova.net. 2010-10-27. http://www.fornova.net/blog/p21. Retrieved 2010-10-27.  Semantic annotation based web scraping "FAQ about linking - Are website terms of use binding contracts". www.chillingeffects.org. 2007-08-20. http://www.chillingeffects.org/linking/faq.cgi#QID596. Retrieved 2007-08-20.  "Internet Law Ch. 06: Trespass to Chattels". www.tomwbell.com. 2007-08-20. http://www.tomwbell.com/NetLaw/Ch06.html. Retrieved 2007-08-20.  "What are the "trespass to chattels" claims some companies or website owners have brought". www.chillingeffects.org. 2007-08-20. http://www.chillingeffects.org/linking/faq.cgi#QID460. Retrieved 2007-08-20.  "Ticketmaster Corp. v. Tickets.com Inc.". 2007-08-20. http://www.tomwbell.com/NetLaw/Ch07/Ticketmaster.html. Retrieved 2007-08-20.  "American Airlines v. FareChase". 2007-08-20. http://www.fornova.net/documents/AAFareChase.pdf. Retrieved 2007-08-20.  David Jonas (2004-04-26). "SW Courts Corp. Clients: Swabiz Portal Pushes Channel Segmentation". http://www.businesstravelnews.com/. http://www.fornova.net/documents/www-businesstravelnews-com.pdf. Retrieved 2010-10-27.  Kenneth A. Adler (2003-07-29). "Controversy Surrounds 'Screen Scrapers': Software Helps Users Access Web Sites But Activity by Competitors Comes Under Scrutiny". http://library.findlaw.com/2003/Jul/29/132944.html. Retrieved 2010-10-27.  "IN THE UNITED STATES DISTRICT COURT FOR THE EASTERN DISTRICT OF VIRGINIA Alexandria Division". 2010-09-15. http://www.fornova.net/documents/Cvent.pdf. Retrieved 2010-10-27.  "Did Iqbal/Twombly Raise the Bar for Browsewrap Claims". 2010-09-17. http://www.fornova.net/documents/pblog-bna-com.pdf. Retrieved 2010-10-27.  "UDSKRIFT AF S- & HANDELSRETTENS DOMBOG". bvhd.dk. 2006-02-24. http://www.bvhd.dk/uploads/txmocarticles/S-ogHandelsrettensafgrelseiOfir-sagen.pdf. Retrieved 2007-05-30.  LK Shields Solicitors (March 2010). "TERMS AND CONDITIONS MAY PROTECT WEBSITES FROM INFRINGING SCREEN SCRAPERS: RYANAIR V BILLIGFLUEGE.DE". http://www.fornova.net/documents/RyanAir.pdf. Retrieved 2010-10-27.  National Office for the Information Economy (February 2004). "Spam Act 2003: An overview for business" (PDF). Australian Communications Authority. pp. 6. http://www.acma.gov.au/webwr/consumerinfo/spam/spamoverviewfor%20business.pdf. Retrieved 2009-03-09.  National Office for the Information Economy (February 2004). "Spam Act 2003: A practical guide for business" (PDF). Australian Communications Authority. pp. 20. http://www.acma.gov.au/webwr/consumerinfo/frequentlyaskedquestions/spambusinesspracticalguide.pdf. Retrieved 2009-03-09.  Trevor Jeffords (2001-06-01). "What is screen scraping and is it lawful in Australia". http://www.fornova.net/documents/Australia.pdf. Retrieved 2010-10-27.  Jason Belows (2009-08-17). "Further thoughts on hindering screen-scraping" (HTML). screen-scraper.com. pp. 1. http://blog.screen-scraper.com/2009/08/17/further-thoughts-on-hindering-screen-scraping/. Retrieved 2010-11-03.  References Schrenk Michael (2007). Webbots Spiders and Screen Scrapers. No Starch Press. ISBN 978-1-59327-120-6. http://books.google.com/booksid5vFSAAAAMAAJ.  External links The Future of Web Sites Web Services (with a section on web scraping) Open Social Web 3.0 Directory Compare and review web scraping companies