diff options
| author | FiveFilters.org <fivefilters@fivefilters.org> | 2016-02-19 22:49:31 +0100 |
|---|---|---|
| committer | FiveFilters.org <fivefilters@fivefilters.org> | 2016-02-19 22:49:31 +0100 |
| commit | b94a698e7353a752baf2db55cef1bc796532d9c5 (patch) | |
| tree | a504f333eedeee323276ca36e0fcc0070dd86e85 /wsj.com.txt | |
| parent | 32d104d0bf3e17a2d9fffc3d441226529a3674e1 (diff) | |
| download | ftr-site-config-b94a698e7353a752baf2db55cef1bc796532d9c5.tar.gz ftr-site-config-b94a698e7353a752baf2db55cef1bc796532d9c5.zip | |
Update wsj.com.txt
Diffstat (limited to 'wsj.com.txt')
| -rw-r--r-- | wsj.com.txt | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/wsj.com.txt b/wsj.com.txt index 467c39c..4b63433 100644 --- a/wsj.com.txt +++ b/wsj.com.txt @@ -18,6 +18,8 @@ strip: //*[contains(@style, 'visibility: hidden;')] strip: //div[contains(@class, 'insetContent') and not(contains(@class, 'image'))] strip: //div[contains(@class, 'carousel')] +http_header(user-agent): Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) + prune: no tidy: no |