forked from apache/nutch
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES.txt
2608 lines (1512 loc) · 96.5 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Fellow committers, Nutch 1.12 contains a breaking change NUTCH-2220. Please use the note below and
in the release announcement and keep it on top in this CHANGES.txt for the Nutch 1.12 release.
* replace your old conf/nutch-default.xml with the conf/nutch-default.xml from Nutch 1.12 release
* if you use LinkDB (e.g. invertlinks) and modified parameters db.max.inlinks and/or db.max.anchor.length
and/or db.ignore.internal.links, rename those parameters to linkdb.max.inlinks and
linkdb.max.anchor.length and linkdb.ignore.internal.links
* db.ignore.internal.links and db.ignore.external.links now operate on the CrawlDB only
* linkdb.ignore.internal.links and linkdb.ignore.external.links now operate on the LinkDB only
Nutch Change Log
* NUTCH-2231 Jexl support in generator job (markus)
* NUTCH-2232 DeduplicationJob should decode URL's before length is compared (Ron van der Vegt via markus)
* NUTCH-2229 Allow Jexl expressions on CrawlDatum's fixed attributes (markus)
* NUTCH-2227 RegexParseFilter (markus)
* NUTCH-2221 Introduce db.ignore.internal.links to FetcherThread (markus)
* NUTCH-2220 Rename db.* options used only by the linkdb to linkdb.* (markus)
* NUTCH-2228 Plugin index-replace unit test broken on Java 8 (snagel via markus)
* NUTCH-2219 Criteria order to be configurable in DeduplicationJob (Ron van der Vegt via markus)
* NUTCH-2218 Update CrawlComplete util to use Commons CLI (Joyce)
* NUTCH-2223 Upgrade xercesImpl to 2.11.0 to fix hang on issue in tika mimetype detection (Tien Nguyen Manh via markus)
* NUTCH-2224 Average bytes/second calculated incorrectly in fetcher (Tien Nguyen Manh via markus)
* NUTCH-2225 Parsed time calculated incorrectly (Tien Nguyen Manh via markus)
* NUTCH-961 Expose Tika's Boilerpipe support (Gabriele Kahlout, Vincent Slot, markus)
* NUTCH-1233 Rely on Tika for outlink extraction (markus)
* NUTCH-2210 Upgrade to Tika 1.12 (markus)
* NUTCH-2209 Improved Tokenization for Similarity Scoring plugin (Sujen)
* NUTCH-2211 Added filterchecker and normalizerchecker to bin/nutch script (markus)
* NUTCH-2197 Add Solr 5 cloud indexer support (Jurian Broertjes via markus)
* NUTCH-2206 Provide example scoring.similarity.stopword.file (sujen)
* NUTCH-2204 Remove junit lib from runtime (snagel)
* NUTCH-2201 Remove loops program from webgraph package (markus)
* NUTCH-1325 HostDB for Nutch (Gui Forget, markus, tejasp)
* NUTCH-2203 Suffix URL filter can't handle trailing/leading whitespaces (Jurian Broertjes via markus)
* NUTCH-2194 Run IndexingFilterChecker as simple Telnet server (markus)
* NUTCH-2196 IndexingFilterChecker to optionally normalize (markus)
* NUTCH-2195 IndexingFilterChecker to optionally follow N redirects (markus)
* NUTCH-2190 Protocol normalizer (markus)
* NUTCH-1838 Host and domain based regex and automaton filtering (markus)
* NUTCH-2178 DeduplicationJob to optionally group on host or domain (markus)
* NUTCH-1449 Optionally delete documents skipped by IndexingFilters (markus)
* NUTCH-2189 Domain filter must deactivate if no rules are present (markus)
* NUTCH-2182 Make reverseUrlDirs file dumper option hash the URL for consistency (joyce)
* NUTCH-2183 Improvement to SegmentChecker for skipping non-segments present in segments directory (lewismc)
* NUTCH-2180 FileDumper skips Corrupt Segments (Harshavardhan Manjunatha via lewismc)
* NUTCH-2042 parse-html increase chunk size used to detect charset (snagel)
* NUTCH-2172 index-more: document format of contenttype-mapping.txt (Nicola Tonellotto, snagel)
Nutch 1.11 Release 03/12/2015 (dd/mm/yyyy)
Release Report: http://s.apache.org/nutch11
* NUTCH-2176 Clean up of log4j.properties (markus)
* NUTCH-2107 plugin.xml to validate against plugin.dtd (snagel)
* NUTCH-2177 Generator produces only one partition even in distributed mode (jnioche, snagel)
* NUTCH-2158 Upgrade to Tika 1.11 (jnioche, snagel)
* NUTCH-2175 Typos in property descriptions in nutch-default.xml (Roannel Fernández Hernández via snagel)
* NUTCH-2069 Ignore external links based on domain (jnioche)
* NUTCH-2173 String.join in FileDumper breaks the build (joyce)
* NUTCH-2166 Add reverse URL format to dump tool (joyce)
* NUTCH-2157 Addressing Miredot REST API Warnings (Sujen Shah)
* NUTCH-2165 FileDumper Util hard codes part-# folder name (joyce)
* NUTCH-2167 Backport TableUtil from 2.x for URL reversing (joyce)
* NUTCH-2160 Upgrade Selenium Java to 2.48.2 (lewismc, kwhitehall)
* NUTCH-2120 Remove MapWritable from trunk codebase (lewismc)
* NUTCH-1911 Improve DomainStatistics tool command line parsing (joyce)
* NUTCH-2064 URLNormalizer basic to encode reserved chars and decode non-reserved chars (markus, snagel)
* NUTCH-2159 Ensure that all WebApp files are copied into generated artifacts for 1.X Webapp (lewismc)
* NUTCH-2154 Nutch REST API (DB) suffering NullPointerException (Aron Ahmadia, Sujen Shah via mattmann)
* NUTCH-2150 Add protocolstats utility (Michael Joyce via mattmann)
* NUTCH-2146 hashCode on the Outlink class (jorgelbg via mattmann)
* NUTCH-2155 Create a "crawl completeness" utility (Michael Joyce via mattmann)
* NUTCH-1988 Make nested output directory dump optional... again (Michael Joyce via lewismc)
* NUTCH-1800 Documentation for Nutch 1.X and 2.X REST APIs (lewismc)
* NUTCH-2149 REST endpoint to read Nutch sequence files (Sujen Shah)
* NUTCH-2139 Basic plugin to index inlinks and outlinks (jorgelbg)
* NUTCH-2128 Review and update mapred --> mapreduce config params in crawl script (lewismc)
* NUTCH-2141 Change the InteractiveSelenium plugin handler Interface to return page content
(Balaji Gurumurthy via mattmann)
* NUTCH-2129 Add protocol status tracking to crawl datum (Michael Joyce via mattmann)
* NUTCH-2142 Nutch File Dump - FileNotFoundException (Invalid Argument) Error (Karanjeet Singh via mattmann)
* NUTCH-2136 Implement a different version of Naive Bayes Parse Filter (Asitang Mishra)
* NUTCH-2109 Create a brute force click-all-ajax-links utility fucntion for selenium interactive plugin (Asitang Mishra)
* NUTCH-2108 Add a function to the selenium interactive plugin interface to do multiple manipulation of driver and then return the data (Asitang Mishra)
* NUTCH-2124 Fetcher following same redirect again and again (Yogendra Kumar Soni via snagel)
* NUTCH-2123 Seed List REST API returns Text but headers indicate/require JSON
(Aron Ahmadia, Sujen Shah via mattmann)
* NUTCH-2086 Nutch 1.X Webui (Sujen Shah, mattmann via lewismc)
* NUTCH-2121 Update javadoc link for Hadoop 2.4.0 in default.properties (Sujen Shah)
* NUTCH-2119 Eclipse shows build path errors on building Nutch (Sujen Shah)
* NUTCH-2117 NutchServer CLI Option for CMD_PORT is incorrect and should be CMD_HOST (zhangmianhongni via lewismc)
* NUTCH-2115 - Add total counts to mimetype stats (Jimmy Joyce via lewismc)
* NUTCH-2111 Delete temporary files location for selenium tmp files after driver quits (Kim Whitehall via lewismc)
* NUTCH-2095 WARC exporter for the CommonCrawlDataDumper (jorgelbg)
* NUTCH-2102 WARC Exporter (jnioche)
* NUTCH-2106 Runtime to contain Selenium and dependencies only once (snagel)
* NUTCH-2104 Add documentation to the protocol-selenium plugin Readme file
re: selenium grid implementation (Kim Whitehall via mattmann)
* NUTCH-2099 Refactoring the REST endpoints for integration with
webui (Sujen Shah via mattmann)
* NUTCH-2098 Add null SeedUrl constructor (Aron Ahmadia via mattmann)
* NUTCH-2093 Indexing filters to use current signatures (markus)
* NUTCH-2092: Unit Test for NutchServer (Sujen Shah via mattmann)
* NUTCH-2096 Explicitly indicate broswer binary to use when selecting
selenium remote option in config (Kim Whitehall via mattmann)
* NUTCH-2090 Refactor Seed Resource in REST API (Sujen Shah
via mattmann)
* NUTCH-2088 Add URL Processing Check to Interactive Selenium
Handlers (Michael Joyce via mattmann)
* NUTCH-2077 Upgrade to Tika 1.10 (Michael Joyce via lewismc)
* NUTCH-1517 CloudSearch indexer (jnioche)
* NUTCH-2085 Upgrade Guava (markus)
* NUTCH-2084 SegmentMerger to report missing input dirs (markus)
* NUTCH-2083 Implement functionality to shadow nutch-selenium-grid-plugin from Mo Omer (lewismc)
* NUTCH-2049 Upgrade to Hadoop 2.4 (lewismc)
* NUTCH-1486 Upgrade to Solr 4.10.2 (lewismc, markus)
* NUTCH-2048 parse-tika: fix dependencies in plugin.xml (Michael Joyce via snagel)
* NUTCH-2066 Parameterize Generate REST endpoint (Sujen Shah via mattmann)
* NUTCH-2072 Deflate encoding support is broken when http.content.limit is set to -1 (Tanguy Moal via mattmann)
* NUTCH-2062 Add Plugin for interacting with Selenium WebDriver (Michael Joyce, mattmann)
* NUTCH-1785 Ability to index raw content (markus, lewismc)
* NUTCH-2063 Add -mimeStats flag to FileDumper tool (Mike Joyce via lewismc)
* NUTCH-2021 Use protocol-selenium to Capture Screenshots of the Page as it is Fetched (lewismc)
* NUTCH-2058 Indexer plugin that allows RegEx replacements on the NutchDocument
field values (Peter Ciuffetti via mattmann)
* NUTCH-2059 protocol-httpclient, protocol-http unit test errors on Jenkins (Peter Ciuffetti via mattmann)
* NUTCH-1980 Jexl expressions for CrawlDbReader (markus)
* NUTCH-1692 SegmentReader was broken in distributed mode (markus, tejasp)
* NUTCH-1684 ParseMeta to be added before fetch schedulers are run (markus)
* NUTCH-2038 fix for NUTCH-2038: Naive Bayes classifier based html Parse filter (for filtering outlinks)
(Asitang Mishra, snagel via mattmann)
* NUTCH-2041 indexer fails if linkdb is missing (snagel)
* NUTCH-2016 Remove unused class OldFetcher (snagel)
* NUTCH-2000 Link inversion fails with .locked already exists (jnioche, snagel)
* NUTCH-2036 Adding some continuous crawl goodies to the crawl script (jorge, snagel)
* NUTCH-2039 Relevance based scoring filter (Sujen Shah, lewismc via mattmann)
* NUTCH-2037 Job endpoint to support Indexing from the REST API (Sujen Shah via mattmann)
* NUTCH-2017 Remove debug log from MimeUtil (snagel)
* NUTCH-2027 seed list REST endpoint for Nutch 1.10 (Asitang Mishra via mattmann)
* NUTCH-2031 Create Admin End point for Nutch 1.x REST service (Sujen Shah via mattmann)
* NUTCH-2015 Make FetchNodeDb optional (off by default) if NutchServer is not used (Sujen Shah via mattmann)
* NUTCH-208 http: proxy exception list: (Matthias Günter, siren, markus, lewismc)
* NUTCH-2007 add test libs to classpath of bin/nutch junit (snagel)
* NUTCH-1995 Add support for wildcard to http.robot.rules.whitelist (totaro)
* NUTCH-2013 Fetcher: missing logs "fetching ..." on stdout (snagel)
* NUTCH-2014 Fetcher hang-up on completion (snagel)
* NUTCH-2011 Endpoint to support realtime JSON output from the fetcher (Sujen Shah via mattmann)
* NUTCH-2006 IndexingFiltersChecker to take custom metadata as input (jnioche)
* NUTCH-2008 IndexerMapReduce to use single instance of NutchIndexAction for deletions (snagel)
* NUTCH-1998 Add support for user-defined file extension to CommonCrawlDataDumper (totaro via mattmann)
* NUTCH-1873 Solr IndexWriter/Job to report number of docs indexed. (snagel via lewismc)
* NUTCH-1934 Refactor Fetcher in trunk (lewismc)
* NUTCH-2004 ParseChecker does not handle redirects (mjoyce via lewismc)
Nutch 1.10 Release - 29/04/2015 (dd/mm/yyyy)
Release Report: http://s.apache.org/nutch10
* NUTCH-1969 URL Normalizer properly handling slashes (markus via mattmann)
* NUTCH-2001 Sub Collection Field Name incorrect in nutch-default.xml
(Jeff Cocking via mattmann)
* NUTCH-1997 Add CBOR "magic header" to CommonCrawlDataDumper
output (Giuseppe Totaro, Luke Sh via mattmann)
* NUTCH-1991 Tika mime detection not using Nutch supplied tika-mimetypes.xml for content based
detection (Iain Lopata, snagel via mattmann)
* NUTCH-1994 Upgrade to Apache Tika 1.8 (lewismc)
* NUTCH-1996 Make protocol-selenium README part of plugin (lewismc)
* NUTCH-1990 Use URI.normalise() in BasicURLNormalizer (snagel, jnioche)
* NUTCH-1973 Job Administration end point for the REST service (Sujen Shah via mattmann)
* NUTCH-1697 SegmentMerger to implement Tool (markus, snagel)
* NUTCH-1987 - Make bin/crawl indexer agnostic (Michael Joyce, snagel via mattmann)
* NUTCH-1989 Handling invalid URLs in CommonCrawlDataDumper (Giuseppe Totaro via mattmann)
* NUTCH-1988 Make nested output directory dump optional (Michael Joyce via mattmann)
* NUTCH-1927 Create a whitelist of IPs/hostnames to allow skipping of RobotRules parsing (mattmann, snagel)
* NUTCH-1986 Clarify Elastic Search Indexer Plugin Settings (Michael Joyce via mattmann)
* NUTCH-1906 Typo in CrawlDbReader command line help (Michael Joyce via mattmann)
* NUTCH-1911 Improve DomainStatistics tool command line parsing (Michael Joyce via mattmann)
* NUTCH-1854 bin/crawl fails with a parsing fetcher (Asitang Mishra via snagel)
* NUTCH-1981 Upgrade to icu4j 55.1 (Marko Asplund via snagel)
* NUTCH-1960 JUnit test for dump method of CommonCrawlDataDumper (Giuseppe Totaro via mattmann)
* NUTCH-1983 CommonCrawlDumper and FileDumper don't dump correct JSON (mattmann)
* NUTCH-1972 Dockerfile for Nutch 1.x (Michael Joyce via mattmann)
* NUTCH-1771 Indexer fails if a segment is corrupted or incomplete (Diaa, Chong Li via snagel)
* NUTCH-1975 New configuration for CommonCrawlDataDumper tool (Giuseppe Totaro via mattmann)
* NUTCH-1979 CrawlDbReader to implement Tool (markus)
* NUTCH-1970 Pretty print JSON output in config resource (Tyler Pasulich, mattmann)
* NUTCH-1976 Allow Users to Set Hostname for Server (Tyler Palsulich via mattmann)
* NUTCH-1941 Optional rolling http.agent.name's (Asitang Mishra, lewismc via snagel)
* NUTCH-1959 Improving CommonCrawlFormat implementations (Giuseppe Totaro via mattmann)
* NUTCH-1974 keyPrefix option for CommonCrawlDataDumper tool (Giuseppe Totaro via mattmann)
* NUTCH-1968 File Name too long issue of DumpFileUtil.java file (Xin Zhang, Renxia Wang via mattmann)
* NUTCH-1966 Configuration endpoint for 1x REST API (Sujen Shah via mattmann)
* NUTCH-1967 Possible SIooBE in MimeAdaptiveFetchSchedule (markus)
* NUTCH-1957 FileDumper output file name collisions (Renxia Wang via mattmann)
* NUTCH-1955 ByteWritable missing in NutchWritable (markus)
* NUTCH-1956 Members to be public in URLCrawlDatum (markus)
* NUTCH-1954 FilenameTooLong error appears in CommonCrawlDumper (mattmann)
* NUTCH-1949 Dump out the Nutch data into the Common Crawl format (Giuseppe Totaro via lewismc)
* NUTCH-1950 File name too long (Jiaheng Zhang, Chong Li via mattmann)
* NUTCH-1921 Optionally disable HTTP if-modified-since header (markus)
* NUTCH-1933 nutch-selenium plugin (Mo Omer, Mohammad Al-Moshin, lewismc)
* NUTCH-827 HTTP POST Authentication (Jasper van Veghel, yuanyun.cn, snagel, lewismc)
* NUTCH-1724 LinkDBReader to support regex output filtering (markus)
* NUTCH-1939 Fetcher fails to follow redirects (Leo Ye via snagel)
* NUTCH-1913 LinkDB to implement db.ignore.external.links (markus, snagel)
* NUTCH-1925 Upgrade to Apache Tika 1.7 (Tyler Palsulich via markus)
* NUTCH-1323 AjaxNormalizer (markus)
* NUTCH-1918 TikaParser specifies a default namespace when generating DOM (jnioche)
* NUTCH-1889 Store all values from Tika metadata in Nutch metadata (jnioche)
* NUTCH-865 Format source code in unique style (lewismc)
* NUTCH-1893 Parse-tika failes to parse feed files (Mengying Wang via snagel)
* NUTCH-1920 Upgrade Nutch to use Java 1.7 (lewismc)
* NUTCH-1919 Getting timeout when server returns Content-Length: 0 (jnioche)
* NUTCH-1912 Dump tool -mimetype parameter needs to be optional to prevent NPE (Tyler Palsulich via lewismc)
* NUTCH-1881 ant target resolve-default to keep test libs (snagel)
* NUTCH-1660 Index filter for Page's latitude and longitude (Yasin Kılınç, lewismc)
* NUTCH-1140 index-more plugin, resetTitle creates multiple values in title field (Joe Liedtke, kaveh minooie via snagel)
* NUTCH-1904 Schema for Solr4 doesn't include _version_ field (mattmann)
* NUTCH-1897 Easier debugging of plugin XML errors (markus)
* NUTCH-1823 Upgrade to elasticsearch 1.4.1 (Phu Kieu, markus via lewismc)
* NUTCH-1592 TikaParser can uppercase the element names while generating the DOM (jnioche)
* NUTCH-1877 Suffix URL filter to ignore query string by default (markus via snagel)
* NUTCH-1890 Major Typo in Documentation for Integrating Nutch and Solr (Boadu Akoto Charles Jnr, mattmann)
* NUTCH-1887 Specify HTMLMapper to use in TikaParser (jnioche)
* NUTCH-1884 NullPointerException in parsechecker and indexchecker with symlinks in file URL (Mengying Wang, snagel)
* NUTCH-1825 protocol-http may hang for certain web pages (Phu Kieu via snagel)
* NUTCH-1483 Can't crawl filesystem with protocol-file plugin (Rogério Pereira Araújo, Mengying Wang, snagel)
* NUTCH-1885 Protocol-file should treat symbolic links as redirects (Mengying Wang, snagel)
* NUTCH-1880 URLUtil should not add additional slashes for file URLs (snagel)
* NUTCH-1879 Regex URL normalizer should remove multiple slashes after file: protocol (snagel)
* NUTCH-1883 bin/crawl: use function to run bin/nutch and check exit value (snagel)
* NUTCH-1865 Enable use of SNAPSHOT's with Nutch Ivy dependency management (lewismc)
* NUTCH-1882 ant eclipse target to add output path to src/test (snagel)
* NUTCH-1876 Upgrade to Crawler Commons 0.5 (jnioche)
* NUTCH-1874 FileDumper comment typos ( Arthur Cinader via lewismc)
* NUTCH-1164 Write JUnit tests for protocol-http (nimafl via snagel)
* NUTCH-1868 Document and improve CLI for FileDumper tool (lewismc)
* NUTCH-1869 Add a flag to -mimeType fiag to FileDumper (lewismc)
* NUTCH-1867 CrawlDbReader: use setFloat to pass min score (lewismc, snagel)
* NUTCH-1826, NUTCH-1864 indexchecker fails if solr.server.url not configured (lewismc, snagel)
* NUTCH-1866 ant eclipse target should not delete runtime (nimafl via lewismc)
* NUTCH-1857 readb -dump -format csv should use comma (lewismc)
* NUTCH-1853 Add commented out WebGraph executions to ./bin/crawl (lewismc)
* NUTCH-1844 testresources/testcrawl not referenced anywhere in code (mattmann)
* NUTCH-1839 Improve WebGraph CLI parsing (lewismc)
* NUTCH-1526 Create SegmentContentDumperTool for easily extracting out file contents from SegmentDirs (mattmann, lewismc, Julien Le Dem)
* NUTCH-1840 the describe function in SolrIndexWriter is not correct (kaveh minooie via jnioche)
* NUTCH-1837 Upgrade to Tika 1.6 (jnioche)
* NUTCH-1829 Generator : unable to distinguish real errors (Mathieu Bouchard via jnioche)
* NUTCH-1835 Nutch's Solr schema doesn't work with Solr 4.9 because of the RealTimeGet handler (mattmann)
* NUTCH-1833 Include version number within nutch binary usage statement (Rishi Verma via mattmann)
* NUTCH-1832 Make Nutch work without an indexer (mattmann)
* NUTCH-1828 bin/crawl : incorrect handling of nutch errors (Mathieu Bouchard via jnioche)
* NUTCH-1775 IndexingFilter: document origin of passed CrawlDatum (snagel)
* NUTCH-1693 TextMD5Signature computed on textual content (Tien Nguyen Manh, markus via snagel)
* NUTCH-1409 remove deprecated properties db.{default,max}.fetch.interval, generate.max.per.host.by.ip (Matthias Agethle via snagel)
Nutch 1.9 Release Change Log - 12/08/2014 (dd/mm/yyyy)
Release Report - http://s.apache.org/1.9-release
* NUTCH-1561 improve usability of parse-metatags and index-metadata (snagel)
* NUTCH-1708 use same id when indexing and deleting redirects (snagel)
* NUTCH-1818 Add deps-test-compile task for building plugins (jnioche)
* NUTCH-1817 Remove pom.xml from source (jnioche)
* NUTCH-926 Redirections from META tag don't get filtered (snagel)
* NUTCH-1422 Bypass signature comparison when a document is redirected (snagel)
* NUTCH-1502 Test for CrawlDatum state transitions (snagel)
* NUTCH-1804 Move JUnit dependency to test scope (jnioche)
* NUTCH-1811 bin/nutch junit to use junit 4 test runner (snagel)
* NUTCH-1799 ANT Eclipse task discovers all plugin jars automatically (jnioche)
* NUTCH-578 URL fetched with 403 is generated over and over again (snagel)
* NUTCH-1776 Log incorrect plugin.folder file path (Diaa via snagel)
* NUTCH-1566 bin/nutch to allow whitespace in paths (tejasp, snagel)
* NUTCH-1605 MIME type detector recognizes xlsx as zip file (snagel)
* NUTCH-1802 Move TestbedProxy to test environment (jnioche)
* NUTCH-1803 Put test dependencies in a separate lib dir (jnioche)
* NUTCH-385 Improve description of thread related configuration for Fetcher (jnioche,lufeng)
* NUTCH-1633 slf4j is provided by hadoop and should not be included in the job file (kaveh minooie via jnioche)
* NUTCH-1787 update and complete API doc overview page (snagel)
* NUTCH-1767 remove special treatment of "params" in relative links (snagel)
* NUTCH-1718 redefine http.robots.agent as "additional agent names" (snagel, Tejas Patil, Daniel Kugel)
* NUTCH-1794 IndexingFilterChecker to optionally dumpText (markus)
* NUTCH-1590 [SECURITY] Frame injection vulnerability in published Javadoc (jnioche)
* NUTCH-1793 HttpRobotRulesParser not configured properly (jnioche)
* NUTCH-1647 protocol-http throws 'unzipBestEffort returned null' for redirected pages (jnioche)
* NUTCH-1736 Can't fetch page if http response header contains Transfer-Encoding：chunked (ysc via jnioche)
* NUTCH-1782 NodeWalker to return current node (markus)
* NUTCH-1758 IndexChecker to send document to IndexWriters (jnioche)
* NUTCH-1786 CrawlDb should follow db.url.normalizers and db.url.filters (Diaa via markus)
* NUTCH-1757 ParserChecker to take custom metadata as input (jnioche)
* NUTCH-1676 Add rudimentary SSL support to protocol-http (jnioche, markus)
* NUTCH-1772 Injector does not need merging if no pre-existing crawldb (jnioche)
* NUTCH-1752 Cache robots.txt rules per protocol:host:port (snagel)
* NUTCH-1613 Timeouts in protocol-httpclient when crawling same host with >2 threads (brian44 via jnioche)
* NUTCH-1766 Generator to unlock crawldb and remove tempdir if generate job fails (Diaa via jnioche)
* NUTCH-207 Bandwidth target for fetcher rather than a thread count (jnioche)
* NUTCH-1182 fetcher to log hung threads (snagel)
* NUTCH-1759 Upgrade to Crawler Commons 0.4 (jnioche)
* NUTCH-1764 readdb to show command-line help if no action (-stats, -dump, etc.) given (Diaa via snagel)
* NUTCH-1700 Remove deprecated code from creativecommons plugin (lewismc)
* NUTCH-1761 Crawl script fails to find job file if not started from inside bin dir (David Hosking, jnioche)
* NUTCH-1603 ZIP parser complains about truncated PDF file (snagel)
* NUTCH-1720 Duplicate lines in HttpBase.java (Walter Tietze via jnioche)
* NUTCH-1750 Improvement of Fetcher's reportStatus (jnioche)
* NUTCH-1747 Use AtomicInteger as semaphore in Fetcher (jnioche)
* NUTCH-1735 code dedup fetcher queue redirects (snagel)
* NUTCH-1745 Upgrade to ElasticSearch 1.1.0 (jnioche)
* NUTCH-1645 Junit Test Case for Adaptive Fetch Schedule class (Yasin Kılınç, lufeng, Sertac TURKEL via snagel)
* NUTCH-1737 Upgrade to recent JUnit 4.x (lewismc)
* NUTCH-1733 parse-html to support HTML5 charset definitions (snagel)
* NUTCH-1671 indexchecker to add digest field (snagel, lufeng)
Nutch 1.8 - 11/03/2014 (dd/mm/yyyy)
Release Report - http://s.apache.org/oHY
* NUTCH-1706 IndexerMapReduce does not remove db_redir_temp (markus, snagel)
* NUTCH-1113 SegmentMerger can now be safely used to merge segments (Edward Drapkin, markus, snagel)
* NUTCH-1729 Upgrade to Tika 1.5 (jnioche)
* NUTCH-1707 DummyIndexingWriter (markus)
* NUTCH-1721 Upgrade to Crawler commons 0.3 (tejasp)
* NUTCH-1253 Incompatable neko and xerces versions (snagel, lewismc)
* NUTCH-1715 RobotRulesParser adds additional '*' to the robots name (tejasp)
* NUTCH-356 Plugin repository cache can lead to memory leak (Enrico Triolo, Doğacan Güney via markus)
* NUTCH-1413 Record response time (Yasin Kılınç, Talat Uyarer, snagel)
* NUTCH-1680 CrawlDbReader to dump minRetry value (markus)
* NUTCH-1699 Tika Parser - Image Parse Bug (Mehmet Zahid Yüzügüldü, snagel via lewismc)
* NUTCH-1695 Add NutchDocument.toString() to ease debugging (markus)
* NUTCH-1675 NutchField to support long (markus)
* NUTCH-1670 set same crawldb directory in mergedb parameter (lufeng via tejasp)
* NUTCH-1080 Type safe members, arguments for better readability (tejasp)
* NUTCH-1360 Suport the storing of IP address connected to when web crawling (lewismc, ferdy and Yasin Kılınç)
* NUTCH-1681 In URLUtil.java, toUNICODE method does not work correctly (İlhami KALKAN, snagel via markus)
* NUTCH-1668 Remove package org.apache.nutch.indexer.solr (jnioche)
* NUTCH-1621 Remove deprecated class o.a.n.crawl.Crawler (Rui Gao via jnioche)
* NUTCH-656 Generic Deduplicator (jnioche, snagel)
* NUTCH-1100 Avoid NPE in SOLRDedup (markus)
* NUTCH-1666 Optimisation for BasicURLNormalizer (jnioche)
* NUTCH-1656 ParseMeta not passed to CrawlDatum for not_modified (markus)
* NUTCH-1606 Check that Factory classes use the cache in a thread safe way (jnioche)
* NUTCH-1653 AbstractScoringFilter (jnioche)
* NUTCH-1562 Order of execution for scoring filters (jnioche, snagel)
* NUTCH-1640 Reuse ParseUtil instance in ParseSegment (Mitesh Singh Jat via jnioche)
* NUTCH-1639 bin/crawl fails on mac os (various contributors via snagel)
* NUTCH-1646 IndexerMapReduce to consider DB status (markus)
* NUTCH-1636 Indexer to normalize and filter repr URL (Iain Lopata via snagel)
* NUTCH-1637 URLUtil is missing getProtocol (markus)
* NUTCH-1622 Create Outlinks with metadata (jnioche)
* NUTCH-1629 Injector skips empty lines in seed files (kaveh minooie via jnioche)
* NUTCH-911 protocol-file to return proper protocol status (Peter Lundberg via snagel)
* NUTCH-806 Merge CrawlDBScanner with CrawlDBReader (jnioche)
* NUTCH-1587 misspelled property "threshold" in conf/log4j.properties (snagel)
* NUTCH-1604 ProtocolFactory not thread-safe (jnioche)
* NUTCH-1595 Upgrade to Tika 1.4 (jnioche, markus)
* NUTCH-1598 ElasticSearchIndexer to read ImmutableSettings from config (markus)
* NUTCH-1520 SegmentMerger looses records (markus)
* NUTCH-1602 improve the readability of metadata in readdb dump normal (lufeng)
* NUTCH-1596 HeadingsParseFilter not thread safe (snagel via markus)
* NUTCH-1597 HeadingsParseFilter to trim and remove exess whitespace (markus)
* NUTCH-1601 ElasticSearchIndexer fails to properly delete documents (markus)
* NUTCH-1600 Injector overwrite does not always work properly (markus)
* NUTCH-1581 CrawlDB csv output to include metadata (markus)
* NUTCH-1327 QueryStringNormalizer (markus)
* NUTCH-1593 Normalize option missing in SegmentMerger's usage (markus)
* NUTCH-1580 index-static returns object instead of value for index.static (Antoinette, lewismc, snagel)
* NUTCH-1126 JUnit test for urlfilter-prefix (Talat UYARER via markus)
Apache Nutch 1.7 Release - 06/20/2013 (mm/dd/yyyy)
Release report - http://s.apache.org/1zE
* NUTCH-1585 Ensure duplicate tags do not exist in microformat-reltag tag set. (lewismc)
* NUTCH-1583 Headings plugin to support multivalued headings (markus)
* NUTCH-1245 URL gone with 404 after db.fetch.interval.max stays db_unfetched in CrawlDb (snagel)
* NUTCH-1527 Elasticsearch indexer (lufeng + markus)
* NUTCH-1475 Index-More Plugin -- A better fall back value for date field (James Sullivan, snagel via lewismc)
* NUTCH-1560 index-metadata to add all values of multivalued metadata (snagel)
* NUTCH-1467 Not able to parse mutliValued metatags (kiran via snagel)
* NUTCH-1430 Freegenerator records overwrite CrawlDB records with AdaptiveFetchSchedule (markus)
* NUTCH-1522 Upgrade to Tika 1.3 (jnioche)
* NUTCH-1578 Upgrade to Hadoop 1.2.0 (markus)
* NUTCH-1577 Add target for creating eclipse project (tejasp)
* NUTCH-1513 Support Robots.txt for Ftp urls (tejasp)
* NUTCH-1249 and NUTCH-1275 : Resolve all issues flagged up by adding javac -Xlint argument (tejasp)
* NUTCH-1053 Parsing of RSS feeds fails (tejasp)
* NUTCH-956 solrindex issues: add field tld to Solr schema (Alexis via lewismc, snagel)
* NUTCH-1277 Fix [fallthrough] javac warnings (tejasp)
* NUTCH-1514 Phase out the deprecated configuration properties (if possible) (tejasp)
* NUTCH-1334 NPE in FetcherOutputFormat (jnioche via tejasp)
* NUTCH-1549 Fix deprecated use of Tika MimeType API in o.a.n.util.MimeUtil (tejasp)
* NUTCH-346 Improve readability of logs/hadoop.log (Renaud Richardet via tejasp)
* NUTCH-829 duplicate hadoop temp files (Mike Baranczak, lewismc, tejasp)
* NUTCH-1501 Harmonize behavior of parsechecker and indexchecker (snagel + lewismc)
* NUTCH-1031 Delegate parsing of robots.txt to crawler-commons (tejasp)
* NUTCH-1547 BasicIndexingFilter - Problem to index full title (Feng)
* NUTCH-1389 parsechecker and indexchecker to report truncated content (snagel)
* NUTCH-1419 parsechecker and indexchecker to report protocol status (snagel + lewismc)
* NUTCH-1047 Pluggable indexing backends (jnioche)
* NUTCH-1536 Ant build file has hardcoded conf dir location (zm via lewismc)
* NUTCH-1420 Get rid of the dreaded � (markus via lewismc)
* NUTCH-1521 CrawlDbFilter pass null url to urlNormalizers (Lufeng via lewismc)
* NUTCH-1284 Add site fetcher.max.crawl.delay as log output by default (tejasp)
* NUTCH-1453 Substantiate tests for IndexingFilters (lufeng via lewismc)
* NUTCH-840 Port tests from parse-html to parse-tika (lewismc, jnioche)
* NUTCH-1509 Implement read/write in NutchField (markus)
* NUTCH-1507 Remove FetcherOutput (markus)
* NUTCH-1506 Add UPDATE action to NutchIndexAction (markus)
* NUTCH-1500 bin/crawl fails on step solrindex with wrong path to segment (Tristan Buckner, snagel)
* NUTCH-1274 Fix [cast] javac warnings (tejasp via lewismc)
* NUTCH-1494 RSS feed plugin seems broken (Sourajit Basak, tejasp and lewismc)
* NUTCH-1127 JUnit test for urlfilter-validator (tejasp via lewismc)
* NUTCH-1119 JUnit test for index-static (tejasp via lewismc)
* NUTCH-1510 Upgrade to Hadoop 1.1.1 (markus)
* NUTCH-1118 JUnit test for index-basic (tejasp via lewismc)
* NUTCH-1331 limit crawler to defined depth (jnioche)
Release 1.6 - 23/11/2012
* NUTCH-1370 Expose exact number of urls injected @runtime (snagel via lewismc)
* NUTCH-1117 JUnit test for index-anchor (lewismc)
* NUTCH-1451 Upgrade automaton jar to 1.11-8 (lewismc)
* NUTCH-1488 bin/nutch to run junit from any directory (snagel via lewismc)
* NUTCH-1493 Error adding field 'contentLength'='' during solrindex using index-more (Nathan Gass via lewismc)
* NUTCH-1491 Strip UTF-8 non-character codepoints in title (Nathan Gass via markus)
* NUTCH-1421 RegexURLNormalizer to only skip rules with invalid patterns (snagel)
* NUTCH-1341 NotModified time set to now but page not modified (markus)
* NUTCH-1215 UpdateDB should not require segment as input (markus)
* NUTCH-1383 IndexingFiltersChecker to show error message instead of null pointer exception (snagel)
* NUTCH-1476 SegmentReader getStats should set parsed = -1 if no parsing took place (snagel)
* NUTCH-1252 SegmentReader -get shows wrong data (snagel)
* NUTCH-1344 BasicURLNormalizer to normalize https same as http (snagel)
* NUTCH-706 Url regex normalizer: pattern for session id removal not to match "newsId" (Meghna Kukreja via snagel)
* NUTCH-1415 release packages to contain top level folder apache-nutch-x.x (snagel)
* NUTCH-1441 AnchorIndexingFilter should use plain HashSet (ferdy via lewismc)
* NUTCH-1470 Ensure test files are included for runtime testing (lewismc)
* NUTCH-1434 Indexer to delete robots noindex (markus)
* NUTCH-1443 Solr schema version is invalid (markus)
* NUTCH-1417 Remove o.a.n.metadata.Office (lewismc)
* NUTCH-1376 Add description parameter to every ant task (lewismc)
* NUTCH-1440 reconfigure non-existent stopwords_en.txt in schema-solr4.xml (shekhar sharma via lewismc)
* NUTCH-1439 Define boost field as type float in schema-solr4.xml (shekhar sharma via lewismc)
* NUTCH-1433 Upgrade to Tika 1.2 (jnioche)
* NUTCH-1388 Optionally maintain custom fetch interval despite AdaptiveFetchSchedule (markus)
* NUTCH-1430 Freegenerator records overwrite CrawlDB records with AdaptiveFetchSchedule (markus)
* NUTCH-1087 Deprecate crawl command and replace with example script (jnioche)
* NUTCH-1306 Add option to not commit and clarify existing solr.commit.size (ferdy)
* NUTCH-1405 Allow to overwrite CrawlDatum's with injected entries (markus)
* NUTCH-1412 Upgrade commons lang (markus)
* NUTCH-1251 SolrDedup to use proper Lucene catch-all query (Arkadi Kosmynin via markus)
* NUTCH-1407 BasicIndexingFilter to optionally add domain field (markus)
* NUTCH-1408 RobotRulesParser main doesn't take URL's (markus)
* NUTCH-1300 Indexer to filter normalize URL's (markus)
* NUTCH-1330 WebGraph OutlinkDB to preserve back up (markus)
* NUTCH-1319 HostNormalizer plugin (markus)
* NUTCH-1386 Headings filter not to add empty values (markus)
* NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling (ferdy via markus)
* NUTCH-1352 Improve regex urlfilters/normalizers synchronization (ferdy via markus)
* NUTCH-1024 Dynamically set fetchInterval by MIME-type (markus)
* NUTCH-1364 Add a counter in Generator for malformed urls (lewismc)
* NUTCH-1262 Map `duplicating` content-types to a single type (markus)
* NUTCH-1385 More robust plug-in order properties in nutch-site.xml (Andy Xue via markus)
* NUTCH-1336 Optionally not index db_notmodified pages (markus)
* NUTCH-1346 Follow outlinks to ignore external (markus)
* NUTCH-1320 IndexChecker and ParseChecker choke on IDN's (markus)
* NUTCH-1351 DomainStatistics to aggregate by TLD (markus)
* NUTCH-1381 Allow to override default subcollection field name (markus)
* NUTCH-XX Commit to add configuration for separation of ant distribution targets (lewismc + jnioche)
Release 1.5.1 - 07/10/2012
* NUTCH-1404 Nutch script fails to find job file in deploy mode (sidabatra, jnioche)
* NUTCH-1415 release packages to contain top level folder apache-nutch-x.x (snagel via lewismc)
* NUTCH-1400 Remove developer -core option for bin/nutch (jnioche)
* NUTCH-1384 Typo in ParseSegment's run-method (Matthias Agethle via markus)
* NUTCH-1398 Upgrade to Hadoop 1.0.3 (jnioche)
Release 1.5 - 04/15/2012
* NUTCH-1208 Don't include KEYS file in bin distribution (jnioche)
* NUTCH-1234 Upgrade to Tika 1.1 (jnioche, markus)
* NUTCH-809 Parse-metatags plugin (jnioche)
* NUTCH-1310 Nutch to send HTTP-accept header (markus)
* NUTCH-1305 Domain(blacklist)URLFilter to trim entries (markus)
* NUTCH-1307 Improve formatting of ant targets for clearer project help (lewismc)
* NUTCH-1299 LinkRank inverter to ignore records without Node (markus)
* NUTCH-1258 MoreIndexingFilter should be able to read Content-Type from both parse metadata and content metadata (jnioche, markus)
* NUTCH-1293 IndexingFiltersChecker to store detected content type in crawldatum metadata (markus)
* NUTCH-1291 Fetcher to stringify exception on // unexpected exception (markus)
* NUTCH-965 Skip parsing for truncated documents (alexis, lewismc, ferdy)
* NUTCH-1210 DomainBlacklistFilter (markus)
* NUTCH-1193 Incorrect url transform to lowercase: parameter solr (Eduardo dos Santos Leggiero via lewismc)
* NUTCH-1272 Wrong property name for index-static in nutch-default.xml (Daniel Baur via jnioche)
* NUTCH-1259 Store detected content-type in crawldatum metadata (jnioche, markus)
* NUTCH-1266 Subcollection to optionally write to configured fields (markus)
* NUTCH-1005 Parse headings plugin (markus)
* NUTCH-1264 Configurable indexing plugin index-metadata (jnioche)
* NUTCH-1242 Allow disabling of URL Filters in ParseSegment (Edward Drapkin via markus)
* NUTCH-1256 WebGraph to dump host + score (markus)
* NUTCH-1260 Fetcher should log fetching of redirects (Sebastian Nagel via markus)
* NUTCH-1255 Change ivy.xml of all plugins to remove "nutch.root" property (ferdy)
* NUTCH-1248 Generator to select on status (markus)
* NUTCH-1177 Generator to select on retry interval (markus)
* NUTCH-1246 Upgrade to Hadoop 1.0.0 (jnioche)
* NUTCH-1139 Indexer to delete gone documents (markus)
* NUTCH-1244 CrawlDBDumper to filter by regex (markus)
* NUTCH-1237 Improve javac arguements for more verbose ouput (lewismc)
* NUTCH-1236 Add link to site documentation to download older versions of Nutch (lewismc)
* NUTCH-1146 Prevent generation of _SUCCESS files in output (jnioche)
* NUTCH-1232 Remove site field from index-basic (markus)
* NUTCH-1239 Webgraph should remove deleted pages from segment input (markus)
* NUTCH-1238 Fetcher throughput threshold must start before feeder finished (markus)
* NUTCH-1138 remove LogUtil from trunk and nutch gora (lewismc)
* NUTCH-1231 Upgrade to Tika 1.0 (markus)
* NUTCH-1230 MimeType API deprecated and breaks with Tika 1.0 (markus)
* NUTCH-1235 Upgrade to new Hadoop 0.20.205.0 (markus)
* NUTCH-1217 Update NOTICE.txt to drop some copyrights (lewismc)
* NUTCH-1129 Add freegenerator, domainstats and crawldbscanner to log4j (markus)
* NUTCH-1184 Fetcher to parse and follow Nth degree outlinks (markus)
* NUTCH-1221 Migrate DomainStatistics to MapReduce API (markus)
* NUTCH-1216 Add trivial comment to lib/native/README.txt (lewismc)
* NUTCH-1214 DomainStats tool should be named for what it's doing (markus)
* NUTCH-1213 Pass additional SolrParams when indexing to Solr (ab)
* NUTCH-1211 URLFilterChecker command line help doesn't inform user of
STDIN requirements (mattmann)
* NUTCH-1209 Output from ParserChecker Url missing a newline (mattmann)
* NUTCH-1207 ParserChecker to output signature (markus)
* NUTCH-1090 InvertLinks should inform when ignoring internal links (Marek Backmann via markus)
* NUTCH-1174 Outlinks are not properly normalized (markus)
* NUTCH-1203 ParseSegment to show number of milliseconds per parse (markus)
* NUTCH-1185 Decrease solr.commit.size to 250 (markus)
* NUTCH-1180 UpdateDB to backup previous CrawlDB (markus)
* NUTCH-1173 DomainStats doesn't count db_not_modified (markus)
* NUTCH-1155 Host/domain limit in generator is generate.max.count+1 (markus)
* NUTCH-1061 Migrate MoreIndexingFilter from Apache ORO to java.util.regex (markus)