v1.1.1

⚠️ Breaking Changes:

The Passive Crawling capability is being removed from katana and released as a separate project urlfinder by @dogancanbakir in #899

PASSIVE:
    -ps, -passive                   enable passive sources to discover target endpoints
    -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

What's Changed

🎉 New Features

• Added no-clobber flag to prevent katana from overwriting existing local files. by @dogancanbakir in #827

-ncb, -no-clobber                 do not overwrite output file

• Added -store-field-dir flag to make per-host field directory optional by @dogancanbakir in #877

   -sfd, -store-field-dir string     store per-host field to custom directory

🐞 Bug Fixes

• Fixed improper logging configuration by @dogancanbakir in #825
• Fixed the issue with setting custom headers/cookie headers by @RamanaReddy0M in #813
• Fixed header marshalling issue by preserving the header name casing by @dogancanbakir in #924
• Fixed ignoring form data extraction with -form-extraction when action is "#" by @dogancanbakir in #1052
• Fixed initial host scope checks to allow continued crawling when -crawl-scope is used by @dogancanbakir in #858
• Fixed duplicate field output when multiple value is used with -field flag by @dogancanbakir in #1031
• Fixed Dockerfile build errors by updating Golang by @dualfade in #1034
• Fixed issue where Ctrl-c could not terminate the process in headless mode with -show-browser option by @zrquan in #972
• Fixed the issue with setting custom headers/cookie headers in headless mode by @michael2to3 in #868
• Fixed cookie handling in hybrid mode to include in output by @alban-stourbe-wmx in #936

Other Changes

• The leakless switch is now enabled by default in headless mode, ensuring the browser is terminated when the Go process ends by @dogancanbakir in #831
• Updated README for recently introduced flag -e by @zy9ard3 in #826
• Corrected typos in comments by @eveneast in #851
• Added directive to ensure compatibility with Windows OS or 386 architecture by @Mzack9999 in #841
• Added deduplication for stored the fields on scan completion by @dogancanbakir in #885
• Updated README for installation command by @dwisiswant0 in #1015
• Added a response parser to extract endpoints from htmx attributes by @zrquan in #994
• Updated chrome launcher logic to prioritize the specified Chrome path with -scp over default paths by @zrquan in #979
• Added automatic form filling for select and textarea in forms by @alban-stourbe-wmx in #921
• Disabled headless auto form filling by @dogancanbakir in #918
• Added more JavaScript Libraries to improve parsing coverage by @geeknik in #900
• Corrected path handling with filepath.Join to use OS-specific separators. @ShuBo6 in #883
• Updated Dockerfile to recent Golang and Alpine images by @o6uoq in #886
• Improved form extraction logic to use HTML placeholder if input value is missing by @fmuttis in #957

New Contributors

• @zy9ard3 made their first contribution in #826
• @eveneast made their first contribution in #851
• @ShuBo6 made their first contribution in #883
• @o6uoq made their first contribution in #886
• @michael2to3 made their first contribution in #868
• @alban-stourbe-wmx made their first contribution in #936
• @zrquan made their first contribution in #972
• @fmuttis made their first contribution in #957
• @dualfade made their first contribution in #1034
• @dwisiswant0 made their first contribution in #1015

Full Changelog: v1.1.0...v1.1.1

v1.1.0

What's Changed

🎉 New Features

• Introduced passive crawling by @dogancanbakir in #781

   -ps, -passive  enable passive sources to discover target endpoints
   -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

Example:

katana -u tesla.com -passive -f qurl
...
https://static-assets-pay.tesla.com/api/payment-schema/creditcard?countrycode=dk
https://static-assets-pay.tesla.com/api/paymenttypes/?clientlibrary=payment-website
https://tradepartnertickets.tesla.com/dist/main.css?029b26e9be3aef4fc82c
https://tradepartnertickets.tesla.com/dist/vendors.vendors.css?029b26e9be3aef4fc82c
https://workforce.tesla.com/auth/callback?code=na_921f11c72db1d416c2fb624ea94ab5e1bad5f803
...
...
...
[INF] Found 208114 endpoints for https://tesla.com in 2m11.65937825s (commoncrawl: 128, alienvault: 1615, waybackarchive: 206371)

• Introduced option to exclude targets to crawl by @dogancanbakir in #731

   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

🐞 Bug Fixes

• Fixed issue with headless redirect by @Mzack9999 in #823
• Fixed issue with filter and match functionality (SDK) by @dogancanbakir in #775
• Fixed issue with -kf option to skip if the maximum depth is exceeded by @dogancanbakir in #799

Other Changes

• Added conditional support to include go-tree-sitter in build process by @RamanaReddy0M in #814
• Fixed nil pointer check by @dogancanbakir in #773
• Fixed error check on hybrid request parsing by @niudaii in #804
• Updated jsluice to new version with more results by @Ice3man543 in #695
• Updated jsluice by @geeknik in #800

New Contributors

• @beac0n5 made their first contribution in #732
• @toufik-airane made their first contribution in #736
• @niudaii made their first contribution in #804
• @geeknik made their first contribution in #800

Full Changelog: v1.0.5...v1.1.0

v1.0.5

What's Changed

🎉 New Features

• Added option to disable redirects with -disable-redirects by @ErikOwen in #630

   -dr, -disable-redirects  disable following redirects (default false)

🐞 Bug Fixes

• Fixed issue with header in output with XHR requests by @ErikOwen in #631
• Fixed with -aff config option by @dogancanbakir in #676
• Fixed issue with -kf option by @dogancanbakir in #655
• Fixed issue with blank field with empty value by @dogancanbakir in #687

Full Changelog: v1.0.4...v1.0.5

v1.0.4

What's Changed

🎉 New Features

• Added support for custom regex on field scope -fs option by @c3l3si4n in #571

katana -fs '(company-staging.io|company.com)' -u company.com

• Added option to disable redirects by @WigzyDev in #588

   -dr, -disable-redirects  disable following redirects (default false)

• Added -resume crawl option by @dogancanbakir in #538

   -resume string  resume crawl using resume.cfg

🐞 Bug Fixes

• Fixed issue with -match-regex option by @dogancanbakir in #565
• Fixed headless options parsing issue by @dogancanbakir in #564
• Fixed missing ExtractRelativeEndpoints() in scriptJSFileRegexParser() by @0x123456789 in #587

Other Changes

• Added build deps for go-tree-sitter -> tree-sitter by @dogancanbakir in #547
• Added readme syntax highlight by @apriil15 in #553

New Contributors

• @apriil15 made their first contribution in #553
• @0x123456789 made their first contribution in #587
• @c3l3si4n made their first contribution in #571
• @WigzyDev made their first contribution in #588

Full Changelog: v1.0.3...v1.0.4

v1.0.3

What's Changed

🎉 New Features

• Added DSL based advance response matcher and filter options by @dogancanbakir in #537

   -mdc, -match-condition string  match response with dsl based condition
   -fdc, -filter-condition string  filter response with dsl based condition

• Added option for TLS connection randomization by @Mzack9999 in #489

   -tlsi, -tls-impersonate  enable experimental client hello (ja3) tls randomization

• Added option for jsluice based javascript parsing by @Ice3man543 in #492

   -jsl, -jsluice  enable jsluice parsing in javascript file (memory intensive)

• Added option to parse and include form, input, textarea & select elements in jsonl output by @aristosMiliaressis in #464

   -fx, -form-extraction  extract form, input, textarea & select elements in jsonl output

• Added option to log XHR requests in jsonl output by @aristosMiliaressis in #481

   -xhr, -xhr-extraction  extract xhr request url,method in jsonl output

• Added the stored response path information in json output when -j used with -sr option by @ErikOwen in #532

🐞 Bug Fixes

• Fixed an issue wiht extension match / filter by @dogancanbakir in #491

🔨 Maintenance

• Fixed release workflow to accommodate tree-sitter bindings by @Mzack9999 in #527

Other Changes

• Added time unit support for -ct option by @RamanaReddy0M in #500

   -ct, -crawl-duration value  maximum duration to crawl the target for (s, m, h, d) (default s)

• Added option to allow katana to use an existing chrome instance by @DanielIntruder in #490

   -cwu, -chrome-ws-url string  use chrome browser instance launched elsewhere with the debugger listening at this URL

• Improved normalization of action & method attributes in form extraction feature by @aristosMiliaressis in #513

New Contributors

• @aristosMiliaressis made their first contribution in #464
• @DanielIntruder made their first contribution in #490
• @jen140 made their first contribution in #511
• @ErikOwen made their first contribution in #532

Full Changelog: v1.0.2...v1.0.3

v1.0.2

What's Changed

🎉 Features

• Added support for custom header input from file by @RamanaReddy0M in #436

katana -H cookie.txt -u https://mail.google.com -headless

• Added option to exclude raw request/response and body in jsonl output by @maik-s in #460

   -or, -omit-raw                    omit raw requests/responses from jsonl output
   -ob, -omit-body                   omit response body from jsonl output

🐞 Bugs

• Fixed issue with regex parsing by @yuzhe-Mortal in #406
• Fixed panic issue in headless mode by @RamanaReddy0M in #428
• Fixed issue with custom header input support in headless mode by @tarunKoyalwar in #437

🔨 Maintenance

• Added documentation for header / cookie input by @dogancanbakir in #461

Issues closed in this release - https://github.com/projectdiscovery/katana/milestone/6?closed=1

New Contributors

• @maik-s made their first contribution in #460

Full Changelog: v1.0.1...v1.0.2

v1.0.1

What's Changed

• Added support to ignore crawling same path with different query-param by @RamanaReddy0M in #371

   -iqp, -ignore-query-params  ignore crawling same path with different query-param values

• Added support for adding url scheme with input by @iamargus95 in #399

katana -u hackerone.com

• Added transparent embedding of common code by @Mzack9999 in #373
• Updated using as lib example to reflect struct changes by @ShubhamRasal in #397
• Updated using as lib example by @dogancanbakir in #390
• Updated net/url with urlutil by @iamargus95 in #399
• Fixed an issue with -do option by @Mzack9999 in #387

Issues closed in this release - https://github.com/projectdiscovery/katana/milestone/5?closed=1

New Contributors

• @h4r5h1t made their first contribution in #378
• @dogancanbakir made their first contribution in #390
• @iamargus95 made their first contribution in #399

Full Changelog: v1.0.0...v1.0.1

v1.0.0

What's Changed

Warning: breaking changes in json output structure.

previous format

{
  "timestamp": "2022-11-05T22:33:27.745815+05:30",
  "endpoint": "https://www.iana.org/domains/example",
  "source": "https://example.com",
  "tag": "a",
  "attribute": "href"
}

new format

{
  "timestamp": "2023-03-20T16:23:58.027559+05:30",
  "request": {
    "method": "GET",
    "endpoint": "https://example.com",
    "tag": "a",
    "attribute": "href",
    "source": "http://www.iana.org/domains/reserved",
    "raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
  },
  "response": {
    "status_code": 200,
    "headers": {
      "accept_ranges": "bytes",
      "expires": "Mon, 27 Mar 2023 10:53:58 GMT",
      "last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
      "content_type": "text/html; charset=UTF-8",
      "server": "ECS (dcb/7EA3)",
      "vary": "Accept-Encoding",
      "etag": "\"3147526947\"",
      "cache_control": "max-age=604800",
      "x_cache": "HIT",
      "date": "Mon, 20 Mar 2023 10:53:58 GMT",
      "age": "331239"
    },
    "body": "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
    "technologies": [
      "Azure",
      "Amazon ECS",
      "Amazon Web Services",
      "Docker",
      "Azure CDN"
    ],
    "raw": "HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: \"3147526947\"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
  }
}

• Refactored katana core by @Mzack9999 in #312
  • for synchronous correlation between http request/response
  • added request/response data into json output
  • added status code information in json output
  • updated json output structure
• Added technology detection of crawled endpoints in json output using wappalyzer by @Mzack9999 in #294

...
    "technologies": [
      "Azure",
      "Amazon ECS",
      "Amazon Web Services",
      "Docker",
      "Azure CDN"
    ],
...

• Added match and filter support for discovered endpoint by @ShubhamRasal in #346

   -mr, -match-regex string[]  regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]  regex or list of regex to filter on output url (cli, file)

• Added custom resolver input support by @ShubhamRasal in #340

   -r, -resolvers string[]  list of custom resolver (file or comma separated)

• Added self update option by @RamanaReddy0M in #356

   -up, -update  update katana to latest version

• Added support for ufile field based filter by @Mzack9999 in #357

katana -f ufile -u https://www.tesla.com

• Added -chrome-data-dir / -cdd flag to implement authenticated crawling #230 by @parthmalhotra in #231
• Added examples using rdn, fqdn, etc. scope filter in readme by @olearycrew in #288
• Fixed tls version error when using burp proxy by @Mzack9999 in #311
• Fixed issues with katana consuming available cpu by @ShubhamRasal in #362
• Fixed Dockerfile to avoid version mismatch by @olearycrew in #279
• Updated regex to detect/extract more endpoints by @yuzhe-Mortal in #249
• Updated debug logging by @tarunKoyalwar in #278

Issues closed in release - https://github.com/projectdiscovery/katana/milestone/4

New Contributors

• @olearycrew made their first contribution in #279
• @tarunKoyalwar made their first contribution in #278
• @yuzhe-Mortal made their first contribution in #249
• @RamanaReddy0M made their first contribution in #356

Full Changelog: v0.0.3...v1.0.0

v0.0.3

What's Changed

• Added custom field support for custom data extraction by @ShubhamRasal in #225
• Added support to store http request/responses to file by @edoardottt in #185
• Added option to specify local chrome binary path (-scp) by @parthmalhotra in #202
• Added chrome requirements recommended setup for ubuntu by @Mzack9999 in #251
• Added example to use katana as a library by @ShubhamRasal in #205
• Added support for error logs (-elog) option by @Mzack9999 in #267
• Added self-diagnostic functionality (-hc) option by @Mzack9999 in #260
• Added prototype of cycle detection by @Mzack9999 in #250
• Fixed a bug to use proxy in headless mode by @parthmalhotra in #169
• Fixed a bug to load complete response in headless mode by @parthmalhotra in #195
• Fixed a bug in the extension match/filer option by @ShubhamRasal in #184
• Fixed a bug in the rate limit option by @ShubhamRasal in #214

Issues closed in release - https://github.com/projectdiscovery/katana/milestone/3?closed=1

New Contributors

• @glaucocustodio made their first contribution in #173
• @parthmalhotra made their first contribution in #169
• @ShubhamRasal made their first contribution in #184
• @0xflotus made their first contribution in #268

Full Changelog: v0.0.2...v0.0.3

v0.0.2

What's Changed

• Added -no-sandbox option to run headless mode as root user by @edoardottt in #132
• Added -headless-options option to include additional chrome options by @edoardottt in #132
• Fixed nil error with -version option by @mschader in #140
• Fixed Dockerfile to include chrome installation by @ehsandeep in #156

New Contributors

• @RohanTheProgrammer made their first contribution in #138
• @edoardottt made their first contribution in #132
• @mschader made their first contribution in #140

Full Changelog: v0.0.1...v0.0.2