Merge branch 'master' into COVID19-API

.github/PULL_REQUEST_TEMPLATE.md

-Thank you for taking the time to work on a Pull Request for this project!
-
-To ensure your PR is dealt with swiftly please check the following:
-
-- [ ] Your submissions are formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md)
-- [ ] Your additions are ordered alphabetically
-- [ ] Your submission has a useful description
+<!-- Thank you for taking the time to work on a Pull Request for this project! -->
+<!-- To ensure your PR is dealt with swiftly please check the following: -->
+- [ ] My submission is formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md)
+- [ ] My addition is ordered alphabetically
+- [ ] My submission has a useful description
 - [ ] The description does not end with punctuation
-- [ ] Each table column should be padded with one space on either side
-- [ ] You have searched the repository for any relevant issues or pull requests
-- [ ] Any category you are creating has the minimum requirement of 3 items
+- [ ] Each table column is padded with one space on either side
+- [ ] I have searched the repository for any relevant issues or pull requests
+- [ ] Any category I am creating has the minimum requirement of 3 items
 - [ ] All changes have been [squashed][squash-link] into a single commit
 
 [squash-link]: <https://github.com/todotxt/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit>

.github/workflows/test.yml

 name: "Run tests"
 
 on:
-  - push
-  - pull_request
+  schedule:
+    - cron:  '0 0 * * *'
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
 
 env:
   FORMAT_FILE: README.md

.github/workflows/validate_links.yml

+name: "Validate links"
+
+on:
+  schedule:
+    - cron:  '0 0 * * *'
+  push:
+    branches:
+      - master
+
+env:
+  FORMAT_FILE: README.md
+
+jobs:
+  test:
+    name: 'Validate links'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+
+      - name: Validate all links from README.md
+        run: build/validate_links.py ${FORMAT_FILE}

.github/CONTRIBUTING.md → CONTRIBUTING.md

@@ -4,6 +4,8 @@
 opened to market company APIs that offer paid solutions. This API list is not a marketing tool, but a tool to help the
 community build applications and use free, public APIs quickly and easily. Pull requests that are identified as marketing attempts will not be accepted.
 >
+> Please make sure the API you want to add has a full free access or at least a free tier before submitting.
+>
 > Thanks for understanding! :)
 
 ## Formatting
@@ -12,7 +14,7 @@ Current API entry format:
 
 | API | Description | Auth | HTTPS | CORS |
 | --- | --- | --- | --- | --- |
-| API Title(Link to API webpage) | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Does the API support [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)? * |
+| API Title(Link to API documentation) | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Does the API support [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)? * |
 
 Example entry:
 
@@ -26,6 +28,7 @@ Example entry:
 * `apiKey` - _the API uses a private key string/token for authentication - try and use the correct parameter_
 * `X-Mashape-Key` - _the name of the header which may need to be sent_
 * `No` - _the API requires no authentication to run_
+* `User-Agent` - _the name of the header to be sent with requests to the API_
 
 \* Currently, the only accepted inputs for the `CORS` field are as follows:
 
@@ -33,13 +36,25 @@ Example entry:
 * `No` - _the API does not support CORS_
 * `Unknown` - _it is unknown if the API supports CORS_
 
-Please continue to follow the alphabetical ordering that is in place per section. Each table column should be padded with one space on either side.
+_Without proper [CORS configuration](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) an API will only be usable server side._
+
+After you've created a branch on your fork with your changes, it's time to [make a pull request][pr-link]. 
 
-If an API seems to fall into multiple categories, please place the listing within the section most in line with the services offered through the API. For example, the Instagram API is listed under `Social` since it is mainly a social network, even though it could also apply to `Photography`.
+*Please follow the guidelines given below while making a Pull Request to the Public APIs*
 
-## Pull Request
+## Pull Request Guidelines
 
-After you've created a branch on your fork with your changes, it's time to [make a pull request][pr-link]!
+* Continue to follow the alphabetical ordering that is in place per section.
+* Each table column should be padded with one space on either side.
+* If an API seems to fall into multiple categories, please place the listing within the section most in line with the services offered through the API. For example, the Instagram API is listed under `Social` since it is mainly a social network, even though it could also apply to `Photography`.
+* Add one link per Pull Request.
+* Make sure the PR title is in the format of `Add Api-name API` *for e.g.*: `Add Blockchain API`
+* Use a short descriptive commit message. *for e.g.*: ❌`Update Readme.md`  ✔ `Add Blockchain API to Cryptocurrency`
+* Search previous Pull Requests or Issues before making a new one, as yours may be a duplicate.
+* Don't mention the TLD(Top Level Domain) in the name of the API. *for e.g.*: ❌Gmail.com ✔Gmail
+* Please make sure the API has proper documentation.
+* Please make sure you squash all commits together before opening a pull request. If your pull request requires changes upon review, please be sure to squash all additional commits as well. [This wiki page][squash-link] outlines the squash process.
+* Target your Pull Request to the `master` branch of the `public-apis`
 
 Once you’ve submitted a pull request, the collaborators can review your proposed changes and decide whether or not to incorporate (pull in) your changes.
 
@@ -52,7 +67,6 @@ merge conflicts will be less likely. See more detailed instructions [here][synci
 * Create a [branch][branch-link] for your edits.
 * Contribute in the style of the project as outlined above. This makes it easier for the collaborators to merge
 and for others to understand and maintain in the future.
-* Please make sure you squash all commits together before opening a pull request. If your pull request requires changes upon review, please be sure to squash all additional commits as well. [This wiki page][squash-link] outlines the squash process.
 
 ### Open Pull Requests
 
@@ -62,11 +76,11 @@ Other contributors and users may chime in, but ultimately the decision is made b
 
 During the discussion, you may be asked to make some changes to your pull request.
 
-If so, add more commits to your branch and push them – they will automatically go into the existing pull request!
+If so, add more commits to your branch and push them – they will automatically go into the existing pull request. But don't forget to squash them.
 
 Opening a pull request will trigger a Travis CI build to check the validity of all links in the project. After the build completes, **please ensure that the build has passed**. If the build did not pass, please view the Travis CI log and correct any errors that were found in your contribution. 
 
-Thanks for being a part of this project, and we look forward to hearing from you soon! 
+*Thanks for being a part of this project, and we look forward to hearing from you soon!* 
 
 [branch-link]: <http://guides.github.com/introduction/flow/>
 [clone-link]: <https://help.github.com/articles/cloning-a-repository/>

LICENSE

+MIT License
+
+Copyright (c) 2021 public-apis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

build/requirements.txt

 flake8>=3.5.0
-httplib2==0.9.2
+httplib2==0.19.0

build/validate_format.py

@@ -5,7 +5,7 @@ import sys
 
 anchor = '###'
 min_entries_per_section = 3
-auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No']
+auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No', 'User-Agent']
 punctuation = ['.', '?', '!']
 https_keys = ['Yes', 'No']
 cors_keys = ['Yes', 'No', 'Unknown']
@@ -20,7 +20,6 @@ num_segments = 5
 
 errors = []
 title_links = []
-previous_links = []
 anchor_re = re.compile(anchor + '\s(.+)')
 section_title_re = re.compile('\*\s\[(.*)\]')
 link_re = re.compile('\[(.+)\]\((http.*)\)')
@@ -68,12 +67,6 @@ def check_entry(line_num, segments):
         title = title_re_match.group(1)
         if title.upper().endswith(' API'):
             add_error(line_num, 'Title should not end with "... API". Every entry is an API here!')
-        # do not allow duplicate links
-        link = title_re_match.group(2)
-        if link in previous_links:
-            add_error(line_num, 'Duplicate link - entries should only be included in one section')
-        else:
-            previous_links.append(link)
     # END Title
     # START Description
     # first character should be capitalized

build/validate_links.py

@@ -5,44 +5,92 @@ import re
 import socket
 import sys
 
+ignored_links = [
+    'https://github.com/public-apis/public-apis/actions?query=workflow%3A%22Run+tests%22', 
+    'https://github.com/public-apis/public-apis/workflows/Validate%20links/badge.svg?branch=master', 
+    'https://github.com/public-apis/public-apis/actions?query=workflow%3A%22Validate+links%22',
+    'https://github.com/davemachado/public-api',
+]
 
 def parse_links(filename):
     """Returns a list of URLs from text file"""
     with open(filename) as fp:
         data = fp.read()
     raw_links = re.findall(
-        'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+',
+        '((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'\".,<>?«»“”‘’]))',
         data)
-    links = [raw_link.replace(')', '') for raw_link in raw_links]
+    links = [raw_link[0] for raw_link in raw_links]
     return links
 
+def dup_links(links):
+    """Check for duplicated links"""
+    print(f'Checking for duplicated links...')
+    hasError = False
+    seen = {}
+    dupes = []
+
+    for link in links:
+        link = link.rstrip('/')
+        if link in ignored_links:
+            continue
+
+        if link not in seen:
+            seen[link] = 1
+        else:
+            if seen[link] == 1:
+                dupes.append(link)
+
+    if not dupes:
+        print(f"No duplicate links")
+    else:
+        print(f"Found duplicate links: {dupes}")  
+        hasError = True  
+    return hasError
 
 def validate_links(links):
     """Checks each entry in JSON file for live link"""
-    print('Validating {} links...'.format(len(links)))
-    errors = []
+    print(f'Validating {len(links)} links...')
+    hasError = False
     for link in links:
-        h = httplib2.Http(disable_ssl_certificate_validation=True, timeout=5)
+        h = httplib2.Http(disable_ssl_certificate_validation=True, timeout=25)
         try:
-            resp = h.request(link, 'HEAD')
+            resp = h.request(link, headers={
+                # Faking user agent as some hosting services block not-whitelisted UA
+                'user-agent': 'Mozilla/5.0'
+            })
             code = int(resp[0]['status'])
-            # check if status code is a client or server error
-            if code >= 404:
-                errors.append('{}: {}'.format(code, link))
+            # Checking status code errors
+            if (code >= 300):
+                hasError = True
+                print(f"ERR:CLT:{code} : {link}")
         except TimeoutError:
-            errors.append("TMO: " + link)
+            hasError = True
+            print(f"ERR:TMO: {link}")
         except socket.error as socketerror:
-            errors.append("SOC: {} : {}".format(socketerror, link))
-    return errors
-
+            hasError = True
+            print(f"ERR:SOC: {socketerror} : {link}")
+        except Exception as e:
+            hasError = True
+            # Ignore some exceptions which are not actually errors.
+            # The list below should be extended with other exceptions in the future if needed
+            if (-1 != str(e).find("[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)")):
+                print(f"ERR:SSL: {e} : {link}")
+            elif (-1 != str(e).find("Content purported to be compressed with gzip but failed to decompress.")):
+                print(f"ERR:GZP: {e} : {link}")
+            elif (-1 != str(e).find("Unable to find the server at")):
+                print(f"ERR:SRV: {e} : {link}")
+            else:
+                print(f"ERR:UKN: {e} : {link}")
+    return hasError
 
 if __name__ == "__main__":
     num_args = len(sys.argv)
     if num_args < 2:
         print("No .md file passed")
         sys.exit(1)
-    errors = validate_links(parse_links(sys.argv[1]))
-    if len(errors) > 0:
-        for err in errors:
-            print(err)
+    links = parse_links(sys.argv[1])
+    hasError = dup_links(links)
+    if not hasError:
+        hasError = validate_links(links)
+    if hasError:
         sys.exit(1)