The Kong Gateway Rate Limiting plugin  is one of our most popular traffic control add-ons. You can configure the plugin with a policy for what constitutes “similar requests” (requests coming from the same IP address, for example), and you can set your limits (limit to 10 requests per minute, for example). This tutorial will walk through how simple it is to enable rate limiting in your Kong Gateway .

Rate Limiting: Protecting Your Server 101

Let’s take a step back and go over the concept of rate limiting for those who aren’t familiar.

What is rate limiting?

Rate limiting protects your APIs from inadvertent or malicious overuse by limiting how often each user can call the API. Without rate limiting, each user may make a request as often as they like, leading to “spikes” of requests that starve other consumers. Once enabled, rate limiting can only perform a fixed number of requests per second. A rate limiting algorithm helps automate the process.

A web crawler is a hard-working bot to gather information or index the pages on the Internet. It starts at some seeds URLs and finds every hyperlink on each page, and then crawler will visit those hyperlinks recursively.

1. Choose an Ideal Programming Language

Based on the ranking of popular languages on developing web crawlers (based on result numbers of relative repositories host on Github on February, 2013). Python or Ruby probably is a wise choice, the mainly speed limit of web crawler is network latency not CPU, so choose Python or Ruby as a language to develop a web crawler will make life easier. Python provide some standard libraries, they are very useful, such like urllib, httplib and regex, those libraries can handle lots of work. Python also has plenty of valuable third-party libraries worth a try: scrapy , a web scraping framework. urllib3 , a Python HTTP library with thread-safe connection pooling, file post support. greenlet , a Lightweight concurrent programming framework. twisted , an event-driven networking engine.  

In the CS 5010 - Programming Design Paradigm course, we need write tons of documents, repeat ourselves again and again, so I develop this snippet to save us time. If you using vim, you need install snipMate first, You can get the snippet on my Github: https://github.com/guanlan/snippet-for-racket download the snippet file and put in ~/.vim/snippets/ You can watch demo here This snippets is also available on sublime, you can read tutorial here .

After about 3 months to learn and using Racket (a programming language in Lisp/Scheme family),  I learning lots of concepts of programming. The most important thing in FL(functional languages) is: "All data are immutable. All functions are pure."  

 Immutable Data

Immutable data cannot be modified after being created. It has many advantages:

1. Inherently Thread safety Parallel programming is the nightmare of some programers, because different threads simultaneously access the same object can cause unexpected problem, such like a race conditions. The most famous and classical example is Bank Account Problem . Today we get more and more cores in our computers, so the multi-core crisis  we need to face, some experts believe in the future, we using Scala or Erlang to deal with this crisis.Interesting thing is many “next generation” language is either the functional or support functional paradigm.
2. Eliminate side-effects There are so many reasons. Most important thing is global variables are difficult to understand. For example, if we want understand one functions in the non-trival project,  and we find a variable define in other place, so we will jump to the definition of this variable and other places where modify it, it will cost us more time to figure out what the exactly meaning of this function. So there is the same reason why the code is much easier to understand if we use immutable objects, because the scope of an immutable object is limited as possible. It will make our programming easier and more robust.

  The programming experts from other languages also know we need use immutable data as more as possible: Joshua Bloch,the author of Effective Java, said, “If an object is immutable, it can be in only one state, and you win big.You never have to worry about what state the object is in, and you can share it freely, with no need for synchronization.” Scott Meyers, Effective C++, Item 3: Use const whenever possible . Other languages also use immutable data frequently, such like Python, the number/string/tuple types are immutable data, the String build-in class in Java is immutable too. But immutable data is inconvenient in some situation:

Lisp has some very effective way to get jobs done, this article give you a direct way to use Python like Lisp.

1
2
3
4
5
6
7
8

cons = lambda el, lst: (el, lst) 
mklist = lambda *args: reduce(lambda lst, el: cons(el, lst), reversed(args), None) 
car = lambda lst: lst[0] if lst else lst 
cdr = lambda lst: lst[1] if lst else lst 
nth = lambda n, lst: nth(n-1, cdr(lst)) if n > 0 else car(lst) 
length = lambda lst, count=0: length(cdr(lst), count+1) if lst else count
begin = lambda *args: args[-1] 
display = lambda lst: begin(w("%s " % car(lst)), display(cdr(lst))) if lst else w("nil\n")

where w = sys.stdout.write

1
2
3

foldr = lambda f, i: lambda s: reduce(f, s, i)
foldl = reduce
mapcar = map

Pay attention about the speed when you use Python, here is a benchmarks for 5 languages:

 from The Great Computer Language Shootout .