>> Welcome back to Curlyboi Theatre, everyone. Next up, we have Vibhu Agarwal to talk to us
about Python web community. Vibhu is a student, a Pythonista and an open source enthusiast.
He likes to play around with web servers. He discusses about snakes and unicorns
in local tech groups like PyDelhi. Vibhu is joining us, all the way from India.
The talk is prerecorded. Vibhu will be in the chat. Is there anything
you'd like to say before we get started, Vibhu? VIBHU AGARWAL: Hi, everyone. I think there will be
a few questions, do pose them in the chat. Even in the text chat or the hall video chat. Let me know.
Hey, everyone, what is going on, especially in Python web? People trying out web development
or they're trying it out for the first time, ever, they find quite a lot of things to get
started with. As the target moved towards production ready website,
a lot of parts get added up. Today, we'll discuss all about those moving parts,
what those tools are, how are they comparative with each other, how do they work together,
and we'll try to understand all of them, as a whole, and not just individual components.
My name is Vibhu, I live in India. I've been using Python for four years. Okay. Let's start
with basics. Where we have a client. We a server. They establish a TCP connection.
Client requests HTTP request. Talking about static websites. They would request a particular
HTML file. They will maybe tell the path, in the HTTP path, and what the server would do is
to embed the content of HTML into the HTTP response and return to the response back.
If we want to store information into a database or maybe we want to calculate some
data accessibility and return some complex, we'd want to do it not in a static website,
but a dynamic website. We load an script and it processes the imports and returns a response
back. Nowadays, you can write the script in any language and that's good thing to have.
Now, what do we want to have a hybrid kind of a structure in which we want to support dynamic
and static content. It needs to redirect from the static content and direct to the backend server
as well. That is serving the dynamic content. I've seen a lot of projects which use Nginx. It
is fetching for going to fetch for static content, it will redirect to the static
files and it will if it receives a request, which needs a dynamic content, it redirects that request
to the backend server, it redirects the requests. Now, it is capable of doing a lot more than
redirecting the request, but I've seen a lot of projects which have a tech stack. But today, we're
going to focus on this thing, right now, that is our backend server, our application server.
Now, the main task or the first task which we see for the application server, is to receive our
listen to a lot of requests. How about we reserve a process just for listening to a request and
this circle is our process which will just listen to the request and doing nothing else.
Our request would be coming along and this process would this main process would be listening to
all of those requests. Now once it has these requests, it needs to parse that request, it
needs to under that request and break it down into different tokens so they know what are the inputs
of the HTTP request so that they can process and do some [Indiscernible] based on those.
Now, we talk about that, this backend server loads the Excel script through forking. Just to listen
to the request, now what this main process does is whenever it receives a request, it creates invokes
a new process and loads the Excel script from the disk and then the imports of the HTTP request,
which would then process the request and returns a response back. Now,
this kind of flow, or structure, is discussed in this [Indiscernible] CGI, Common Gateway
Interface. But they also discuss how to parse the imports or the request into the process.
This happens through environment variables. So, whenever a request arrives, it listens
to the request, parsing it down, and puts all of the HTTP inputs it and one it is created,
it inherits it from the parent and once the script is loaded into the memory, the loader script would
then pick up those imports from the environment, process these and return the response back.
So, that's basically CGI. Okay. Let's notice that we are doing a fork. Fork, as some people might
have heard, is a heavy thing because that involves a system call. Also, loading the excel script is
low. These are the two tasks which are not related to one particular request and you
may not even want to do it because what you just want to do is to just pass the request
and return the response back by running the script and you don't want the external overhead
of forking a new process and loading the script from the disk into the memory. To deal with this,
we have the pre fork model. Whenever we set up our backend server, we spin up a few processes which
load the external script into the memory. So, whenever a request arrives, what the server does
is to pick up one of the processes and assigns that request to one of to that process and then
the process would just execute the script and return the response back. So, there's overhead of
forking a new process and loading the script as the request arrives because that has been done
previously, before even the request arrived. So, that saves a lot of overhead. Because
we have introduced a lot of processes, this process management feature also needs to be
there at the backend server because we might want to increase or decrease the number of
processes or these processes, which process the request dynamically and so as to optimize the
usage of resources on a side. Okay. Let's talk about this external script
we keep saying about. So, what can this external script do about? It can do anything because it has
maybe all of the permissions, but what are the common tasks? One of the common tasks would be
to embed the dynamic information into HTML, that is done using the template. One of the
tasks would be to map different parts to different differentiation or URL patches. So, mapping one
path to its corresponding dispatcher is known as routing. And a lot of other tasks, which might
must not have seen developers doing because these all common tasks are separated out in a
separate tool, and what we know as frameworks. Now, might be familiar with the frameworks in
the form of Django, Flask, FastAPI. Flask is a microframework because it helps us do
micromanage our applications. We're importing the Flask, and then defining our path, our URL.
So, what is happening over here? So what is happening is that you're defining, or you're
writing a business logic and your dispatch somewhere in the your code and it's forming
a layer around your business logic so whenever it arrives, it is the front line, executes some code
and passes that result to your business logic, which should then process [Indiscernible] produces
some result. This is handed back to the web framework and the web framework would then
may or may not perform exit operations and return the response back. So, that's how the framework
forms a layer around your business logic and handles and forms a front line and it may
intercept the request somewhere, as well, because it may just receive the request and then send
it back after performing some operation and not even hand it to your business logic, based on the
depending on the implementation of that framework. Moving on, Django is very much framework. It has
been there for a long, long time. Has a great community and is one of my favorite frameworks.
So, Django is as they say batteries included. And one of the common theme common a lot of features
have been provided using the use of middlewares. So, if you've not used Django, what middleware
is perform extra layers between your framework and business logic. Once it receives the request, it
passes the request or result [Indiscernible] after executing code, it passes a result to the first
meta layer and then it executes some code and then passes the result back to the then it passes the
code the results to the next layer and then finally it hands it to your business logic.
Now, the business logic then executes some code and hands it back. That's how your middleware
work in Django. And this is and this way, it adds up to a lot of features in your entire system.
Okay. We discussed about our application server and a lot of people must have heard
about this application server in the form of u WSGI. This thing comes a lot of comes up a lot
in a development. So, remember the CGI, which is an [Indiscernible] load and execute the script.
Well, WSGI went one step further in stretching it further and restricting it and restricting it even
further in the way to how to invoke that script. What WSGI we're going to have two different
things. One is the server and one is that external script. They they should be kind of independent
from each other and how can they be made independent of each other? If one part knows what
is expected from the other and what is expecting of them, then they can be kind of independent of
each other and kind of development independently and they can be working independently.
So, okay, that didn't make much sense, let's quickly look at the code. So, the external script
should be prototyped into a one function, what is usually known as an application.
That application would receive two parameters and it should return one value.
The HTTP response status, you're going to finally pass
pass back and the second is the response headers. And, somewhere in between all of this,
you'll write your code, your business logic. Now, remember the Flask code,
this is the WSGI application. If you go and see your Django projects, you'll notice a WSGI file
and that will have this application variable. This is WSGI application. So, what do their
frameworks do is to form a WSGI application and gives this one object or variable back,
which it can be plugged into any web server. And, this is the point, because we discussed,
this is the point where you [Indiscernible] business logic. Wrap all of your business logic
and then just plug it at this point. So, now, it forms a layer around your business logic and your
web framework so the app is the front line of your request, it hands it off to the framework,
it may pass through middleware and your business logic handles it and reverse its way back.
So, the one part of the so, this external script part is handled by
your framework and your business logic and the other part is the application server,
which are commonly used in the form of gunicorn and UWSGI. It is to pass the request and receive
the request, listen to the request and to process management. So, [Indiscernible] over here.
What these processes, which actually handle the requests, we these are known as the
process as which handled the request are worker processes and the process which just listens to
the incoming requests are master processes. These are the two terminologies over here and let's
quickly now summarize what we saw until now. So, you write your code, you write your
business logic, along with all of the stuff which is provided by your web framework.
That gets wrapped into one app or app function. So this script is then passed on to the web server,
which may receive all which may receive extra inputs, for number of workers if it has to set up.
It may also receive some certificate files or locations so it can have the location of
those files. And, once everyone knows about the these inputs, it can spin up that many number
of processes. So, whenever a request arrives, it picks up one of the available pool of processes,
assigns that request to that process and then that process executes they execute the
thing and then returns a response back. That's pretty much all of it, over here.
Okay. Now, let's move on to asynchronous. Just a moment.
Now, asynchronous development, in the last three or four years, has seen a lot of maturity, a lot
of tools and libraries and a lot of frameworks have been developed using the asynchronous
development. And similarly, in the web world, as well, we've seen a lot of growth in web frameworks
and libraries so before moving into into those, let's quickly discuss what asynchronous is.
So, let's let's take an example, you are playing a cards game and you now you play your turn
and your opponent is thinking about their turn and is a little bit slow, what he thought is that
you can attend a talk while they are playing the cards game. So what you essentially did is to
utilize your time efficiently and did some work when the other person or some other task is not
depending on you. So while you are listening to the talk, you thought of a good turn and
they played their card and as they were going to disturb you just to poke you to play your card,
play your card, there was a [Indiscernible] standing and they asked your opponent to wait for
a while. Once you're free, you will be available back. So once you're done with the talk, you're
hopefully happy with my talk. You go to the negotiator and you see your opponent. You ask your
opponent, hey, I'm ready to play the turn. You go and you then play and then you play your turn.
So this negotiator and this list kind of act like an event loop so the task can be
popped up from the list and then added back by the negotiator into this list. So, you can
essentially be doing one task and then if the other if the task is not really depending on you,
you can drop doing or processing that task and you can pick some other task. That's kind of
asynchronous that's kind of what asynchronous is. And if you want to learn with more,
you can definitely check this out, if I don't explain it properly.
Okay. Let's talk about ASGI. It is kind of a super set to WSGI and provides a lot more flexibility.
Similar to WSGI, there would be two parts. One would be a server and one would be an external
script and they would be independent of each other and should be developed independently now.
ASGI is scope, one is events. Scope is something that stores the information.
It stores some preferences about the connection. Now, events are the messages are exchanged between
the web server and the application. Let's look at the code to understand it, even better.
Again, it the excel script should be wrapped up into one callable that is application.
Now notice over here, it is a function and not a normal function. It receives three parameters,
one is the scope, this would contain the information that is the same scope as
we discussed just now. It's going to contain the information about [Indiscernible] request
headers and the connection or other connection information. The second parameter is the receive.
This is a callable. This is an async function which you can await and get the message from the
web server. Now once you get your message from the web server, then you can take some action and once
you've taken action, you can return the response back. This, again, a callable, which is expecting
something. This is expecting another event. Event is just a small [Indiscernible] which represents
one method. It may have the it has to contain this type key, which categorizes which kind of
message you're returning back and that and you can await on this "send" callable.
So, now you notice there is no return statement over here, so
what you can do is do it on a loop and you can so, with one instance sharing of application,
you can get multiple messages and you can return multiple messages back.
With along with our asynchronous app, we have our we have a lot of asynchronous web frameworks,
as well. Why do we have a lot of different web frameworks, other than the ones which
we discussed about earlier? Because they're natively synchronous and natively synchronous
[Indiscernible] transformed into asynchronous that straightforward in a straightforward manner.
A lot of work has to be done if you want to develop fully if you want to
fully optimize resources and take full advantage of concurrency. Sanic is a fast framework. Quart
is very similar to Flask. FastAPI is a recent framework. It focuses on developing the web APIs.
FastAPI is built on top of Starlette. It's a fantastic [Indiscernible] framework,
which I'll be using a lot. Django Channels is one of the asynchronous framework, which comes
under the Django project. Now, Django does have some async [Indiscernible] I think, last year,
the whole of Django is not natively built or natively asynchronously built.
It has been there for a long time. It does best in what it's to do. But channels is
a relatively newer project and it is a fully asynchronous framework.
Now, that was the that is the third that is the second side, or the right side of our
of the framework. We have these three three of the most popular servers out there. Daphne comes under
Django project. It is intended to be used along with Django Channels. It seems to be just working
fine and actually it seems to be working great. Hypercorn is one, two, and I think, three,
as well. It's lightning fast and it focuses on performance.
And, Uvicorn I've [Indiscernible] the most. So, I know a little bit about how [Indiscernible] works.
And if you want to see a little more about discussion,
if you want to see a little more discussion about servers, you can check out check out this link.
And, the working of ASGI servers are similar to how WSGI works. Whenever a request arrives, it
web server takes a process from its available pool of processes, which will have the ASGI app in its
memory. It will then take on the request, process the [Indiscernible] and return the response back.
The only difference is that what WSGI does, whenever it receives multiple requests,
it processes those requests singly, even if they [Indiscernible] operation, just the
the [Indiscernible] just wait. But WSGI can show you some other task when one of the
process goes for [Indiscernible] operation, it can show you some other task, which
WSGI [Indiscernible] in the future, before, and that utilizes the resources when nothing
is dependent on us or the application. So, it gives a better response time to
all the requests and also saves the overall time. So, that improves performance.
So, that was pretty much it. What we could takeaway, from over here, is that we know about
all of the different parts, how do they work. So, let's say if if you [Indiscernible] imports
and the output is not is not coming out correct, there may be some error in your business logic and
you might want to fix that fix that thing in your business logic. And if you see that there's some
error somewhere in your tech stack, we couldn't pass the request, that may by an error on the
HTTP, which is used by the web server. You may want to file a bug report there. If you think of
a feature that that may be used commonly, you can re you can ask for that feature in one of
the frameworks and if they do not accept it, you always have the luxury to create a third parties
library, which the people can plug it into their web frameworks and that would just work.
How did I where did I read about all of these? Well, I read all of this. I read all of this from
the amazing documentation provided by all of these wonderful, wonderful projects. These just contain
all of the information which I talked about, most of information which I talked about today.
And, I'm ready for some questions now. Thank you. >> Okay. We're back. Thank you so much for that
talk, Vibhu. That was, like that was really helpful. I know that I'm going
to rewatch that to pay attention and to learn a lot and I really appreciate you
sharing what you learned through your journey through all of the web frameworks with all of us.
Is there anything you'd like to say? VIBHU AGARWAL: I do see a few questions,
I'll answer them in the Curlyboi Theatre chat and I'm glad that people found it helpful.
>> I think I think a lot of people did. Thank you, again. So, yes, you can talk to Vibhu some more
and ask some more questions over in the Curlyboi Theatre text chat, in the channel section and
we are going into our afternoon or our time zone appropriate time of day break. So, we'll see you
back here, when I get the right tab open that's got the right thing, in about 45 minutes. All
right? Have a nice break, everyone. Have have have a nice drink of water. Stretch your legs. Bye.