>> Hello, welcome back to Curlyboi Theatre for our last talk of the day.
Um, because this is our last talk of the day, please remember that the conference
closing will be in this room, where we are right now, after this talk. They'll
be a very short break in between, stick around to hear from our lovely conference organizers.
Okay. So, we have, for us all today, Sebastian Witowski to talk about Python Versions and
Dependencies Made Easy, if I get the title correct. Sebastian is a consultant in Poland.
He fell in love with Python and teaching and is helping companies untangle their complicated
architecture and build all sorts of interesting Python projects. So, if there is time at the end,
Sebastian will be taking questions so please put your questions in the question tab in Venueless.
You should all know where it is by now, hopefully, if not, ask away. And I'll ask
Sebastian at the end. Over to you, Sebastian. SEBASTIAN WITOWSKI: Cool. Thank you so much. Hi,
everyone. My name is Sebastian Witowski. I join different teams and work on different projects.
And as I work on those different projects, I notice they often suffer from one of two problems.
They're overengineered or stacked in the 90s. The overengineered ones involve jumping on every
shiny, new tool that you find on the first page of Hacker News
and then everyone gets excited because it's fun to work with something brand new. What is not fun is
when your tool gets deprecated. Maybe the creator gets bored and stops working on it. You have to
keep it up to date yourself or do it yourself. Some people think because it was fun to manage
everything using bar scripts, but to be honest, when was the last time you wrote one?
Writing them from scratch is not that bad. Maintaining a bar script that someone else
wrote is not great. So, neither of the situations is good. You should update your tools but at the
same time, you should choose those tools that you know that they are proven and they will be
supported for the next few years. So that's why I've decided to make this presentation. I want
to show you some useful Python tools, tools that I recommend to my colleagues and to my clients.
We'll talk about things that a lot of beginner programmers struggle with at some point. First,
I will show you how to install different Python versions and how to manage them using pyenv.
Next, I will talk about Python dependency and I will explain you what are virtual environments,
how they work and why you need them if you work with multiple Python projects.
I'll compare the built in module, like Virtual and Wrapper. We will talk about pipx,
this is a great tool if you want to install some Python package globally,
but you still want to isolate it from other packages on your computer.
And before we move on, I really want to highlight one thing, those are all just
tools and there are a lot of other tools that do the same thing. If you use something else,
that's great. My goal, here, is not to convince you that those tools are the best. No. I just
want to show you how to solve some basic problems when you configure your development environment.
If you are using conda and you are happy with it, the first two bullet points are
already solved for you. But if you don't know what to use, let me show you some recommendations.
So, first let's talk about installing Python on your computer.
Depending on your operating system, your computer might already come with some version of Python
installed. If you're using Mac OS, it is 2.7. In windows, you don't have any Python installed.
Let's say that your operating system comes with some Python version,
this Python version that is already installed on your computer is often called System Python.
No matter what system version of Python you have, I strongly suggest you don't use it. In
many cases, as we saw with Mac OS, it's outdated. 2.7 is not supported by the core developers so
hopefully you are not using it since a long time. You might be tempted to update the system Python
to Python 3, but you probably have some programs that require Python 2.7. You would be surprised
how many programs, or even parts of the operator system still requires Python 2.7
in Mac OS. If you change the Python version that comes pre installed, you risk your computer will
stop working at all and that is not fun. I have done this in the past, when I didn't know much
about programming and operating systems and I had to re install the whole operating system.
My advice is to leave it alone and pretend that it doesn't exist. No matter what operating system you
have, you will need to install Python and there are many different ways how you can do this.
You can download the installer for your operating system.
You can use a package manager like Home brew on Mac and compile Python from the source files.
However, my favorite way of installing Python, that I have been using since a few years,
is to use a tool called pyenv. It managing Python versions. You can install a Python version on
your computer and quickly switch between those versions you have already installed.
It might not be a big deal if you only use one Python version at the time. But
if you're working with multiple Python projects that require different Python versions, this is
extremely useful. I try to use the latest version with Python. But my clients don't usually use the
latest version of Python. I've been using pyenv for years and it works very well. If you have
worked with you might recognize similar tools, for example, there is Ruby and Node JS. Pyenv
will work fine on Mac OS and Linux, check out the pyenv win. It might not have all the
features that the standard pyenv has, but it has all the essential ones that I will talk about.
But then again, if you are Windows and you are a happy conda user, there's actually no point
in switching to use pyenv. One of the things conda can do is creator virtual environments.
You can use pyenv with Home brew and use the tool called pyenv Installer. And, while this last
option requires you to download a bar script from the internet and run it in a terminal which is a
big security no no, it's actually very convenient. It will install some additional plugins like
pyenv doctor that you can use to verify that your pyenv installation is working fine.
So, once you have pyenv installed, you can use it to install new Python version.
To see the list of all the available Python versions, you can run pyenvinstall list
and you will see a huge list of Python available installations. We have the standard Python
versions. Apart from that, we have Anaconda, and so on. All those different versions of Python
can be installed with pyenv. If you ever wanted to try PyPI, now you can very easily do this.
Once you select the version from this list, type "pyenvinstall" and the version number and
go make yourself a coffee. I mean it. The installation usually takes a few minutes.
There is OpenSSL that you can use. If you don't have those libraries installed, each time you
try to download a new Python version, pyenv will try to download it and temporarily use it
so I suggest you take a look at the GitHub page for the list of the required dependencies.
Okay. So, once this is done, you can run pyenv versions to see if this new version is on this
list. If you can see it there, then it means that you're all set. So, now we have to tell
pyenv to use that version. And for that, we have to choose one of three different levels at which
we can change the Python version. We have global, local or shell. So, what does it mean? Well,
9 out of 10 times, you probably want to change the global Python version and to do that, you just run
"pyenv global 3.9.0" and that changes the global Python version on your computer.
Then we have pyenv local. So, imagine that most of the time you are working with Python 3.9, but
you have this one project on your computer that requires Python 3.7 and for some strange reason,
it won't work with any other Python version. So, instead of changing the global Python version
to 3.7 when you work on this project and then changing it back to 3.9 when you stop working on
this project you can set a local Python version. And that's where the pyenv local command comes in.
You can run pyenv local 3.7 or 3.9 and will set that version for the current folder and
all its subfolders. So, whenever you go inside this folder, Python version will automatically
change and when you go outside of this folder, it will change back to the global one.
That's very handy when you work with multiple projects that use different Python versions. So,
instead of changing the global Python version back and forth, you just have to run pyenv local
in one folder, pyenv local in another folder and then pyenv will automatically change the version
when you go inside each of those folders. And then finally, we have pyenv shell. And
this changes the Python version you use in your current shell session. You might
want to use it, for example, when you want to temporarily change the Python version.
You want to run code under Python 2, you run pyenv shell 2.7.18 and you can use Python 2. But once
you open a new session in your terminal, you go back to using the previous version of Python.
So, that's how we can easily manage Python versions on your computer.
Let's talk about dependencies. Let's start with pypi. Pypi is a package manager for Python.
Whenever you want to install a new package, all you need to do is to run pip install, the name
of the package, and it's running. However, pypi has one big problem. Whenever you ask
it to install a specific version of a package, it will uninstall the previous versions and install
the one that you asked for. Let me show you an example of what I mean. Imagine you are a web
developer and you want to build a Django website, you install the latest version of Django and pypi.
Everything works great. You build an awesome website and the website is so awesome and they ask
you, hey, you're making awesome Django websites so can you fix my Django website? As it often
happens with clients, they are not using Django 3, they are still using, let's say, Django 2.2.
So, you install that specific version by typing "pypi install Django"
and you start working on this website for your client. So far, so good. But, later that day,
you discover that your personal website, so this one, built with Django 3, has a bug. When
you want to test it and make sure that it is working fine, you get an error message saying
"Django 3 is not installed." Like, what? I mean, we installed it yesterday, what happened with it?
Well, when we told pypi to install Django 2.2, pypi first checked if we already have Django
installed and we did, it's just it wasn't Django 2.2 so pypi uninstalled that version. So we just
run into a problem with dependencies management. We have this problem because pypi installs Python
packages inside the side packages folder and puts each package into a separate folder
named after that package. So, Django 3 is placed inside side package/Django.
But when we want to install Django 2, it will also be placed inside sidepackages/Django. So, pypi has
to first remove what's inside this Django folder and then install a different version. If you only
work with one Python project all the time, then you're probably not affected by this problem.
But sooner or later, you will need to install different versions of the same package and you're
going to run into issues with pypi uninstalling some dependencies of some other packages.
The problem with pypi is that it installs all the packages in the same folder so
how about we tell pypi to temporarily install packages into a separate folder and then we
tell our Python interpreter to use packages from that folder? Well, that's exactly what virtual
environment does. A virtual environment is a folder that contains a Python installation and
any additional packages that you install. When you activate a virtual environment, two things happen.
So, first, you tell pypi, hey, I want you to install any new Python package in that folder and
you tell Python interpreter, hey, I want you to use Python and Python packages from that folder.
If we try to install Python packages, pypi won't uninstall because it's no longer using the global
site packages. It's using side packages from this specific virtual environment.
So, how does it look in practice? Well, first we need to create a virtual environment.
Python has a built in module called Venv. So, in my case, I'm creating a folder called ".venv"
are both a common name used for folders with virtual environments so it makes it obvious that
whenever you go into some Python project and you see this folder inside, there are things
related to virtual environment. But, it also has another benefit. Some code editors, like PyCharm
will recognize this as a virtual environment and automatically start using it on your project.
So, we have created a virtual environment and now we have to activate it.
We have a bin directory and then we have the activate script. Since this is a bar script,
we have to run it with source command and of course, if you're using a different shell,
there are different files that you can use. For Windows users, there is activate and so on.
Once you activate it, you should see a difference in your terminal's prompt.
It should not display the name of the current virtual environment
so if you see something like this venv in parenthesis, that's great because it means
a given virtual environment is now active. And, now we do everything that we would normally do.
It we install it with pypi, it will be installed with this virtual environment
and it will have access to the Python packages from this virtual environment.
If you want to stop using the virtual environment, you have to run the deactivate command.
When you call it, it will revert all the changes that the activate did. So,
you will go back to using the global Python version and back to global pypi packages.
So, a typical workflow using virtual environments involves creating one virtual environment for each
of your projects. If you're working with two Django projects, you first create a table one
virtual environment, you work on it. Then if you want to switch to another project,
you create another virtual environment. You activate and you work on that project.
Venv module is perfectly fine for managing virtual environments, but I want to show you another tool
I have been using for a long time. It's called Virtualenvwrapper. You can install it with pypi
and of course, it works on Linux and Mac and Windows users, you have to do something else.
There is virtualenvwrapperwindows. If you're a happy conda user, you can stick to this.
It stores all the virtual environments inside the .virtual env folder inside your home directory and
it provides you with useful commands. You can create a new virtual environment by typing
"mkvirtualenv." It's not creating the virtual environment in the folder.
If you want to activate, you have to call "work on Django to app" and then it will figure out
where is the activation script. You don't have to type the whole path to the activate script
and figure out in which folder it's located. You can call "lsvirtualenv." Again, it's impossible to
get this list because venv doesn't keep track. And virtualenvwrapper stores everything in the list.
To remove, you run "rmvirtualenv" and the virtual environment.
You are probably wondering which is better? Both are great. I use virtualenvwrapper because I don't
want to type this whole thing where, of course, half of the time, I'm in, like, some subfolder
and get an error that it doesn't exist so I have to figure out where the activation script is
located. I like to say, "work on Django 2 up" and be done with it. I create a table virtual
environments that connected to a project. I have a bunch of data science libraries.
Virtualenvwrapper can create temporary virtual environment. It will make a virtual environment
and when you deactivate it, it disappears. It's nice for testing some pypi packages.
If you store virtual environments in the same folder, your code editor will pick it up and
start using it automatically. When you delete this project, you also delete the virtual environment.
All right. So, we have covered how to manage Python packages
inside your projects, what about global Python packages on your computer?
Some tools are much more convenient to use when you install them globally instead of
installing them in a specific virtual environment. You want to use those tools
across all your projects, or even before you create a table a project. For example,
with virtualenvwrapper, we would create a virtual environment.
But then again, if two different tools require two different versions of the same dependency, we have
a problem. You install one tool and everything is fine, you install a second tool, it changes
dependencies of the first tool and then the first tool stops working.
We could install each of those tools inside a separate virtual environment,
but this is a lot of hassle to use it like that. Each time, you'd have to activate,
run it and deactivate. We can easily solve this problem with pipx. Pipx installs Python
packages inside separate environments, but at the same time, those packages act like if they
were installed globally so you don't have to activate any virtual environment to use them.
How does it look in practice? Let's say you want to install [Indiscernible] for Python code.
I run well, we run pipx install black and after a few seconds, we get a message saying that
Black has been installed and now we have those three commands, black, black primer and black D.
And now we can use Black as if it was installed with pypi.
It doesn't matter if we were inside of a virtual environment or not. We don't have to
activate anything. We run the same commands as we would run if we weren't using pipx.
And if we want to install a different version of Black inside of a virtual environment,
we can still do this. This Black, from inside a virtual environment,
will take precedence over the global Black. You'll be using Black from this virtual environment.
What else we can do with pipx? We can install packages to see what commands they offer.
We can uninstall a package and this will also clean up its virtual environment.
We can upgrade with one command and pipx inject. It will install a pypi package inside of a virtual
environment of another package. You might use it to install pytest. We can't just run pipx
install Python Discov because it will be inside its own virtual environment. So we have to call
pipx inject pytest and it will install it inside the virtual environment of pytest.
And that brings us to the end of my presentation so let's quickly summarize what we saw.
First, use pyenv to install new Python versions. Once you set it up, it's really easy to install
new Python versions. Then use virtual environments when you want to isolate their dependencies.
Create a separate virtual environment for each of your projects.
It has venv and it's perfectly fine to use it. Virtualenvwrapper is also a really great choice.
And, finally, if you want to install some tools globally, you can use pypi outside of the virtual
environment but you risk that if there is some version conflict, they won't work together.
So a much better tool to use is called pipx and this will install global package into a separate
into a separate virtual environment but for you, there won't be any difference in how you use them.
Thank you very much for listening and I think we still have some time for questions.
>> Thank you, Sebastian. We probably have time for one question. And,
we we have we have two questions. So, I'm going to use my use my powers to pick
I lost it. Um, okay. So, someone asks, I'd love to know which one is used in production?
Is there a set of best practices to follow? SEBASTIAN WITOWSKI: For production, I use yet
another tool that I didn't have time to put in this talk. I use a tool called what was the name?
Pypi tools. It has a command called pypi compile and you can provide a list of dependencies,
like, you just specified, I want Django, I want pytest.
It will take those requirement files and resolve all the dependencies and
third party packages, dependencies that you use. You take this and you put it into a Docker image
or venv. That's what I would use in production. >> Pypi compile is great. I use that,
too. A lot of questions came in. We are out of time. So, Sebastian, are you would you
like to head over to the text chat channel for a few minutes, to answer some more?
SEBASTIAN WITOWSKI: Sure. >> Do jump into the the Curlyboi Theatre text
chat channel and I will paste the questions over there. But, leave when you jump to the text chat,
the stream for Curlyboi Theatre will keep playing up in the top corner, leave that playing because
the conference closing is coming up in 15 minutes, okay, and we don't want anyone to miss that.
All right. So, thank you, once again, Sebastian. SEBASTIAN WITOWSKI:
Thank you so much for having me. >> I think that was really helpful.
The chat was on fire. So, thank you. SEBASTIAN WITOWSKI: Thanks. Bye bye.