A guide to using Conda for managing virtual environments

You should always use a virtual environment for coding in Python, period. This might not be an instruction necessary for people who are on Windows or Mac. This is because Python is not something installed in the system by default and one of the ways to install Python is using something like the Anaconda distribution that comes with virtual environments by default. But for people who use Linux, there is always a system Python, an executable file you can find in `/usr/bin/python`.  However even on Linux it is still better to use something like Anaconda for your Python needs. 

Difference between Pip and Conda

Pip is an acronym for Pip Installs Packages. It is the official package manager for python packages. All packages that you install using the Pip command can be found in the Python Packaging Index (PyPI).  Conda is also a  package manager that can be used to install Python packages. But then, why do you need conda when you have Pip? The answer is that, there are multiple voids which only conda can fill. For example, say you want to use a different version of Python from the one installed on your system. Pip cannot help you here since it cannot install Python. However conda can install any Python version. Moreover since conda can be installed fully within your home directory, you do not need administrative privileges to install it. You might not have these administrative privileges if you are using a shared system or a server etc. The other interesting thing is that it is not just Python that conda can install but several other non-Python softwares are available through conda's channels. This includes softwares like ffmpeg, git, etc. 

Conda's dependency solver

Sometimes when you have used conda for sometime the wait time for installing new packages can be frustratingly high. The reason behind this is Conda's dependency solver. When using Pip alone, you might often run into broken environment situations. However this is quite rare if using conda for installing packages. This is because as already mentioned conda tracks non-Python dependencies as well and also spends some time making sure that the package you want to install is compatible with all other packages already existing in your environment. However sometimes the dependency solving can be very time consuming especially when your environment already has a lot of packages. There seems to be an alternative dependency solver that you can use with conda called libmamba. If you are an advanced user you can mess around with that. 

Anaconda or miniconda

Conda and Anaconda are not the same thing. Anaconda is a distribution that comes with conda along with several commonly used python packages. If you want to save on data download and storage space you can install an alternative version of Ancaonda called Miniconda. It is a lightweight version of Anaconda that comes with the bare essentials like conda.  Download miniconda from here. Select the Linux-64 bit version if you are on Linux. Make it executable using 
chmod u+x  <installer.sh>
Run the installer using ./installer.sh

Working with virtual environments

When you install conda, most often a default environment is created for you called `base`. You can see this name appearing beside the command prompt in your terminal.  If this is not there you might have to run a conda initialization command that writes the necessary activation commands to your your shell startup file (.bashrc if you are using bash). You need to re-login to the shell for this to take effect.  However it is best not to use this default environment but create a new one named after any new project that you are starting. 

conda create -n astro python==3.9

Activate the environment using `conda activate astro`.

Notice that apart from creating an environment we also install Python in the same step. This could also be done separately after creating the environment. However installing Python automatically install pip. Now if you want to install something using pip you can just use `pip install <packagename>`. 

Now certain packages need to be installed by specifying the channel names in the install command. For. eg, in the following command,

conda install sunpy -c conda-forge

The sunpy package is not available from the default conda channels hence we specify the channel where it is available from.

Some useful commands

Deactivate a conda environment using the following command. 
        conda deactivate

Remove a conda environment
        conda env remove -n <envname>

Export an existing conda environments to a requirements file
        conda list -e > requirements.txt

Replace the default dependency solver with a faster one

        conda install -n base conda-libmamba-solver
        conda config --set solver libmamba


References:



Comments

Post a Comment

Popular posts from this blog

Beginner's Guide to Machine Learning

This article is targeted towards people from a science (mainly astrophysics) background with very little hands-on experience in Python programming, especially tasks related to machine learning projects.  The goal is to get acquainted with the jargons of the trade and to understand where everything fits in, in the grand scheme of things. Maybe its helpful to think of this as a user's manual for jump starting your machine learning journey. However before we move on to the implementation side of things it is better if we address two fundamental questions. One - Why use machine learning at all? What problems does it solve? Two - Why Python? If you find yourselves already convinced about these questions do skip ahead. Why machine learning? Most things that we as humans learn cannot be put into a sequence of instructions to be followed word to word by another human or even a machine. Think about how you learnt to walk, speak, identify plants, birds and animals, distinguish a familiar fac...
Read more

Multi-layer Perceptrons or MLPs in Python

Image
In this article, I provide code in the form of a notebook that can be used to understand how Multi-Layer Perceptrons can be implemented in Python. The code also visualizes the decision boundary learnt by the model from the data. You can also play with the actual code on  colab . You can definitely implement a  neural network from scratch in Python  but in this article we make use of the MLP classifier implemented in the scikit-learn Python package. You can get an idea of the default parameters used, like the number of layers, activation function etc. by going to the  documentation  in their website. In the code that follows, we use a single hidden layer with 100 neurons. The activation function used is `relu` which is the default. The model optimizes the log-loss function using stochastic gradient descent. For the purpose of this tutorial we also make use of a synthetic dataset provided by the scikit-learn team and generated by the function `make_moons`. ...
Read more

Neural Network from Scratch using Python

In [1]: import numpy as np def sigmoid ( x ): return 1.0 / ( 1 + np . exp ( - x )) def sigmoid_derivative ( x ): return x * ( 1.0 - x ) class NeuralNetwork : def __init__ ( self , x , y ): self . input = x self . weights1 = np . random . rand ( self . input . shape [ 1 ], 4 ) self . weights2 = np . random . rand ( 4 , 1 ) self . y = y self . output = np . zeros ( self . y . shape ) def feedforward ( self ): self . layer1 = sigmoid ( np . dot ( self . input , self . weights1 )) self . output = sigmoid ( np . dot ( self . layer1 , self . weights2 )) def backprop ( self ): # application of the chain rule to find derivative of the loss function with respect to weights2 and weights1 d_weights2 = np . dot ( self . layer1 . T , ( 2 * ( self ...
Read more